import { writeFileSync } from 'node:fs'; import { dirname, join } from 'node:path'; import { fileURLToPath } from 'node:url'; const scriptDir = dirname(fileURLToPath(import.meta.url)); const root = join(scriptDir, '../../..'); function u(s) { return s.replace(/\\u([0-9a-fA-F]{4})/g, (_, hex) => String.fromCharCode(parseInt(hex, 16))); } const content = u(`package com.fs.company.service.workflow.deepdialogue; import java.util.ArrayList; import java.util.LinkedHashSet; import java.util.List; import java.util.regex.Pattern; /** * Split customer-facing text into: main answer first, then each question as its own message. */ public final class CustomerFacingMessageSplitter { private static final Pattern SENTENCE_SPLIT = Pattern.compile("[\\u3002\\uff01\\uff1f?!\\uff1b\\\\n]+"); private CustomerFacingMessageSplitter() { } /** * @return non-empty list: index 0 = answer body; rest = follow-up / rhetorical questions */ public static List split(String reply, String plannedProbe, boolean shouldProbe) { List out = new ArrayList<>(); if (reply == null || reply.isBlank()) { appendUnique(out, plannedProbe, shouldProbe); return out; } String body = UniversalDeepDialogueService.stripProbeFromReply(reply, plannedProbe); List answerParts = new ArrayList<>(); List questionParts = new ArrayList<>(); for (String sentence : SENTENCE_SPLIT.split(body)) { String part = sentence != null ? sentence.trim() : ""; if (part.isEmpty()) { continue; } if (isQuestionSentence(part)) { questionParts.add(normalizeQuestion(part)); } else { answerParts.add(part); } } String answer = String.join("\\u3002", answerParts).trim(); if (!answer.isEmpty() && !answer.endsWith("\\u3002") && !answer.endsWith("\\uff01") && !answer.endsWith("!") && !answer.endsWith("~") && !answer.endsWith("\\u2026")) { answer = answer + "\\u3002"; } appendUnique(out, answer.isEmpty() ? null : answer.replaceAll("\\u3002+$", "\\u3002"), false); LinkedHashSet seen = new LinkedHashSet<>(); if (!out.isEmpty() && out.get(0) != null) { seen.add(out.get(0)); } for (String q : questionParts) { appendUnique(out, q, false, seen); } appendUnique(out, plannedProbe, shouldProbe, seen); return out; } /** Keep declarative sentences only; questions are sent as separate follow-up messages. */ public static String extractAnswerOnly(String reply) { if (reply == null || reply.isBlank()) { return reply; } List parts = split(reply, null, false); if (parts.isEmpty()) { return reply.trim(); } String answer = parts.get(0); return answer != null && !answer.isBlank() ? answer.trim() : reply.trim(); } public static List followUps(String reply, String plannedProbe, boolean shouldProbe) { List all = split(reply, plannedProbe, shouldProbe); if (all.size() <= 1) { return List.of(); } return new ArrayList<>(all.subList(1, all.size())); } private static void appendUnique(List out, String text, boolean required) { appendUnique(out, text, required, null); } private static void appendUnique(List out, String text, boolean required, LinkedHashSet seen) { if (!required && (text == null || text.isBlank())) { return; } if (text == null || text.isBlank()) { return; } String normalized = text.trim(); if (seen != null) { String key = normalized.length() > 12 ? normalized.substring(0, 12) : normalized; if (seen.contains(key) || seen.stream().anyMatch(s -> s != null && s.contains(key))) { return; } seen.add(key); } out.add(normalized); } static boolean isQuestionSentence(String sentence) { if (sentence == null || sentence.isBlank()) { return false; } String t = sentence.trim(); if (t.endsWith("?") || t.endsWith("\\uff1f")) { return true; } return t.matches(".*(\\u5417|\\u5462|\\u5417\\uff1f|\\u5462\\uff1f|\\u4e48|\\u554a|\\u5427|\\u561b|\\u662f\\u4e0d\\u662f|\\u80fd\\u4e0d\\u80fd|\\u53ef\\u4e0d\\u53ef\\u4ee5|\\u8981\\u4e0d\\u8981|\\u6709\\u6ca1\\u6709|\\u591a\\u5c11|\\u51e0|\\u54ea|\\u8c01)[\\uff1f?]?$"); } private static String normalizeQuestion(String sentence) { String t = sentence.trim(); if (!t.endsWith("?") && !t.endsWith("\\uff1f")) { t = t + "\\uff1f"; } return t; } } `); for (const rel of [ 'java/fs-service/scripts/lobster_utf8_sources/CustomerFacingMessageSplitter.java', 'java/fs-service/src/main/java/com/fs/company/service/workflow/deepdialogue/CustomerFacingMessageSplitter.java', ]) { writeFileSync(join(root, rel), content.replace(/\r\n/g, '\n'), 'utf8'); console.log('fixed', rel); }