One message, two notes

The rest of this post is the deep dive: the two incidents, the merge that intersects rather than picks a winner, and the general rule about when validation is the right lever and when it is a way of turning a small defect into data loss. File paths refer to the real code.

The setup: a plan over a batch of messages

Inbound capture over WhatsApp and Telegram doesn't process messages one at a time. Messages accumulate into a session, and one model call turns the whole session into a plan: which people appear, which messages belong to whom, which groups of messages become notes, and which messages are too ambiguous to attribute and should be parked as a question for the user.

Every message carries a seq. The plan refers to messages only by seq — since we stopped letting the model write note text at all, a note is nothing but a list of sequence numbers plus the substrings that were instructions to the bot.

The plan has one invariant, stated in the prompt: every seq must appear in exactly one place. Nothing dropped, nothing duplicated.

Only half of that was ever enforced. The accounting function collected every planned seq into a Set and compared it against the input. That catches a forgotten message perfectly. It cannot see a duplicated one — a Set is, definitionally, the data structure that discards the information you'd need. The "nothing dropped" half was code. The "nothing duplicated" half was a sentence.

Incident one: two notes on one contact

The first failure: a forwarded message came back as two notes on the same contact. The first note was the whole message. The second was the whole message minus its closing line, which the planner had quoted into that note's directives — the list of substrings meant for the bot rather than for storage. So the contact got the same paragraph twice, one copy a sentence shorter, under a reply saying "Added 2 notes" for one message.

This is the sort of bug that only becomes possible after a fix. While the model composed each note's body, two notes over one message were two different paragraphs — odd, but not obviously wrong. Once the note is the message, they are the same paragraph written twice.

Why first-wins is the wrong way to merge

The fix is mergeOverlappingNotes in apps/web/src/lib/messaging/process-session/apply/merge-notes.ts. It folds any notes on one person that share a seq into a single note. Two properties matter:

It's transitive. Notes are clustered by shared seqs, and a note overlapping two existing clusters absorbs both into one. So notes over [1], [3] and [1,3] collapse to one note whatever order the model emitted them in.

The directives are intersected, not deduped by first-wins. That is the load-bearing decision:

// Illustrative: a directive is only cut when EVERY note claiming that
// message agreed it was an instruction.
const [first, ...rest] = directiveSets;
const kept = first.filter((d) => rest.every((set) => set.includes(d)));

Dropping the later note and keeping the first one would have been one line shorter and quietly wrong. First-wins keeps whatever the first note happened to cut — so if the planner had emitted these two notes the other way round, the sender's closing line would have been thrown away for good, and nothing would have said so.

The reason to intersect is not a preference for merging. It's that plan order carries no information. The sequence in which a model emits notes is not a decision about the user's text. When two derived outputs disagree and the disagreement is evidence of nothing, you cannot break the tie by picking one — you have to resolve it toward the outcome that's recoverable. Keeping a line the user can delete beats deleting a line they will never know existed.

The tests read like the argument:

• "keeps a line one of them wanted cut, because the other called it content"
• "still cuts what BOTH of them called an instruction"
• "collapses a chain that only overlaps pairwise, whatever order it arrives in"

Incident two: the same rule, a different seam

A week later, the same rule broke somewhere the first fix couldn't see. One message was claimed by a person and by an unclear note at the same time. Two independent claimants, each parking its own "who is this about?" card. The user answered both, in good faith, and got two notes.

mergeOverlappingNotes covers notes within one person. A separate guard covered person versus person. Neither covered person versus unclear. Three overlapping guards, one uncovered seam — which is what happens when the actual invariant is plan-wide and every guard is local.

The fix we rejected

The obvious move is to put the invariant where the shape is: a superRefine on the plan schema that fails validation when a seq is claimed twice. It's four lines, it's declarative, and it is the wrong lever. The reasoning is preserved in a comment on the function that replaced it, in packages/core/src/schemas/batch-plan/index.ts:

Deliberately NOT a .superRefine on batchPlanSchema. That schema IS the structured-output format of the planning call, so a refinement failure throws inside extract(), the batch is recorded as failed, and the sender is asked to retry a plan the same input would very likely reproduce — their captured text held hostage to a planner slip.

Two things are going on there, and both generalise.

The cost of failing is asymmetric. The bug it prevents is a duplicated note: annoying, visible, deletable in one tap. The failure it introduces is a lost capture: the user's own words, gone, with a "please try again" that will very likely produce the same plan from the same input. Validation traded a small recoverable defect for a large unrecoverable one.

The refinement never reaches the model. This is the part that's easy to miss. A Zod refinement is a post-parse check. It cannot be expressed in JSON Schema, so it is not in the structured-output format the model is handed. It could never have steered the planner away from the duplicate — it could only have thrown after the fact. We would have paid full price and bought nothing.

The general rule we took from it:

Validation is the right lever when the producer can retry differently. A model handed the same input, the same prompt and the same schema will very likely produce the same output. If the retry is a coin flip you've already flipped, rejection isn't a safety mechanism — it's just deletion with extra steps.

Repairing instead of rejecting

So the duplicate check ships as a plain function rather than a refinement: duplicatePlanSeqs() walks the whole plan and returns the seqs with more than one claimant, counting claimants rather than testing set membership. It reports; it doesn't throw. Callers decide.

What the apply step does with that report is the interesting half. It strips the contested seq from every claimant and parks it as one question offering the claimants as answers — stamped with reason: "contested", so the capture log can tell a planner conflict apart from a message that was genuinely ambiguous to begin with. It does not pick a winner. Same premise as the merge: plan order is not evidence about which person the user meant.

Underneath, partial unique indexes on confirmations.source_item_id and notes.source_item_id give idempotency that no in-process guard can — a guard protects one pass over one plan; the index protects the table.

Worth noticing that the same premise produces two opposite behaviours:

Within a person, the safe default is keeping the text. Across people, there is no safe default, so the only honest move is a question.

The takeaways

1. A rule that lives only in the prompt is not enforced. It's a request. If an invariant matters, something deterministic has to check it — and if you only write it in the prompt, you'll believe it's covered.
2. Check the shape of your invariant check, not just its existence. A Set catches omissions and is structurally blind to duplicates. Ours had been silently half a check for months.
3. A Zod refinement is not a constraint on the model. It doesn't reach the emitted JSON schema, so it can't steer generation — it can only throw afterwards.
4. Don't validate what you can repair. Rejection is right when the producer can retry differently. A model given identical input will very likely repeat itself, so rejecting its output mostly converts a small defect into lost user data.
5. When two derived outputs disagree, ask what the disagreement is evidence of. If the answer is "nothing", don't break the tie — resolve toward the recoverable outcome, or ask the person who actually knows.
6. Local guards don't add up to a global invariant. Three guards covering pairs of cases left a seam that the plan-wide check closed in one function.