The problem
"We protect your privacy" is unfalsifiable as written, which is why every product in the category says it. To make it mean anything, it has to name an attacker, a corpus, an attribute, and a number.
The design
- Paired. Each of 100 messages is attacked twice, once raw and once rewritten. Pairing removes the variance that comes from some messages simply being more revealing than others, which is most of the variance.
- Three attacker tiers, so the number degrades gracefully as attackers improve rather than being true only against a weak one. The headline is quoted against the strongest.
- A confidence interval, from a 10,000-sample bootstrap. A single point estimate on n=100 invites exactly the overreach the work exists to prevent.
- Utility scored separately. Any rewrite can score perfectly on leakage by destroying the message. Without a utility number on the same run, the privacy number is meaningless.
The result
| Measure | Raw | Rewritten | Note |
|---|---|---|---|
| Overall attribute inference | 87.5% | 47.5% | Paired delta 40.0 points, 95% CI 35.6 to 44.5 |
| Location | 83% | 3% | Strongly protected |
| Occupation | 88% | 60% | Weakly protected, see below |
| Message utility | 100 | 88.6 | Independent judge, out of 100 |
Correct attribute guesses by the strongest of three attacker tiers, n=100 paired, drawn to scale from the run. The red line is the finding: occupation is the category the rewrite could not protect.
The n=100 run reproduced an earlier n=50 result (40.0 against 41.1 points) with a confidence interval roughly 40% tighter, which is the outcome that told me the measurement was stable rather than lucky.
The finding that mattered
Location collapsed. Occupation barely moved, and that is the result worth having.
Occupation cues are not sitting in the message as removable tokens. They are the message. Someone asking how to phrase a difficult note to a patient's family has disclosed their profession in the shape of the question, and a rewrite that removes it has answered a different question than the one they asked.
So the honest conclusion was not that the anonymizer needed more work. It was that identity signal and usefulness are partly the same substance, that the tradeoff has a floor, and that the published claim should name the categories that clear it rather than gesture at all of them. That distinction went straight into the claim ladder and cost us a line of live copy.
The refinement that followed: the re-identifying part of occupation is the specific employer, which is a removable token and is now stripped. The generic role is question-intrinsic and stays. Separating those two took the measurement, not an argument.
What I would do differently
- Run against the shipping anonymizer, not a stand-in. The harness drove a proxy model rather than the on-device path that actually ships. That makes the honest phrasing "measured the approach", not "measured the product", and it is the single biggest credibility gap in the number.
- Use donated text. The corpus was written and attacked by language models. Model-authored messages are plausibly more uniform, and more polite, than the real ones. Real text needs a consent flow first, which is why it has not happened rather than an oversight.
- Close the judge-agreement loop before quoting utility. The 88.6 rests on one LLM judge. A sample sheet for human rating exists and is unrated, so the figure has nothing checking it. I would rather have a worse number I can defend.