Measurements
How well does the redaction actually work?
Veldrun's whole promise is that your documents can be searched by an AI without the AI ever seeing who they are about. That promise is only worth what the measurement behind it is worth — so here is the measurement, the method, and the parts that are not finished.
Last measured 26 August 2026. Every figure names the corpus it came from.
The headline
On a public benchmark nobody at Veldrun annotated
The Text Anonymization Benchmark is a public research corpus of 1,268 European Court of Human Rights judgments, annotated over two months by twelve law students at the University of Oslo — people who have never heard of us, working to published guidelines. We ran our shipped configuration against its held-out test split: 127 judgments the system had never been tuned on, each annotated by an average of more than four annotators.
Direct identifiers removed
Names, identity numbers, contact details — the things that identify a person on their own. 286 annotated spans.
Quasi-identifiers removed
Nationality, occupation, place, dates — things that identify in combination. The harder half, and the honest one.
Why both numbers are on this page. Quoting the first without the second is the most misleading thing we could do. Direct identifiers are the easier problem and we do very well at them; quasi-identifiers are where residual re-identification risk actually lives, and 87.6% means roughly one in eight annotated quasi-identifier spans survived. Any vendor showing you one number in this category is showing you the flattering one.
Method
What we counted, exactly
A number like "99.3%" is meaningless without the rule that produced it. Ours:
- Take every span the human annotators marked as an identifier.
- Run the document through Veldrun's shipped redaction.
- Ask a mechanical question of the output: does the annotated value still appear in the text? Exact string match, plus partial matches of four characters or more.
- Recall is the proportion of annotated spans that no longer appear.
We call this a leak-based measure, because it asks the question a person handing over a document cares about — did the name survive — rather than whether our span boundaries agreed with the annotator's.
These figures are not comparable to published TAB scores. The benchmark's authors define their own evaluation framework, and it asks a different question from ours.
Theirs scores an entity as protected only when every mention of it was masked, and micro-averages across up to twelve annotators. Ours asks whether the annotated string survives anywhere in the output. Those produce different denominators on the same documents — the benchmark counts 1,157 direct-identifier entities in this split where our scorer counts 286 annotated spans — which is the clearest possible demonstration that the two numbers are not the same measurement.
Same documents, different ruler. We are not claiming to beat any published system, any leaderboard position, or any named competitor. If you see our number placed beside a score from the TAB paper anywhere, that comparison is invalid and did not come from us.
And a caution that applies to our own number. Every measure on this page, including ours, asks whether text was removed. That is not the same as asking whether a person can still be worked out. Research published in 2026 found a system scoring 0.997 on TAB's direct-identifier entity recall still left roughly two-thirds of personal information inferable from what remained (Oh et al., arXiv:2604.21211).
We cite this against ourselves deliberately: our scorer is string-based too, so it is subject to the same overstatement. High span-removal numbers are necessary and not sufficient, and anyone selling you one — us included — is showing you a floor, not a guarantee.
Why the held-out split matters
Benchmarks have a development split and a test split. The development split is what you tune against — thresholds, word lists, fixtures — so a score on it partly measures how well you fitted it. The test split is only meaningful while it has never been used that way.
The 99.3% figure was taken on 24 August 2026, while our test split was still untouched. Our development-split score is higher. We publish the lower, older number, because it is the one that estimates how the system behaves on documents it has never seen — which is every document you own.
And we will not print 100%. A later run did produce that figure on direct identifiers. We are not putting it on this page: by then the split had been drawn from for test fixtures, which spends exactly the property that made it meaningful — and a round 100% claim about a detection system is not one any serious evaluator would accept, nor one we would want you to rely on.
Household documents
Court judgments are not your kitchen drawer
The benchmark is court prose. Real working archives are letters, forms, invoices, scanned agreements and photographed correspondence, so we maintain our own annotated corpora of those shapes too.
| Measure | Corpus | Result |
|---|---|---|
| Direct-identifier recall | Unknown names — people in no registry, 40 documents | 100.0% |
| Direct-identifier recall | Everyday documents with real names, 168 documents | 99.1% |
| Person-name precision | Everyday documents, 168 documents | 91.2% |
| Ordinary words wrongly hidden | Clean prose containing no identifiers at all | 79 |
That last row is the cost of the first three. Veldrun deliberately over-redacts when a span is ambiguous, because the cost of hiding an ordinary word is that you have to click to reveal it, and the cost of missing a name is that a name leaked. On a corpus with no identifiers in it at all, that policy hides 79 words it did not need to. We think that is the right trade; you can see the size of it rather than take our word for the direction.
Regulatory mapping
HIPAA Safe Harbor, category by category
Safe Harbor is the de-identification method defined in 45 CFR §164.514(b)(2): remove eighteen categories of identifier. We map every one to the detector that removes it and the test that proves the mechanism, and in several places we remove more than the rule requires — whole postcodes rather than the census carve-out, all stated ages rather than only those over 89, and the year of a date where the rule permits keeping it.
One category is not fully covered by default, and you should know which. Category (C) covers date elements. Every date detector that ships enabled requires a month, so a day element written without one — "seen again on the 14th", "ongoing since the spring" — is not removed in the default configuration. This is pinned by a test that asserts the gap, so it cannot quietly become true or quietly become false.
What this is not. There is no such thing as HIPAA certification, and we do not claim it. Safe Harbor is a method, not a badge. We are not SOC 2 audited — that is an audit of an organisation, not a property of software, and no audit has taken place. Compliance is something an operator achieves with practices; software can only help.
Limits
What we do not claim
- Not "100% safe". The measurements above are the claim, and they are below 100%.
- Not code-signed yet. The installer's SHA-256 is published beside the download and verified after transfer, but the binary does not yet carry an Authenticode signature.
- No third-party penetration test. An internal adversarial security review ran in July 2026 and fixed two high-severity and one medium-severity finding. An external test is a stated gate before anyone is charged, and it has not happened.
- One install in production. Every figure here comes from public corpora or from the developer's own archive. Veldrun has no customers yet.
- Windows only, and multi-machine sync has been built and tested in pieces but never run end to end on two real machines.
We publish this list because a buyer handing over custody of their clients' records is buying judgement as much as software. A vendor that will tell you what is broken is demonstrating the thing you are actually purchasing.
Reproducibility
Check it yourself
The benchmark corpus is public and MIT-licensed. These are the exact inputs behind the headline figures.
- Corpus
- TAB echr_test (held-out split), 127 documents
- Corpus SHA-256
- cd0f0f15f84a8739…
- Configuration
- rules + GLiNER ONNX encoder, no LLM (the shipped default)
- Metric
- leak-based recall, strict string matching, partial threshold 4 characters
- Measured
- 2026-08-24 (direct) / 2026-08-26 (quasi)
- Source
- github.com/NorskRegnesentral/text-anonymization-benchmark
One thing this page does not yet have, and should. The figures above carry no confidence interval. They rest on 127 documents, so the honest error bar — bootstrapped over documents rather than over spans, because spans within one judgment are not independent — is materially wider than the span count alone suggests. Roughly ±4 points at this sample size, not the ±1 you would get by treating every span as its own trial. We would rather say that than print a bare number and let you assume otherwise.
The next improvement to this page is running the benchmark authors' own scorer alongside ours and publishing both, so our results sit directly beside the published literature instead of near it.
Text Anonymization Benchmark (TAB), Copyright (C) 2021-2026 Norsk Regnesentral, MIT License. Veldrun is not affiliated with Norsk Regnesentral or the University of Oslo. These results were produced by us and have not been reviewed or verified by the TAB authors — no such verification process exists. Nothing here should be read as endorsement by them.
What this does not mean. TAB is English-language court judgments. Your engagement letters, contracts, medical notes and tax workpapers are a different domain, containing identifiers this benchmark does not cover. Read these figures as evidence that the system works, not as a prediction about your own documents — which is why the everyday-document corpora above exist alongside it.
Questions about any of this?
If you evaluate privacy tooling for a living and something here does not add up, we would rather hear it. The community board is public and we answer in the open.