PII Bench (piibench)

What it measures. Detection of personal data that has no fixed format — PERSON ORGANIZATION LOCATION ADDRESS DATE_TIME — in text shaped the way outbound traffic to AI assistants is shaped: JSON request bodies, XML, SQL, CSV, key=value pairs, log lines, Markdown tables and OCR output, as well as prose. Public PII benchmarks score prose; a model that is accurate on prose can still miss the same name when it sits in a JSON string value, and a prose-only benchmark cannot show it.

What it does not measure. Identifiers with a fixed format — email addresses, payment-card, IBAN and passport numbers — are excluded from scoring, because a regular expression resolves them exactly. A prediction that lands on such an identifier is not counted as a false positive. Touch rate reports how often a model marks those characters anyway; it is reported, never ranked. A touch rate of — means the dataset contains no excluded identifiers, so it was not measured — it is not zero, and the average skips it.

Protocol. 3,500 rows: 500 from each of seven third-party datasets (TAB, GermEval 2014, FactRuEval-2016, MEDDOCAN, IAHLT, Gretel, Nemotron), serialised into twelve carrier formats with every gold offset re-mapped so each span still covers the same substring. Precision, recall and F1 are computed over characters and are label-agnostic: a character counts as correctly detected when it is predicted as PII and lies inside a gold span of any scored type, so no entrant's label names are re-interpreted. Scores are micro-averaged within each dataset and then averaged across the seven, so no single dataset dominates. 95 % confidence intervals are percentile bootstrap intervals over rows, resampled within each dataset. Entrants are ranked by average F1 only, each at a pinned model revision, with the host and numeric precision of every run recorded beside its result.

Data and code. The rows, gold spans, per-dataset licences, the scoring code and every entrant's raw predictions are published as ScienceSoft/piibench. This page computes no metric of its own; it renders those result files. How to run the benchmark, below, reproduces any row or scores a new model.

Conflict of interest. piibench is built and published by ScienceSoft, which also develops one of the models scored here; rows prefixed scnsoft- are ScienceSoft's. On this board scnsoft-pii-encoder ranks first: its average F1 is +0.0233 above that of the strongest independent entrant, nvidia-gliner-pii (paired bootstrap over rows, 2,000 resamples, 95 % CI [+0.0147, +0.0322]). Weigh that knowing who published the board.

What that result is evidence of. It concerns one artefact — the published weights, revision 1c1e5be8cbe359c0, trained with one random seed — not the training recipe. The recipe was trained with three seeds, and each run was scored through the same production inference engine (the -engine-raw path). Against the same reference, only one of the three clears it: seed 1234, the released model, +0.0192 [+0.0105, +0.0278]; seed 4321, +0.0057 [−0.0028, +0.0145]; seed 7777, +0.0011 [−0.0080, +0.0103]. The released model is therefore the upper end of the recipe's observed spread, and another training run of the same recipe would not be expected to be separably ahead of the reference. The two other seeds are listed under Seed variance, reported and never ranked, and their weights are published on the seed-4321 and seed-7777 branches of the model repository. The margin is also specific to this corpus of 3,500 rows.

Rows that are one model. scnsoft-pii-encoder, scnsoft-pii-encoder-engine-raw and scnsoft-pii-encoder-engine-shipped are one artefact, the same weights, scored three ways: through the Hugging Face ONNX path used for every other entrant; through the product's Rust engine before its post-filters (-engine-raw); and after them (-engine-shipped). The post-filters trade recall for precision on the product's own traffic, so -engine-shipped scores lower here. The engine rows describe the product's pipeline around the model, not a model on their own; otherwise this benchmark scores one model at a time, never a full product.

Steering. A dataset is steered for a model if that model's training was shaped by examining it — the difference between a measurement and a rehearsal. Four of the seven datasets steered ScienceSoft's model: TAB, FactRuEval, MEDDOCAN and IAHLT each motivated a change to its training corpus (TAB's mark is kept although the change it motivated was later switched off — DESIGN.md §4). GermEval, Gretel and Nemotron are unsteered. Steering for other entrants is not known and not claimed either way. Read ScienceSoft's placement with this in mind.

Overall ranking

Ranked by average F1 over the seven datasets. Licence, model revision and notes are shown under each entrant's name.

95 % CI: percentile bootstrap over rows, 1000 resamples.

By carrier — the axis this benchmark exists for

By dataset

† GermEval and FactRuEval annotate no dates; any date a model finds there counts against it, for every model.

By language

Seed variance — reported, never ranked

Further training runs of the released model's recipe that differ from it only in the random seed. They are not entrants: they are listed so that the released model's result can be read against the spread of its recipe, and they appear in no ranked table on this page. The released model on the same path, scnsoft-pii-encoder-engine-raw, scores 0.7725.

Speed — inference latency, batch size 1

Median and p95 milliseconds per row, timed one request at a time after a discarded warmup — the shape this product actually runs in (an agent inline in a request path), not batch throughput. Reported and separately sortable; it never reorders the F1 ranking above, and a model this benchmark cannot yet time shows an em dash, never a zero.

Entity-level — SemEval-2013 exact / partial (supplementary, not ranked)

Not run, and why

The benchmark names failures rather than omitting them. A model absent with no explanation is indistinguishable from one quietly dropped for scoring badly.

Hardware and software

Inference ran on CPU (accelerators present but unused).

• Apple M4 Max, accelerator mps, device cpu, transformers 4.57.6, onnxruntime 1.19.2

How to run the benchmark

Everything needed is in the dataset repository: the rows, the scoring harness and every published result. Python 3.9 or later, and the Hugging Face command-line tool (pip install -U huggingface_hub) for step 1.

1. Download the benchmark and install the harness.

   hf download ScienceSoft/piibench --repo-type dataset --local-dir piibench
   cd piibench
   pip install -e ".[predict]"

2. Run a model over the published rows. The default pipeline is Hugging Face token-classification with aggregation_strategy="simple"; --pipeline onnx, gliner and presidio are also available and additionally need onnxruntime, gliner or presidio-analyzer respectively. Steps 3 and 4 need only the base install.

   piibench predict --rows data/rows.jsonl --model <hf-model-id> --revision <commit> --out preds.jsonl

3. Score the predictions. Writes result.json, a copy of the predictions and meta.json (host, precision, library versions).

   piibench score --rows data/rows.jsonl --preds preds.jsonl --model <name> --out-dir results/<name>

4. Compare with any published entrant on the same rows — a paired bootstrap on the difference in average F1:

   piibench compare --rows data/rows.jsonl --a preds.jsonl --b results/nvidia-gliner-pii/preds.jsonl --iterations 2000 --seed 0

Reproducing a published row. Run step 2 with that row's model id and the revision recorded in results/<slug>/result.json (_provenance.model_revision), or re-score its published preds.jsonl directly with step 3. Scores are deterministic for a given rows file; each result records that file's SHA-256 as _provenance.rows_sha256 — cite it with any number you report. Rows produced by ScienceSoft's Rust engine (-engine-*) cannot be regenerated with this harness; their predictions are published so that they can be re-scored.

Submitting a model. Open a pull request on the dataset repository that adds results/<name>/ as written by step 3, with the model id and pinned revision.

Generated from result files; this page computes no metric of its own. Framework, datasets, licences and the honesty rules are in the dataset repository.