titlebert

A multi-head encoder classifier (backbone: jhu-clsp/mmBERT-small) for validating short free-text strings — built to attack the specific bottleneck found in the job-title-validation project: a 251,931-entry ESCO+O*NET+Wikidata lexicon resolves only 32.9% of real job titles at 24% precision. Taxonomy lookup alone does not solve job-title validation. This project replaces "is this string in my dictionary?" with "does this string read like a well-formed job title, and not like a person's name, a department, an address, or junk?" — a learned classifier rather than a lookup table.

Four heads share one encoder:

Head	Task	Classes
validity	is this real, well-formed text of some type	binary
type	what kind of string is it	multi-label: job_title, person_name, org_name, department, place, phone, email, address, product, other_junk
toxicity	does it contain profanity/abuse anywhere in the string	binary
language	which language (auxiliary regularizer + lets eval report worst-language, not macro-average)	23-way

type is multi-label on purpose: "Baker" is legitimately both person_name and job_title out of context, and pretending otherwise is how earlier engines in this domain (see csuite-title-ontology, titlevalidate) passed "Sales Department" or "New York" as valid job titles.

What this is not — corrections to the source design note

This project started from a design note recommending mmBERT + LLM-teacher distillation + TensorRT-INT8 serving. That advice is sound in general, but this VPS is 4 CPU cores, 7.6 GB RAM, no GPU (configs/INFRASTRUCTURE.md). Several of the note's assumptions do not hold here, and the code is honest about the gap rather than quietly pretending otherwise:

• Backbone: mmBERT-small (140M params), not mmBERT-base (307M). Measured on this box: ~45 items/s training throughput at --freeze-layers 20 (only the last 2 of 22 ModernBERT layers + heads trainable). mmBERT-base would be materially slower with no GPU to absorb it.
• No TensorRT, and int8 quantization was tried and abandoned. nvidia-smi doesn't exist on this host. Serving is ONNX Runtime on CPU — fp32 only. onnxruntime.quantization.quantize_dynamic was tried (default config, MatMul-only, per-channel, with and without quant_pre_process shape-inference preprocessing — see git history of serve/export_onnx.py) and every configuration produced a model whose outputs are near-constant regardless of input: a silent correctness failure, not a crash, caught only by comparing per-input logits against the fp32 ONNX model and the raw PyTorch checkpoint (which agree with each other). The speedup being chased was modest anyway (~50 items/s fp32 vs ~54 items/s int8, measured before the correctness bug was caught). serve/api.py and serve/cli.py only ever load the fp32 ONNX model; int8 export is available behind --try-int8-experimental for debugging only, clearly not wired into serving. This is nowhere near the "tens of thousands of items/sec on H100-class GPU" figures in the note — measured CPU throughput is in results/throughput_cpu.json.
• Hard-label teacher, not soft-label/logit distillation. The note assumes calling Qwen3-32B/Claude/GPT via API for soft labels with temperature. This VPS runs keyless-only (see project memory feedback-no-api-keys) and uses the locally installed Grok CLI (grok -p ... --json-schema ..., headless, structured JSON output) as the teacher instead. The Grok CLI returns constrained JSON, not logprobs, so training uses hard cross-entropy on ~1,300 Grok-labeled real-world strings (scripts/label_with_grok.py), not KL-divergence against a soft distribution. This is a real deviation from the note's recipe, not a hidden one.
• Thousands of training examples, not millions. No Wikidata dump ingestion (the SPARQL endpoint was unreliable from this host in testing); positives are drawn from data already fetched for job-title-validation (ESCO/O*NET/Wikidata occupation labels, capped at 45k to avoid drowning the minority classes) plus Faker-generated multilingual negatives (person names, orgs, addresses, phones, emails, places across 20 locales) and hand-curated department/product seed lists. See scripts/build_dataset.py for the exact recipe and results/dataset_stats.json for the class distribution actually produced.
• No Jigsaw/HateCheck toxicity corpora. Toxicity uses better-profanity's bundled wordlist, including mid-string injection ("Chief <profanity> Officer") to specifically target the exact failure mode a prior engine in this domain shipped with: a classifier that only reads the first/last token and lets mid-string profanity through (see project memory titlevalidate-project).

None of this makes the approach wrong — encoder + calibration + abstention against throwing an LLM at every row is still the right shape for this problem. It means the numbers in this repo are real numbers measured on this box on this data, not the note's GPU-datacenter numbers. Read results/*.json before quoting any figure.

Results

The numbers previously published in this section were wrong. A review on 2026-08-15 found three separate reporting defects and one broken evaluation set. The corrected numbers are below; the retraction is in RETRACTION.md with the evidence for each finding. The model weights are unchanged — only the measurement was broken, and the honest numbers are lower.

Measured on data/gold/eval_holdout_v2.jsonl (scripts/build_holdout.py): 814 rows, 57.7% of them not job titles, verified zero overlap with training data by exact and normalized text. Thresholds are fit on a 40% calibration half and scored on the disjoint 60% test half. Regenerate with make holdout && make eval && make calibrate && make verify.

Real-world job-title recognition (results/eval_gold_holdout_v2.json, real_only block — Faker-generated rows are excluded so structurally-trivial classes can't inflate it):

Metric	Published before	Honest	Why it changed
job_title precision	1.000	0.775	old holdout had zero job_title negatives, so precision could not fall below 1.0
job_title recall	0.933	0.932	unchanged
job_title F1	0.966	0.846	follows from precision
validity F1	0.989	0.946	old figure was also below the 0.940 "always say valid" baseline
toxicity F1	0.811	0.000	old figure was measured on the same four carrier templates used in training

The lexicon comparison this project was built to make still holds, but stated honestly: the prior job-title-validation lexicon resolves 32.9% of real titles at 24% precision, while this classifier gets 0.775 precision at 0.932 recall on real held-out text. That is a large, real improvement over taxonomy lookup — it is simply not the 1.000-precision result previously claimed, and the two evaluations still measure different tasks.

Choose your operating point. A single threshold hides the tradeoff, so results/calibration.json publishes the curve (operating_points per class). For job_title:

Target precision	0.80	0.85	0.90	0.95
Recall achieved	0.93	0.61	0.26	0.14

The shipped calibration targets 0.80 precision, the only point where the model is practically usable. Demanding 95% precision leaves it accepting 14% of real titles. Anything below threshold returns abstain, not reject — those rows are meant for human review, not rejection.

What still does not work (all reproducible, all gated by make verify, which exits non-zero today with 6 required failures):

• The toxicity head is memorization, not detection. make probe-toxicity holds the slur constant and varies only the surrounding words: 100% detection inside the four carrier templates used in training ("<word> Director", "<word> Manager", "Senior <word> Lead", "Chief <word> Officer") versus 4.2% in unseen carriers — the same word scores 0.979 in a trained carrier and 0.0015 in a new one. "Regional <slur> Coordinator" is currently accepted as a valid job title. This is the exact failure mode the README used to claim it had fixed.
• department and product remain F1=0.000 on non-leaked data, so "Sales Department" is accepted as a job title — the specific bug csuite-title-ontology was rebuilt to fix.
• other_junk recall is 0.125. Most junk strings are not caught by the type head (the validity head catches placeholders like See Remarks separately).
• validity barely beats its own baseline (0.946 vs 0.940 for "always say valid"). On this holdout the validity head is close to uninformative.
• phone/email/address score ~1.000 but only on Faker rows; they have no real-world support in this holdout, so treat them as unmeasured on real data.

person_name (F1 0.957) and place are the heads that genuinely work.

Synthetic validation set (results/eval_val.json, 10,203 rows, mostly ESCO/ O*NET/Faker-generated — easier than real text, read as an upper bound): validity F1=0.997, and per-class type F1 ranges from 0.999 (email) down to 0.000 for department and product — the two classes that only ever had ~300/~200 hand-seeded examples. This is the thin-class problem flagged in NEXT-STEPS.md, not a training bug: those two classes need real seed data, not more augmentation.

Toxicity: F1=0.811 on synthetic val (where profanity was injected mid-string on purpose) but 0/0/0 on the gold holdout — the 405 real SEC Form 4 titles simply contain no profanity, so there's no real-world signal to evaluate against yet. Treat the toxicity head as validated only on synthetic adversarial data until real toxic examples are collected.

Worst-language validity F1 (val set, languages with ≥20 examples): lowest is en-us at 0.985 (n=33, a Faker locale quirk overlapping with plain en), everything else ≥0.99, several non-English languages hit 1.000. No language is a systematic weak point in this run — see results/eval_val.json worst_language_f1 if that changes on retraining.

Calibration (results/calibration.json, target precision 0.95): phone, email, address hit 100% recall at their threshold (easy, structurally distinct classes); job_title gets 98.5% recall; place drops to 31% recall to hit 95% precision (genuinely harder — cities/countries overlap with org names and job titles in ways person/phone/email don't); product has threshold: null — no threshold reaches 95% precision, i.e. the abstain-everything case documented in NEXT-STEPS.md.

CPU throughput (results/throughput_cpu.json, batch=32, fp32 ONNX, no GPU): ~50 items/sec. See "What this is not" above for why int8 isn't in this number.

Repo layout

scripts/build_dataset.py   synthetic + reused-data corpus builder (no pandas, streams to JSONL)
scripts/label_with_grok.py Grok-CLI teacher labeling of the real-world SEC Form 4 slice
scripts/build_holdout.py   leak-free eval holdout; refuses to write if any row overlaps training
scripts/probe_toxicity_carriers.py  proves the toxicity head memorized its training templates
model/metrics.py           dict-returning PRF + no-negatives guard + majority baseline
model/decisions.py         accept/reject/abstain policy with reasons, toxicity veto, calibration
train/verify.py            pass/fail gates from results/GOALS.json; exits non-zero
serve/score_file.py        score a CSV/TSV/JSONL/TXT/Parquet column, local-only
tests/                     regression tests for the metric math and decision rules
data/external/jtv/         vendored ESCO/O*NET/Wikidata/SEC subset (~3MB) so a fresh
                            clone can run `make data` standalone -- no sibling repo needed
model/classifier.py        mmBERT-small backbone + 4 heads
model/device.py            cuda > mps > cpu device selection (training.py/evaluate.py/calibrate.py --device)
train/training.py             training loop (CPU or MPS/CUDA via --device)
train/evaluate.py          per-class precision/recall + worst-language F1
train/calibrate.py         temperature scaling + per-class threshold search + abstain
serve/export_onnx.py       fp32 ONNX export + throughput benchmark (int8 tried, broken, not shipped)
serve/api.py               FastAPI inference server (127.0.0.1 only)
serve/cli.py                batch scorer, no server needed
GUIDE.md               how to run each stage
NEXT-STEPS.md          open items, including what needs YOUR input

Read RETRACTION.md first — the results originally published here were wrong, and it documents each defect with reproducible evidence. Then GUIDE.md to run it, NEXT-STEPS.md for what's still open, and MAC_QUICKSTART.md if you're cloning this on a Mac (MPS device support, ships a ready-to-use trained model via Git LFS, no VPS-specific setup needed).

Before using this on real data, run make verify. It exits non-zero today.