Validation research handbook

Self-contained technical context for the job-title / contact-validation research line run on this host. Clone this one repo and feed its files to a new session. Do not expect that session to fetch sibling repos.

How to use it: you (or a script) clone or download this repo and paste or attach the files the routing table names. git.binary.ovh is the host; the session does not need to find it autonomously. What would break that workflow is this repo internally depending on a second repo you did not also hand over — which is why closed lists, decision trees, and eval construction live in the files, not behind "see RETRACTION.md in repo X."

Routing table — start from the question

Load 00-SHARED.md plus the two or three project files for the actual question. Do not paste the whole tree.

If the task is…	Read
Is this string a job title?	00-SHARED.md, projects/csuite-title-ontology.md, projects/titlevalidate.md (L1), projects/titlecheck.md
Same question, but as an ML classifier	00-SHARED.md, projects/titlebert.md, projects/titlebert-mac.md (never trained)
Does this title exist in the world (attested)?	projects/titlecheck.md
Could this title exist at this company (L2)?	projects/title-at-company.md, projects/contact-trust.md (L2), projects/titlevalidate.md (L2 caps)
Does this person hold it now (L3 / registry)?	projects/contact-trust.md, projects/titlevalidate.md (L3)
Row-level, keyless (title vs name/company/email/country)	projects/contact-title-validity.md, projects/job-title-workbench.md
Matching / retrieval / MAP, not validation	projects/multilingual-job-title-matching.md — and stop. Matching numbers do not transfer.
Taxonomy resolve / SOC codes	projects/job-title-validation.md
How do I evaluate any of this without repeating a retraction?	00-SHARED.md (measurement failures), projects/titlebert.md §10, projects/titlevalidate.md §10, projects/titlecheck.md §10
Domain-blind stats on a table (not titles)	projects/insight-miner.md, then projects/patternlab.md / projects/data-insight-recipes.md
What has not been tried?	UNEXPLORED-METHODS.md
Score ~65M unique title-field strings (precision-first, no LLM at serve, no HITL)	PLAN-65M-PRECISION-CASCADE.md — veto → attest → grammar → distill only HOLD
Product/webapp context, not research	OTHER-PROJECTS.md (one paragraph each, fenced)
Why the previous handbook repos were retired	DEPRECATED-DOCS-NOTICE.md

Status at a glance

project	status now	artifact tag
csuite-title-ontology	live grammar scorer	reproducible
job-title-validation	playbook + engine; v1 100% was contamination	historical / continuable
titlevalidate	L1/L2/L3 current; 78-rule legacy	mixed
titlebert	retracted 2026-08-15	continuable
titlebert-mac	abandoned before first run	historical
multilingual-job-title-matching	matching complete; validation unsolved (AUC 0.535 / 0.426)	reproducible matching, historical as validator
job-title-workbench	live teaching UI	reproducible
title-at-company	live teaching L2; L3 not implemented	reproducible
contact-title-validity	live workbench + LLM bake-off	reproducible
contact-trust	7/10 built, 2 rescoped, 1 design-only	mixed
titlecheck	live, QA: results overstated	reproducible
patternlab	superseded by insight-miner	historical
insight-miner	v0.15.1, 24/24 planted, 0 FD	reproducible
data-insight-recipes	public recipe menu	reproducible

Through-line (short)

Four earlier projects published numbers that did not survive re-measurement: circular canaries, holdouts with no negatives, leakage, template memorization, eval on a population that could not exercise the defect. Later work split the question into L1 (string) / L2 (at this company) / L3 (this person, now), put categorical vetoes in front of scores, derived vocabularies instead of training a "is this a title" classifier, and capped PLAUSIBLE so a plausibility notebook cannot be quoted as employment verification.

What is still unsolved: accept/reject from similarity (AUC 0.535 embedding, 0.426 lexical). Occupational surnames (Baker). Non-English titles outside ESCO. US statutory-officer coverage (there is no federal officer registry). A written labelling protocol with inter-annotator agreement — still the highest-value unbuilt item (UNEXPLORED-METHODS.md E1). The next planned attack on L1-at-scale is PLAN-65M-PRECISION-CASCADE.md (not built).

Standing rules (few, testable, each tied to a file)

1. Name the label. "Valid title" drifted. If you cannot point at a row of the table in 00-SHARED.md, you are about to compare incompatible numbers.
2. Print the degenerate baseline beside every metric. Always-INVALID on a negative corpus, always-VALID on a 97% positive holdout, random 1/|taxonomy| on matching. A number a stub matches is not evidence (titlevalidate.md, titlebert.md).
3. State how negatives were built, including whether they can exercise the defect you claim to have fixed (titlevalidate.md 0.4% vs 28%; multilingual-job-title-matching.md recombined vocabulary).
4. Vetoes are not a weighted sum. Profanity, placeholders, contact-junk, title-equals-name. Clamp or short-circuit (contact-title-validity.md, titlebert.md, titlecheck.md).
5. Silence is evidence only in-scope. L3 NOT_LISTED vs OUT_OF_SCOPE; SQL NULL is INDET not PASS (contact-trust.md, titlevalidate.md).
6. PLAUSIBLE is capped; IMPLAUSIBLE is not. No registry → cannot claim they hold it (title-at-company.md).
7. Do not train a classifier for "is this a title." There is no non-circular label source. Derivation, company resolution, and REVIEW ranking are the sanctioned ML uses (contact-trust.md Evidence Ladder).
8. Matching is not validation. MAP is a retrieval number (multilingual-job-title-matching.md).

How this repo is laid out

README.md                 this file
00-SHARED.md              definitions, shared mechanisms, retraction pattern
ARTIFACT-MAP.md           reproducible / continuable / historical + public URLs
UNEXPLORED-METHODS.md     not-tried families, with predicted break modes
PLAN-65M-PRECISION-CASCADE.md  next L1-at-scale plan (veto → attest → grammar → HOLD-only student)
published-blog/           copies of posts published to blog.binary.ovh
OTHER-PROJECTS.md         webapps and ML-learning repos, fenced
DEPRECATED-DOCS-NOTICE.md why workspace-context / job-title-validation-context died
projects/                 one file per research project, 12-point template
_research-notes/          raw agent extractions (provenance, not the handoff)

_research-notes/ is the working pile the files in projects/ were authored from. A new session should load projects/, not the notes, unless it is extending a file and needs the citations.

Process note

The plan for this repo was reviewed twice by an independent model (Grok) before project files were written. Round 1 forced: full closed-set inlining for rules, an honest ML vs rules bar, one file per project, a routing table, and reading unread sources first. Round 2 left one objection (public-host vs git.binary.ovh). The owner chose clone-and-feed: this repo is handed to the session; the session does not have to discover it. The search-only reader was explicitly dropped. Deep extraction of 14 project source trees followed; five finished before an account-level spend cap, the remaining six were completed in a subsequent session on the same host.