- Jupyter Notebook 63.5%
- Python 36.5%
brahm
0178a56290
The rule engine scored 0.45 on department names — worse than chance, on one of the most common kinds of junk in real contact data. A 25-item spot check suggested an LLM fixes that. This is the same eval set, seed and metrics, so the numbers are directly comparable. grok-4.6, zero-shot, one prompt, no fine-tuning: majority baseline 0.500 rule engine @0.5 0.813 AUC 0.947 rule engine @0.65 0.922 AUC 0.947 LLM judge 0.962 AUC 0.993 Department names: 0.45 -> 0.983. Places, people and companies: 1.000. The error list matters more than the headline. 17 of the 23 disagreements are job-board vacancy titles the LLM called DEPARTMENT or JUNK — "RETAIL", "OPERATION", "Digital marketing", "3D Rendering". The LLM is right and the ground truth is wrong: evaluate.py labels every string from the job board's title column as a real title, but a job board's title field contains junk too. That is this repo's own premise, applied to everyone else's data and then forgotten about ours. The positive class is contaminated and the LLM is being penalised for being correct. New failure mode for this project family: the four retracted predecessors all had evals rigged to look good. This one has labels wrong in the direction that makes a good method look worse. Same root cause — not reading the rows. Limits recorded: 95% self-consistency on re-run (rules are 100%), 0.82 items/sec = ~34h per 100k rows, single model/prompt/run, untested on a local model. Next steps this argues for: fix the benchmark labels before quoting anything further; distill LLM labels into a small fast classifier; cascade so the LLM only adjudicates the uncertain band; keep deterministic vetoes as rules. |
||
|---|---|---|
| data | ||
| models/JobBERT-v3 | ||
| notebooks | ||
| results | ||
| scripts | ||
| workbench | ||
| .gitattributes | ||
| .gitignore | ||
| README.md | ||
| requirements.txt | ||
contact-title-validity
Question: given a contact row — company, country, first name, last name, and maybe email and phone — is the job title on that row valid or invalid?
This is phase 2. Five prior repos already tried to answer "is this string a real job title" using the title text alone, measured honestly, and each found the same wall:
| Repo | What it found |
|---|---|
| job-title-validation | A 251,931-title lexicon from ESCO+O*NET resolves only 32.9% of real titles at 24% precision — taxonomy lookup alone doesn't solve this |
| titlevalidate | 28% false-accept rate on English company names; reads only 2 edge tokens so mid-string junk passes |
| csuite_title_ontology | Grammar/role-head rules catch "Sales Department" and "New York" that string-hygiene missed |
| titlebert | A trained toxicity head memorized 4 carrier templates (100% vs 4.2% detection on templates it hadn't seen) — looked solved, wasn't |
| multilingual-job-title-matching | JobBERT-v3 is the best matcher (title-to-title similarity) measured against TalentCLEF, but matching two titles to each other is a different task from deciding if one title is real |
The wall, stated plainly: no one publishes a golden "this is not a job title" dataset, and title text alone under-determines validity. "Regional Coordinator" is fine at a logistics company and suspicious paired with a slur. "Chief Technology Officer" is fine at a 4,000-person company and suspicious at a 2-person one. The string can't tell you that. The row can.
Start here: the workbench notebook
notebooks/workbench.ipynb — an interactive instrument for
developing and testing job-title validation logic. Load a progression (a named sequence
of titles), play it at a tempo, watch each title light up with a verdict and the reasons
behind it, spot what's wrong, change a rule or a weight, play it again. The value is in
the loop, the same way it is when you practise a chord progression.
Every cell was executed on the VPS before being committed, so the outputs you see on git.binary.ovh are real — no need to run anything to read it.
What's in it:
- Multi-signal input — job title (required) plus optional company, country, first/last name, email. Every field is optional and the engine degrades gracefully.
- Verdict + confidence with a full explanation panel: every rule that fired, its weight, its reason, and the arithmetic that produced the number.
- Nearest official titles from ESCO, O*NET and UK SOC, by both lexical similarity and JobBERT-v3 embedding, shown side by side because they fail differently.
- 14 presets grouped by kind of difficulty — clean, messy-real, companies, org units, places, people, placeholders, gibberish, credentials, contact junk, prose, offensive mid-string, multilingual, genuinely-ambiguous.
- Batch player with tempo (bpm), play/pause/step/loop, a filmstrip highlighting the active title, and an editable sequence.
- Live ipywidgets panels, including weight sliders that re-score a whole progression as you drag them.
- An honest evaluation (section 8) against negatives the rules have never seen.
git clone https://git.binary.ovh/brahm/contact-title-validity.git
cd contact-title-validity && git lfs pull
pip install -r requirements.txt
python3 scripts/build_taxonomies.py # ~40s
python3 scripts/build_embeddings.py # ~11 min on 4 vCPU (gitignored output)
jupyter notebook notebooks/workbench.ipynb
Mac users: the code auto-detects Apple Silicon and uses the mps backend with no edits.
Full steps are in section 0 of the notebook — written from the docs, not verified on
real Apple hardware, and labelled as such.
What it measures, honestly
On a 600-row eval set (300 real job-board vacancy titles, 300 negatives drawn from company names, person names, places, org units and junk), with 0% overlap between the negatives and the word lists the rules consult:
| Metric | Value |
|---|---|
| ROC AUC | 0.947 (0.5 = coin flip) |
| Accuracy @ 0.5 | 0.813 (majority baseline 0.500) |
| Accuracy @ 0.65 | 0.922 — the default 0.5 cut is a bad operating point |
| Weakest slice | org units, 0.45 — worse than chance |
Two things that number-set is not: calibrated (the confidence has no probabilistic meaning, only its ordering is informative), and validated on real vendor contact data (none can be shared with an AI system, by standing constraint).
Three bugs the notebook found in itself
Written up in the notebook where they happened, because they're the useful part:
- The eval was circular. First version scored accuracy 0.950 / AUC 0.991 — with
78% of its negatives literally present in a list the rules check membership in.
Same trap that got four predecessor repos retracted, reproduced from scratch inside a
repo written to avoid it. Fixed with a hash-based train/holdout split of every word
list, plus a permanent
leakage_report()self-check. - Profanity was outvotable.
"Regional fuck Coordinator"scored 0.485, "uncertain" — the profanity rule's −1.00 was nearly cancelled by +0.60 for the real role head coordinator and +0.37 for a decent taxonomy neighbour. Fixed with a veto mechanism for categorical rules. - Independent-looking evidence wasn't. An exact taxonomy hit guarantees a ~1.0 nearest neighbour and a role-head match, so one fact was being counted three times. Fixed by having dominant signals suppress the ones they imply — visibly, marked in the output rather than silently dropped.
The full option menu
Per list all options, don't choose for them — every approach below is listed, not filtered down to a recommendation. Some are already partly built in prior repos and can be reused; some are untried.
A. Sourcing invalid-title negatives (the missing golden dataset)
- Adjacent-category negatives — company names, place names, person names, department names used as negatives. Already the basis of titlevalidate/titlebert. Known failure mode: too easy, doesn't teach the hard boundary, and a model trained on a narrow negative set (titlebert: 4 carrier templates) memorizes the template instead of the concept.
- Synthetic corruption — typo injection, truncation, token shuffle, mixed-script garbage. Cheap, catches only "obviously broken."
- Adversarial LLM generation — have an LLM (Grok, no token limits per standing instruction) generate deliberately plausible-sounding fake titles across many templates and languages, explicitly labeled synthetic, not ground truth. Fixes titlebert's single-biggest weakness (too few templates).
- Grammar-violation mining — reuse csuite_title_ontology's role-head parser: anything that fails the grammar is a self-generated negative, no external source needed.
- Tangential real dataset — Kaggle EMSCAD (17,880 job postings, 866 fraudulent, University of the Aegean). This is fake posting detection, not fake title detection — different task — but the title field shows a real pattern (scam postings lean "Data Entry," "Home Based," "Earn Daily") worth mining as one signal, not a full solution.
- Field-level gap, confirmed not assumed: this list is a set of workarounds, not a substitute for a real negative-title benchmark. If future research (the papers below) surfaces one, it goes at the top of this list.
B. Row-level cross-validation (the new angle — title is not judged alone)
This is the part none of the five prior repos tried, because none of them had company/country/name context to test against.
- Title–company plausibility — resolve the company via a keyless business registry (GLEIF, OpenCorporates, brreg NO, recherche-entreprises FR — same sources titlevalidate already verified keyless), then check title-to-company-size / title-to-sector consistency. "Chief Technology Officer" at a 2-person shell company is a plausibility outlier even though the string itself is a perfectly valid title.
- Title–country/language consistency — title script/language should match the contact's stated country (ESCO/multilingual-job-title-matching's language coverage is the tool for this). A Cyrillic title on a Japan-country row is a strong invalid signal regardless of what the string says.
- Title–email domain cross-check — if the email domain matches the company's own domain (from the registry lookup in #7), and the company's public website (team/about page) can be scraped for a title match — a real OSINT triangulation signal, entirely keyless. Weak/noisy but adds independent evidence.
- Row-completeness as a prior — rows missing email and phone and have a generic-sounding title are lower-trust a priori; this is a cheap Bayesian prior to combine with whatever the string/row classifier outputs, not a classifier by itself.
- Confidence tiering, not a boolean — per contact-trust's already-validated approach (Mickey Mouse @ Equinor scored 0.05, a real officer scored 0.95): the output should be a trust score combining title-string plausibility + row cross-validation, not a single valid/invalid bit. This matches what contact-trust already proved works for the adjacent fake-profile problem.
C. What ML/embedding approach to use for the string half
- Fine-tune JobBERT-v3 for validity, not just matching — it's the best multilingual title encoder measured so far (see multilingual-job-title-matching's TalentCLEF numbers). Bi-encoders are trained to place similar titles near each other, not to separate valid from invalid — that's an unproven extension, to be measured, not assumed to work.
- Classification head on top of a frozen encoder vs full fine-tune — the former is cheap and testable first (matches this VPS's memory-conscious-coding constraint), the latter is the fallback if the frozen-encoder signal turns out too weak.
- Contrastive learning with hard negatives — pair valid titles with the row-level negatives from section B instead of only string-level negatives from section A; this is the standard approach in the papers being reviewed (see below) for teaching a boundary that a template-memorizing classifier (titlebert's failure) can't shortcut.
Country/global job-title registries (verified, keyless, downloadable)
Already in use (multilingual-job-title-matching): ESCO (EU, 27+ languages), O*NET (US). Not yet pulled into this repo:
| Country/region | Source | What it adds over ESCO/O*NET |
|---|---|---|
| UK | ONS SOC2020 Vol 2 | Index of thousands of real observed job titles → SOC code, CSV |
| Canada | StatCan NOC 2021 | Same title-index structure |
| France | ROME (France Travail) | 14,301 "appellations" — real-world title variants per occupation |
| Australia/NZ | ANZSCO (abs.gov.au) | Shared occupation classification + title index |
| Germany | KldB 2010 (Bundesagentur für Arbeit) | Occupation classification + title index |
| India | NCO-2015 | Exists but PDF-only — needs OCR/parsing, lower data quality |
| International | ISCO-08 (ILO) | Parent taxonomy all the above map to |
UK SOC and French ROME are the two highest-value adds — both give real-world title variants, not just canonical occupation names, closer to what messy contact data looks like than ESCO/O*NET's cleaner label sets.
Also in this repo: gpriday/job-titles
data/gpriday-job-titles/jobs.parquet —
65,248 deduplicated valid titles, combining ESCO + O*NET + OSCA (Australia,
not yet in the table above), published by
gpriday on HuggingFace.
Read data/gpriday-job-titles/DISCLAIMER.md
before using it — short version: valid-titles-only (no negatives), English-only,
EU/US/AU-biased, ~35% of raw titles were AI-merged together, mixed licensing
(EUPL + US public domain + CC-BY 3.0 AU).
Models
TechWolf/JobBERT-v3 — 0.3B params, XLM-RoBERTa base, 1.1GB, MIT licence, the
best multilingual title encoder measured in multilingual-job-title-matching.
Checked into models/JobBERT-v3/ via git-lfs — cloning this repo gets you the
weights, no HuggingFace account or download step needed.
git clone https://git.binary.ovh/brahm/contact-title-validity.git
cd contact-title-validity
git lfs pull # if your git-lfs isn't set to auto-pull on clone
python3 -m venv venv && source venv/bin/activate
pip install torch --index-url https://download.pytorch.org/whl/cpu # CPU build, works on a MacBook
pip install sentence-transformers
python3 -c "
from sentence_transformers import SentenceTransformer
m = SentenceTransformer('models/JobBERT-v3', device='cpu')
emb = m.encode(['senior software engineer', 'ingeniero de software'], normalize_embeddings=True)
print(float(emb[0] @ emb[1])) # cosine similarity
"
On a MacBook (Apple Silicon): replace the pip torch line with a plain
pip install torch (the default wheel includes MPS/Metal support) and pass
device="mps" instead of "cpu" in SentenceTransformer(...) for a GPU-speed
local run — everything else above is identical, no CUDA or cloud needed.
git lfs pull needs git-lfs installed locally first: brew install git-lfs
on macOS, then git lfs install once per machine.
JobBERT-v2 (0.1B, 425MB) exists too but isn't packaged here — v3 is state of the art per TalentCLEF and the repo's own measurements, so it's the one to build on unless a specific reason comes up to compare against v2.
Not near a terminal, or don't want git-lfs? The same files are also served as plain HTTPS downloads directly from this VPS (no HuggingFace, no third party):
download.binary.ovh/models/jobbert-v3/
mkdir -p models/JobBERT-v3 && cd models/JobBERT-v3
curl -O https://download.binary.ovh/models/jobbert-v3/model.safetensors # 1.1GB, the weights
curl -O https://download.binary.ovh/models/jobbert-v3/config.json
curl -O https://download.binary.ovh/models/jobbert-v3/config_sentence_transformers.json
curl -O https://download.binary.ovh/models/jobbert-v3/tokenizer.json
curl -O https://download.binary.ovh/models/jobbert-v3/tokenizer_config.json
curl -O https://download.binary.ovh/models/jobbert-v3/special_tokens_map.json
curl -O https://download.binary.ovh/models/jobbert-v3/modules.json
curl -O https://download.binary.ovh/models/jobbert-v3/sentence_bert_config.json
mkdir -p 1_Pooling 2_Asym && curl -o 1_Pooling/config.json https://download.binary.ovh/models/jobbert-v3/1_Pooling/config.json
# 2_Asym/ has two Dense sub-folders — browse https://download.binary.ovh/models/jobbert-v3/2_Asym/ to grab those too
Same SentenceTransformer('models/JobBERT-v3', device='cpu') load code as above
works once these are in place — the directory layout is identical either way.
STATUS
Scaffolding only, session started 2026-08-19. Not yet done:
- JobBERT-v3 packaged into this repo (git-lfs) or a sibling repo for offline clone
- The 10-15 research papers — waiting on the user to provide the list/links; whichever of them aren't already covered by RESEARCH.md in multilingual-job-title-matching or job-title-validation need a fresh read
- Decision matrix scoring options A1-6 / B7-11 / C12-14 against each other, once the papers are in — per list all options, don't choose for them this will be published as a full matrix, not collapsed to one recommendation
- Real contact-row test data — none exists yet and none can be sourced from the user's real vendor data (feedback-vendor-data-no-ai-sharing — this pipeline must run entirely on synthetic/public data, never the user's real 3P-vendor rows)