multilingual-job-title-matching

Matching and validating job titles across languages — recognising that software developer, développeur de logiciels, Softwareentwickler and 软件开发员 are the same occupation, and (harder) that Acme Holdings LLC is not an occupation at all.

Everything here is measured, not claimed. Every notebook was run, its output is saved in the file, and every accuracy number sits next to the baseline it has to beat. Where a result is mediocre or a method fails, that is written down.

---

Reading the notebooks

Forgejo (this repo's host) has no built-in Jupyter renderer — clicking a .ipynb file in the web UI shows the raw JSON source, not the notebook. That's a host limitation, not a problem with the files: every notebook under notebooks/ is a real, fully-executed .ipynb (code cell → its actual output cell, in order), and opening one in Jupyter Lab, JupyterLab Desktop, or VS Code after cloning renders it exactly as expected.

To read one without cloning anything, use the rendered HTML mirror:

document.binary.ovh/multilingual-job-title-matching

#	Notebook	Rendered
01	Data exploration	HTML
02	Baseline: lexical / taxonomy matching	HTML
03	Multilingual embedding matching	HTML
04	Evaluation	HTML
05	Retrieval / ANN on TalentCLEF	HTML
06	Classification vs. taxonomy codes	HTML
07	Gender bias / fairness	HTML

These HTML files are a generated mirror (jupyter nbconvert --to html), not checked into git — the source of truth is always the .ipynb under notebooks/.

---

Results at a glance

Retrieval — scored on TalentCLEF 2025, a public benchmark we did not build

The answer key was made by the competition organizers, not by us. MAP, validation split, with the number of queries and two floors stated for every language.

Language	queries	random	lexical fuzzy	MiniLM	mult-e5-base	JobBERT-v3
English	105	0.0019	0.1957	0.4803	0.4836	0.6161
German	203	0.0009	0.1597	0.2517	0.2543	0.4145
Spanish	185	0.0012	0.1588	0.3375	0.3387	0.4669
Chinese	103	0.0025	0.2886	0.4159	0.4337	0.5698

These fall inside the ~0.29–0.63 MAP band that published TalentCLEF systems report — nothing here is implausibly high. Our MAP/MRR/nDCG/P@k implementations match ranx, the library behind the organizers' official scorer, to 0.00e+00 (scripts/verify_metrics_against_official.py).

Two results worth noticing: the English-only JobBERT-v2 scores 0.6329 on English, beating multilingual v3 — the multilingual upgrade did not help English. And mmBERT-base, mean-pooled with no fine-tuning, manages only 0.2873: a raw masked language model is not a drop-in matcher, no matter how good it is.

Classification — when is a fixed taxonomy label space worth it?

Predicting one of ~1,300 ESCO occupation codes from real vacancy titles (6,408 test rows). Baselines: majority class 0.0462, random 0.00089.

Method	accuracy@1	accuracy@5
JobBERT-v3 embeddings + LogReg	0.4515	0.7260
MiniLM embeddings + LogReg	0.4065	0.6670
JobBERT-v3 nearest-neighbour retrieval	0.3939	0.6323
TF-IDF + linear SGD	0.3424	0.4042

The interesting part is not the totals — it is what happens when you split by how many training examples each occupation had:

Training examples for the true class	test rows	classifier	retrieval (1-NN)
1–2	318	0.0566	0.1887 ← retrieval wins 3×
3–10	819	0.2210	0.2552
11+	5,271	0.5111	0.4278 ← classifier wins

So "classification is better when the label space is fixed" is only half true. It is better when each label also has enough examples to learn from. In the long tail it collapses, and retrieval — which needs only one entry in an index — wins. Notebook 06 measures this rather than asserting it.

Gender bias — every model tested favours the masculine form

2,181 gendered pairs taken from ESCO's own official dual-form labels. Identical top-1 rate = how often the masculine and feminine spellings of the same occupation return the same best match (1.0 = fair):

Model	German	Spanish	Italian	French
JobBERT-v3	0.949	0.849	0.798	0.759
MiniLM	0.822	0.723	0.730	0.500

MiniLM changes its top answer for half of all French pairs purely because the query was spelled in the feminine. All eight measured accuracy gaps favour the masculine form; the worst is MiniLM on French, 0.470 vs 0.319.

The thing that does not work: knowing when to say "no"

Neither approach can reliably tell a real job title from garbage. Used as an accept/reject signal against 2,439 negatives, embedding similarity scores AUC 0.535 — barely better than a coin flip — and lexical fuzzy scoring manages 0.426, worse than random, because garbage built from recycled job-title vocabulary scores higher (79.2) than genuine titles do (75.2).

This matters more than any number above. Matching is largely solved; validation is not. See CONCLUSION.md §4.

---

Repo map

├── README.md            ← you are here
├── METHODS.md           ← every approach that exists, with trade-offs. A menu, not a recommendation
├── MODELS.md            ← the 6 models, what each is, how to use it
├── RESEARCH.md          ← literature review, 11 papers
├── CONCLUSION.md        ← honest findings, limits, and what would make this repo wrong
├── PROGRESS.md          ← build log
│
├── data/
│   ├── raw/             ← the actual data, checked in
│   │   ├── DATA_SOURCES.md   ← every URL, licence, and a link directory for sources we couldn't script
│   │   ├── esco/ onet/ talentclef/ jobbert_eval/
│   └── processed/       ← cleaned tables the notebooks read
│
├── notebooks/           ← open the .ipynb files: code AND results, already run
│   ├── 01_data_exploration.ipynb
│   ├── 02_baseline_lexical_taxonomy_match.ipynb
│   ├── 03_multilingual_embedding_matching.ipynb
│   ├── 04_evaluation.ipynb               ← validation task, with real negatives
│   ├── 05_retrieval_ann_talentclef.ipynb ← retrieval + FAISS/HNSW, official benchmark
│   ├── 06_classification_taxonomy_codes.ipynb
│   ├── 07_gender_bias_fairness.ipynb
│   ├── common.py  talentclef.py          ← shared helpers, so logic can't drift between notebooks
│
├── results/             ← every table and chart the docs cite
└── scripts/             ← download data, download models, verify metrics

Each notebook has a paired .py file. They are the same thing: the .ipynb is what you read (it has the saved output), the .py is what you diff in git.

---

Reproducing this

git clone https://git.binary.ovh/brahm/multilingual-job-title-matching.git
cd multilingual-job-title-matching

python3 -m venv venv && source venv/bin/activate
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install -r environment/requirements.txt

python scripts/download_models.py            # ~6.4 GB, all keyless
python scripts/download_talentclef.py        # benchmark data (already checked in)
python scripts/verify_metrics_against_official.py   # proves our metrics match the official scorer

jupyter lab notebooks/                       # or re-run:
jupyter nbconvert --to notebook --execute --inplace notebooks/05_retrieval_ann_talentclef.ipynb

You do not need to run anything to read the results — the notebooks already contain their output. Runtimes on a 4-core CPU box, for reference: notebook 05 ~35 min, 06 ~12 min, 07 ~25 min.

No API keys, tokens, logins or paid services are used anywhere in this repo. Every data source and model is a plain keyless download.

---

Want to try something else?

METHODS.md exists so you can choose an approach rather than inherit mine. It covers 11 method families — lexical, taxonomy lookup, classical ML, frozen embeddings, contrastive fine-tuning, cross-encoders, domain-adaptive pretraining, LLM prompting, hybrids, rejection/calibration, and validation-specific methods — each with what it costs, whether it runs on a CPU-only box, what researchers found, and what they said they wanted to try next. There is a decision matrix at the end.

Practical starting points:

• Change the model → edit the MODELS list at the top of notebook 05.
• Add a language → edit LANGS / the language tuples; ESCO ships ~28.
• Train your own → TalentCLEF ships 28,880 English training pairs (plus Spanish and German) at data/raw/talentclef/TaskA/training/, ready for contrastive fine-tuning. METHODS.md §5 explains the method; expect it to be slow on CPU.
• Add a taxonomy → DATA_SOURCES.md §5 lists ROME, KldB, KZiS, ANZSCO and others with verified links and an honest note on which can be scripted.

---

Why this repo is careful

Four earlier job-title projects on this machine were retracted after review. Their headline numbers were measurement error: a holdout set with zero negatives making "100% precision" pure arithmetic; a lexicon evaluated against the very taxonomy it was built from; near-duplicate titles leaking across a train/test split; a model that had memorised template artefacts.

So, as standing rules here:

1. Every eval set contains real negatives, and the negative count is stated next to any precision figure.
2. A naive baseline is reported beside every model number.
3. Deduplicate by normalized text before splitting. Notebook 06 found the JobBERT dataset's own shipped splits share titles, and built a clean one.
4. Never evaluate a lexicon against the source it came from. TalentCLEF's answer key is external; the classification inputs are real vacancy titles.
5. Report the honest number, especially when it's bad — hence the AUC 0.535 finding sitting in this README rather than buried in an appendix.
6. Say when a result is guaranteed by construction. Notebook 07's mitigation reaches 1.000 symmetry only because normalisation makes both queries the same string; that is arithmetic, and it is labelled as such.

If you check one of these and find it doesn't hold, that's a bug — please say so. CONCLUSION.md §7 lists specifically what would make this repo wrong.

Licence and data

Code: use freely. Data belongs to its publishers — ESCO and O*NET (CC BY 4.0), TalentCLEF 2025 (CC BY 4.0), JobBERT evaluation dataset (MIT). Full attribution in data/raw/DATA_SOURCES.md.