multilingual-place-extractor-mdeberta-13lang-tagger

Multilingual Place Entity Tagger — mDeBERTa-v3, 13 languages

A fine-tuned mDeBERTa-v3-base token classifier that labels cities, countries, and entities (hotels / points of interest / landmarks) in free text across 13 languages, using a typed BIO scheme (7 labels: O, B-/I-CITY, B-/I-COUNTRY, B-/I-ENTITY).

Languages: English, French, German, Spanish, Italian, Dutch, Portuguese, Turkish, Russian, Chinese, Japanese, Korean, Arabic.

This repository is the standalone PyTorch / safetensors tagger (load it with AutoModelForTokenClassification). Turning the tagged spans into linked (text → "<entity> <city> <country>") query strings is a separate, deterministic, parameter-free step (a positional linker + a city→country gazetteer + country canonicalization). An ONNX build for in-browser use (transformers.js) is at Berk/multilingual-place-extractor-mdeberta-13lang-onnx.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForTokenClassification

repo = "Berk/multilingual-place-extractor-mdeberta-13lang-tagger"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForTokenClassification.from_pretrained(repo).eval()

text = "I booked Hotel Lungomare in Rimini then flew to Bologna"
enc = tok(text, return_tensors="pt")
with torch.no_grad():
    pred = model(**enc).logits[0].argmax(-1)
print([(tok.convert_ids_to_tokens([i])[0], model.config.id2label[p.item()])
       for i, p in zip(enc["input_ids"][0], pred)])
# -> Hotel/B-ENTITY Lung/I-ENTITY ... Rimini/B-CITY ... Bologna/B-CITY

Linking spans to (city, country)

The tagger emits typed spans; to turn them into (text → "<entity> <city> <country>") the repo bundles the full ~1.03M-city gazetteer (GeoNames-derived):

• gazetteer/city_country_gazetteer.json — {"case_insensitive": {city → country}}
• gazetteer/city_country_multi.json — {ambiguous_city → [[country, population], …]} (pick the candidate country that is named nearby in the text; else the highest-population one)
• gazetteer/country_lookup.json — {surface_form → English country} to canonicalize a tagged COUNTRY

import json
gaz = json.load(open("gazetteer/city_country_gazetteer.json"))["case_insensitive"]
clk = json.load(open("gazetteer/country_lookup.json"))
norm = lambda s: " ".join(s.lower().split())

# CITY span "Rimini" -> country
print(gaz.get(norm("Rimini")))               # Italy
# COUNTRY span "미국" (native script) -> canonical English
print(clk.get(norm("미국")))                  # United States

The full positional linker (nearest CITY/COUNTRY by character distance, in-sentence disambiguation, entity→city inheritance) is parameter-free; a JavaScript reference implementation is cascade.js in the companion ONNX repo Berk/multilingual-place-extractor-mdeberta-13lang-onnx.

Training data — sources & licenses

Trained on public data only. The labels are derived deterministically from public gazetteers and place databases; an open LLM was used only to write natural query phrasing (never to invent place facts).

Training text is synthetic, location-rich travel prose balanced across all 13 languages, with the non-Latin-script languages (Russian, Chinese, Japanese, Korean, Arabic) generated in native script so place names are learned in the form they actually appear.

Results

Held-out evaluation:

v2 (current) adds landmark-list recall (a city followed by a comma-separated list of landmarks/museums) and a city/country reclassification fix. Held-out landmark-list field_f1 rose from 0.29 to ~0.86 with no regression elsewhere.

Per-language full-system field_f1 (small-n languages are noisy): it 0.93 · fr 0.93 · nl 0.94 · en 0.94 · pt 0.90 · es 0.86 · tr 0.90 · de 0.92 · zh 0.82 · ko 0.89 · ja 0.93 · ar 0.82 · ru 0.42 (n=4).

License

Released under MIT (a derivative of mDeBERTa-v3-base, MIT). Respect the upstream data-source licenses listed above (notably the CC BY 4.0 attribution requirements and the ODbL terms for OpenFlights-derived content).

Limitations

• Domain-specific: trained on location-rich, itinerary-style text; may not generalize to arbitrary-domain NER.
• Small per-language eval slices are noisy (Russian especially, n=4).
• The tagger emits typed spans only; city→country linking is the separate deterministic step.