nel-mgenre-multilingual

Impresso multilingual entity linking (nel-mgenre-multilingual)

Links named-entity mentions in historical texts to Wikipedia and Wikidata. The model is mGENRE (multilingual generative entity retrieval, on mBART) fine-tuned by the Impresso project on historical documents. This repository ships it with the impresso-nel pipeline, the one used in the Impresso datalab notebooks.

Use it

# pip install transformers torch nltk marisa-trie
from transformers import pipeline

ner = pipeline("impresso-ner", model="impresso-project/ner-hipe2020-mmbert-base-final",
               revision="impresso-ner-v1", trust_remote_code=True)
nel = pipeline("impresso-nel", model="impresso-project/nel-mgenre-multilingual",
               revision="impresso-nel-v1", trust_remote_code=True)

text = "In 1789, King Louis XVI convened the Estates-General at the Palace of Versailles."
nel({"text": text, "entities": ner(text)})
# the NER entities it keeps, each with link fields added:
# [{'type': 'pers', 'surface': 'King Louis XVI', ..., 'confidence_nel': 1.0,
#   'wkdata_qid': 'Q7732', 'wkpedia_lg': 'en', 'wkpedia_pagename': 'Louis XVI of France',
#   'label_de': 'Ludwig XVI.', ...}, ...]

nel("The event was held at the [START] Palace of Versailles [END].")   # one marked mention -> one dict
nel([{"text": t, "entities": ner(t)} for t in texts])                  # one list per text

• Requires transformers ≥ 5. trust_remote_code=True runs the Python files of the pinned revision; you can read them in the repository.
• Inputs. Either {"text": ..., "entities": [...]}, where each entity has lOffset and rOffset (character offsets) and optionally type and surface (the output of impresso-ner fits), or a string with one [START] mention [END] marker (only the first pair is used; the result is {} when the mention is skipped, e.g. a stopword or a one-letter surface). A string without markers raises ValueError. A marked mention, or an entity without type, comes back with type: "unk".
• Dropped mentions. Some mentions are left out of the result, so it can be shorter than the input list: a surface under 2 characters, a stopword or title (M., Herr, der, The, …), a surface containing |, offsets missing or outside the text, or a [START] mention [END] block longer than 256 tokens on its own. The rest keep the input order; match them back by lOffset / rOffset, not by position.
• Output. Per mention: wkdata_qid (Wikidata), wkpedia_lg / wkpedia_pagename (Wikipedia page; URL https://<lg>.wikipedia.org/wiki/<pagename>), Wikidata labels label_fr / label_en / label_de (label_other + label_other_lg when those three are missing), confidence_nel (length-normalised title probability) and confidence_margin. When the title has no Wikidata item, wkdata_qid and the labels are None. nel_unlink says why a predicted link was removed (e.g. a disambiguation page, or a type that can't match), and type_ner_original records an NER type corrected from Wikidata.
• How it works. For each mention, mGENRE reads up to 256 tokens of text centred on the marked mention and generates a Wikipedia title (greedy decoding). Titles are mapped to Wikidata offline with the lookup tables of the impresso-project/nel-mgenre-trie dataset (about 1.5 GB, downloaded on the first call; pinned revision), then filtered and labelled. This is the code of the Impresso batch pipeline, flattened into this repository. No Wikipedia or Wikidata API is called.
• Speed. About 1 s per mention on a laptop CPU; much faster on a GPU (device=0).
• Mac (MPS). The pipeline runs on CPU when given an MPS device: transformers 5.20 fails in generate on MPS for encoder-decoder models.
• Tokenizer. tokenizer.json is converted from sentencepiece.bpe.model and gives the same token ids as the original slow tokenizer on transformers 4. Transformers 5 can't rebuild that slow tokenizer correctly.

Model

• Developed by: EPFL, the Impresso team, an interdisciplinary project on historical media across languages, time and modalities. Funded by the Swiss National Science Foundation (CRSII5_173719, CRSII5_213585) and the Luxembourg National Research Fund (grant No. 17498891).
• Model type: mBART sequence-to-sequence model (MBartForConditionalGeneration) that generates "<Wikipedia title> >> <language>" for a mention marked [START] … [END] in its context.
• Languages: multilingual (100+ languages), tuned on French, German and English.
• Fine-tuned from: facebook/mgenre-wiki.
• Training data: the HIPE-2022 datasets:

• Training compute: 1× A100 (80 GB), about 12 hours, about 2.3 kg CO₂eq.
• Limitations: sensitive to badly placed mention boundaries; weaker on entities without a Wikipedia page and in very ambiguous contexts; tuned on historical texts, so it may do less well on modern ones.

Versions

The earlier generic-nel pipeline is deprecated: it calls the Wikipedia and Wikidata APIs for each mention. To run it, load it from its branch. On transformers 5, also pass this revision's tokenizer, because the branch's own tokenizer is rebuilt with wrong token ids:

from transformers import AutoTokenizer, pipeline

m = "impresso-project/nel-mgenre-multilingual"
tok = AutoTokenizer.from_pretrained(m, revision="impresso-nel-v1")  # transformers 5 only
old = pipeline("generic-nel", model=m, revision="generic-nel", tokenizer=tok, trust_remote_code=True)
old("Le 0ctobre 1894, [START] Dreyfvs [END] est arrêté à Paris.")

Licences

The weights are fine-tuned from mGENRE, released by Meta under CC BY-NC 4.0, and the lookup tables are CC BY-NC 4.0: non-commercial use. The pipeline code is AGPL-3.0-or-later (see LICENSE).

The two code files (hf_pipeline.py, process.py) are copied unchanged from the Impresso impresso-inference repository, where process.py is the processing code the Impresso batch pipeline runs; do not edit them here.

Contact

impresso-project.ch