polygraf-ner-stage4-model

Polygraf / NER Stage 4: Second Model

distilbert-base-cased fine-tuned end to end for token classification on inovruzova/polygraf-ner-stage4-expanded. All layers were updated, not only the classification head.

This is the Stage 4 output of the Polygraf Applied NLP / NER technical task. It replaces the Stage 3 model, which is left published at inovruzova/polygraf-ner-stage3-model.

Labels

Eight entity types, seventeen tags including O.

PERSON, ORGANIZATION, LOCATION, TIMEDATE, PRODUCT, WORKOFART, JOB, AMOUNT

The full annotation policy is on the dataset card.

Usage

from transformers import pipeline

ner = pipeline(
    "token-classification",
    model="inovruzova/polygraf-ner-stage4-model",
    aggregation_strategy="simple",
)

ner("The software engineer at Google filed the patent in March 2024.")

Model facts

Throughput measured one passage at a time on CPU, after warm-up, over 100 passages from the test split.

Training

No validation split, following the task instruction, and therefore no checkpoint selection.

Only two things changed from Stage 3: the base model, and the learning rate.

The model change was the deliberate one. Stage 3 used google/electra-small-discriminator, which is 13.5 million parameters and uncased. Every model below about 35 million parameters is uncased and the smallest cased one is five times larger, so choosing the smallest reasonable model meant giving up capitalisation. That cost was written down as a prediction before the Stage 3 model was trained, and this stage tests it.

The learning rate is a dependent setting rather than a second experiment. A rate has to be chosen either way. Stage 3 used 1e-4 because ELECTRA small was undertrained at 5e-5; DistilBERT is five times larger and close to the size 5e-5 was designed for. Train F1 of 1.000 confirms the rate is not too low, and that check uses the training split only.

Results

Micro-averaged F1: 0.826 on test, 1.000 on train.

Test, per label

Compared with the Stage 3 model

The test split grew from 119 records to 156 when new records were added, so the comparison below uses only the 600 spans present in both.

What the gains say

The three biggest gains are the three labels that were predicted not to improve.

WORKOFART, ORGANIZATION and PRODUCT are governed by four policy rules that need knowledge outside the token sequence: whether a spec-like token belongs to a product's commercial name, whether a brand is acting as a company, whether a title refers to a game or a film, and whether a place name stands for the institution it houses. Stage 1 called those rules unlearnable, Stage 2 repeated it, and Stage 3 appeared to confirm it, since those three labels came last by a wide margin.

They moved the most. All three are proper-name labels, and capitalisation is the strongest cue for a proper name in English. The Stage 3 model was uncased and could not see it, so the binding constraint was the tokenizer, not the rules.

The alternative explanation is ruled out: those same three labels received the fewest new spans in the data expansion, so extra data cannot account for the gain.

The ceiling argument was not wrong, only premature. Those three labels are still the weakest three.

Limitations

The model memorises its training set. Train 1.000 against test 0.826 is a gap of 0.174, down from 0.254 in Stage 3 but still large. This is left uncorrected rather than fixed by shortening training, because with no validation split, shortening until the test gap closes would be selecting a training length on test performance.

It over-predicts. Recall exceeds precision on six of eight labels, and by the widest margins on ORGANIZATION, PRODUCT and WORKOFART.

Test metrics are agreement, not accuracy. The test labels were measured at 98.3 percent span precision in a 40-record hand audit, so these numbers compare the model against an imperfect reference.

Three of the ten Stage 3 failure patterns are unchanged and one regressed. Vague time expressions are still labeled, possessives are still split off establishment names, and adjacent time expressions are still merged.

Two new failure modes appeared, both label confusions on capitalised words: Ramadan labeled PERSON where R10 makes named observances TIMEDATE, and Apache Hadoop and Django labeled LOCATION where R16 makes software PRODUCT. The Stage 3 model had all three right.

Those regressions share one cause. Capitalisation is a strong signal that something is a name and no signal at all about which kind of name. The uncased model missed these words; the cased model finds them and guesses wrong. That is the predictable other half of the change made in this stage.

Related repositories

Licensing

The model weights are released under Apache 2.0, matching the base model.

The training data is a collection of corpora under different terms, including share-alike obligations from ODbL and CC BY-SA. Those obligations attach to the dataset rather than to a set of trained weights, but anyone reusing this model should read the dataset card.

Produced during the Polygraf Applied NLP / NER technical task.