gliner-multi-v2.1-nemotron-pii-tr

Model & Benchmark Report: gliner-multi-v2.1-nemotron-pii-tr

This report outlines the architecture, training details, and comparative benchmark performance of the gliner-multi-v2.1-nemotron-pii-tr model against its base model (urchade/gliner_multi-v2.1).

---

1. Model Overview

Model Name: gliner-multi-v2.1-nemotron-pii-tr

Base Model: urchade/gliner_multi-v2.1

Task: Span-level Named Entity Recognition (NER) for PII detection.

Language: Turkish (Fine-tuned on Nemotron-derived EN→TR translated data).

This model is designed for PII detection and general entity extraction in Turkish, utilizing a span-based GLiNER architecture with marker-based decoding. It predicts entities as token spans rather than token-level BIO tags.

Configuration Notes (How to read this config)

This model’s saved gliner_config.json contains two layers of settings in one place:

• Backbone (HuggingFace encoder config) under encoder_config
• GLiNER span/decoder + run settings as top-level fields (e.g., span_mode, max_width, max_len, dropout)

Because the full HuggingFace encoder config is embedded, the file looks unusually large — this is expected.

Key effective settings (what actually matters at inference)

Common confusion points

• 512 vs 384 length: the encoder supports 512, but this model/run caps inputs at 384 (max_len), so 384 is the effective limit.
• Hidden size values: encoder_config.hidden_size=768 is the encoder dimension, while the top-level hidden_size=512 is an internal GLiNER head size (they are not the same parameter).
• Training fields may differ: values like num_steps, train_batch_size, eval_every, and warmup_ratio can reflect the saved run artifact and may not match a separate YAML used in another run.

Training Hyperparameters

• Total Steps: 13,750
• Batch Size: 4
• Eval Every: 500
• Learning Rate: 1e-5 (Encoder), 5e-5 (Others)
• Loss Coefficients: Span Loss (0.5), Decoder Loss (0.5)
• Optimization: Gradient clipping (10.0), Cosine scheduler with 0.1 warmup.

Training Data

The model was fine-tuned on newmindai/Nemotron-PII-EN-TR, a Turkish PII dataset translated from NVIDIA's Nemotron-PII.

Dataset Size:

• Train: 11,215 samples (84,179 entities)
• Test: 1,403 samples (10,744 entities)
• Validation: 1,401 samples (10,542 entities)

Entity Labels (8): person, contact_info, digital_identifier, financial_identifier, identity_number, health_identifier, organization, other_identifier

For detailed dataset information, label distributions, and usage examples, see the dataset card.

---

2. Benchmark Environment

The model was evaluated against the base model to measure the impact of fine-tuning on a specific test set.

• Hardware: NVIDIA L40S (CUDA 12.6)
• Inference Parameters: Threshold: 0.5, Batch Size: 2

---

3. Executive Summary (Comparative Results)

The fine-tuned model demonstrated a ~173% relative increase in Micro F1-Score compared to the base model. While the base model failed to recognize specific PII formats, the fine-tuned model showed high adaptability.

3.1 Benchmark Summary (All Variants)

🏆 Best F1-Score: Model 2 (gliner-multi-v2.1-nemotron-pii-tr (ONNX)) with 0.5919  

⚡ Fastest: Model 4 (Base Model) with 9.54 ms/sample

Comparison vs Baseline (Model 1: gliner-multi-v2.1-nemotron-pii-tr):

• Model 2 (gliner-multi-v2.1-nemotron-pii-tr (ONNX)): F1 ↑ +0.0014 (+0.14%), 291.91 ms slower
• Model 3 (gliner-multi-v2.1-nemotron-pii-tr (ONNX Quantized)): F1 ↓ -0.4779 (-47.79%), 277.60 ms slower
• Model 4 (Base Model): F1 ↓ -0.3743 (-37.43%), 0.34 ms faster

Key Category Improvements (F1 Score)

---

4. Detailed Performance: gliner-multi-v2.1-nemotron-pii-tr

Overall Metrics:

• Precision: 53.80%
• Recall: 65.46%
• F1-Score: 59.06%

Per-Class Metrics

Confusion Matrix (gliner-multi-v2.1-nemotron-pii-tr)

Rows = True Labels | Columns = Predicted Labels

• NONE (Column): False Negatives (Missed entities)
• NONE (Row): False Positives (Hallucinated entities)

---

5. Detailed Performance: gliner-multi-v2.1-nemotron-pii-tr (ONNX)

Overall Metrics:

• Precision: 53.86%
• Recall: 65.69%
• F1-Score: 59.19%

Per-Class Metrics

Confusion Matrix (gliner-multi-v2.1-nemotron-pii-tr (ONNX))

---

6. Detailed Performance: gliner-multi-v2.1-nemotron-pii-tr (ONNX Quantized)

Overall Metrics:

• Precision: 86.50%
• Recall: 6.03%
• F1-Score: 11.27%

Per-Class Metrics

Confusion Matrix (gliner-multi-v2.1-nemotron-pii-tr (ONNX Quantized))

---

7. Detailed Performance: Base Model

Overall Metrics:

• Precision: 34.06%
• Recall: 15.84%
• F1-Score: 21.62%

Per-Class Metrics

Confusion Matrix (Base Model)

---

8. Observations & Technical Notes

1. Necessity of Fine-Tuning: The results confirm that the base model is insufficient for this specific dataset and label schema. The other_identifier and contact_info categories were virtually undetectable without fine-tuning.
2. Truncation Warning: Long texts in the test set exceeded the effective token limit. This truncation likely contributes to the lower Recall (higher False Negatives), as entities at the end of documents are ignored.
3. False Positive Rate: The model shows a high number of False Positives (rows labeled NONE in the confusion matrix). Increasing the inference threshold (currently 0.5) could improve Precision.
4. Data Limitation: The model is trained on synthetic, machine-translated data. While effective, artifacts from translation may impact performance on native Turkish corpora.
5. ONNX vs PyTorch: ONNX export preserved accuracy (slightly higher F1), but inference latency in this benchmark was significantly higher than the PyTorch variant.
6. Quantization Caveat: The quantized ONNX model shows very high Precision but collapses in Recall (most entities are missed), leading to a low F1.

---

9. Usage

This model performs span-based PII detection for Turkish text using the gliner library.

pip install gliner
from gliner import GLiNER

labels = [
    "person",
    "contact_info",
    "digital_identifier",
    "financial_identifier",
    "identity_number",
    "health_identifier",
    "organization",
    "other_identifier",
]

text = (
    "Benim adım Ahmet Yılmaz ve e-posta adresim [email protected]. "
    "23 Mayıs 1990 tarihinde İstanbul’da doğdum. "
    "Telefon numaram 0555 123 45 67, TC kimlik numaram ise 12345678901’dir. "
    "Ayrıca banka kartı numaram 5432 1234 5678 9012."
)

model = GLiNER.from_pretrained(
    "newmindai/gliner-multi-v2.1-nemotron-pii-tr"
)

entities = model.predict_entities(text, labels, threshold=0.7)

for e in entities:
    print(f"Text: {e['text']}")
    print(f"Label: {e['label']}")
    print(f"Score: {e['score']:.3f}")
    print(f"Position: [{e['start']}:{e['end']}]")
    print("---")

---

Expected Output (Truncated Real Example)

Below is a real model output, shortened for readability (full output omitted with ...):

Text: Ahmet
Label: person
Score: 1.000
Position: [11:16]
---
Text: Yılmaz
Label: person
Score: 1.000
Position: [17:23]
---
Text: ahmet
Label: person
Score: 0.998
Position: [43:48]
---
Text: [email protected]
Label: contact_info
Score: 0.855
Position: [49:65]
---
Text: 23 Mayıs 1990
Label: other_identifier
Score: 0.891
Position: [67:80]
---
...

---

Notes

• The model is span-based and returns character-level positions.
• Adjacent spans (e.g. first name / last name) are not merged automatically.
• Recommended threshold: 0.5–0.75 depending on precision–recall tradeoff.

---