docred-stage3-atlop-fine-tuning-v4

Evaluation Results (Stage 3 — GAIN-lite, v4)

Evaluation was conducted on the DocRED dev set.

Dataset Statistics

• Documents: 998
• Predicted Triples: 13,536
• Gold Triples: 12,275
• Correct Predictions: 6,842

---

Main Metrics

• Micro F1: 53.02%

  • Precision: 50.55%
  • Recall: 55.74%

• Ign F1: 50.10%

  • Precision: 47.70%
  • Recall: 52.75%

• Evidence F1: 42.15%

  • Precision: 40.95%
  • Recall: 43.41%

---

Detailed Analysis

• Intra-sentence F1: 62.50%

  • Precision: 64.54%
  • Recall: 60.59%

• Inter-sentence F1: 43.43%

  • Precision: 38.43%
  • Recall: 49.93%

---

Interpretation

This version (v4) shows a significant drop in overall performance (Micro F1: 53.02%) compared to later versions.

A key observation is the unusually high number of predicted triples:

• Predicted: 13,536 > Gold: 12,275

This indicates that the model is:

• Over-predicting relations
• Producing many false positives

As a result:

• Precision drops sharply (50.55%)
• Recall is relatively high (55.74%)

---

Key Issue

The main problem in this version is threshold miscalibration:

• Too many relations are accepted
• Adaptive threshold (ATLOP) is likely too permissive
• Leads to precision collapse

---

Additional Observations

• Inter F1 (43.43%) is significantly lower than later versions (~52%)
• Indicates weaker multi-hop reasoning performance
• Suggests that graph reasoning was not effectively utilized or not yet stabilized

---

Key Insight

This version demonstrates that:

Increasing recall without controlling precision leads to severe performance degradation in multi-label relation extraction.

Proper threshold tuning and calibration are critical in ATLOP-based models.

---

Output Files

• Predictions: results/stage3/v4/predictions.json
• Submission File: results/stage3/v4/result.json