docred-stage3-atlop-fine-tuning-v3

Evaluation Results (Stage 3 — GAIN-lite, v3)

Evaluation was conducted on the DocRED dev set.

Dataset Statistics

• Documents: 998
• Predicted Triples: 13,870
• Gold Triples: 12,275
• Correct Predictions: 6,862

---

Main Metrics

• Micro F1: 52.49%

  • Precision: 49.47%
  • Recall: 55.90%

• Ign F1: 49.54%

  • Precision: 46.58%
  • Recall: 52.91%

• Evidence F1: 42.20%

  • Precision: 40.94%
  • Recall: 43.55%

---

Detailed Analysis

• Intra-sentence F1: 62.49%

  • Precision: 64.35%
  • Recall: 60.74%

• Inter-sentence F1: 42.59%

  • Precision: 37.03%
  • Recall: 50.11%

---

Interpretation

This early version (v3) shows unstable performance with a Micro F1 of 52.49%.

A critical issue is the excessive number of predicted triples:

• Predicted: 13,870 >> Gold: 12,275

This indicates:

• Severe over-prediction
• Large number of false positives

As a result:

• Precision collapses (49.47%)
• Recall remains relatively high (55.90%)

---

Key Issues

1. Threshold Miscalibration

   • Too many relations are accepted
   • Adaptive threshold is not properly tuned

2. Precision–Recall Imbalance

   • Model favors recall over precision
   • Leads to degraded overall F1

3. Weak Multi-hop Reasoning

   • Inter F1 (42.59%) is significantly low
   • Graph reasoning not yet effective

---

Key Insight

Without proper threshold control, graph-based models tend to over-predict relations, leading to severe precision degradation.

---

Output Files

• Predictions: results/stage3/v3/predictions.json
• Submission File: results/stage3/v3/result.json