docred-stage3-atlop-fine-tuning-v6

Evaluation Results (Stage 3 — GAIN-lite)

Evaluation was conducted on the DocRED dev set.

Dataset Statistics

• Documents: 998
• Predicted Triples: 10,484
• Gold Triples: 12,275
• Correct Predictions: 6,877

---

Main Metrics

• Micro F1: 60.43%

  • Precision: 65.60%
  • Recall: 56.02%

• Ign F1: 58.22%

  • Precision: 64.06%
  • Recall: 53.36%

• Evidence F1: 42.28%

  • Precision: 40.93%
  • Recall: 43.71%

---

Detailed Analysis

• Intra-sentence F1: 67.53%

  • Precision: 72.19%
  • Recall: 63.44%

• Inter-sentence F1: 51.67%

  • Precision: 57.17%
  • Recall: 47.13%

---

Interpretation

The Stage 3 model introduces graph-based reasoning (GAIN-lite) on top of Stage 2, enabling interaction between entities across the document.

• The Micro F1 (60.43%) indicates stable overall performance in multi-label relation extraction.
• The Ign F1 (58.22%) confirms that the model generalizes reasonably well beyond memorized relational facts.
• The Evidence F1 (42.28%) shows that evidence prediction remains a challenging task, though it provides meaningful interpretability signals.

A key observation is the gap between:

• Intra-sentence F1 (67.53%)
• Inter-sentence F1 (51.67%)

This demonstrates that:

• The model performs significantly better on local (within-sentence) relations
• While cross-sentence (multi-hop) reasoning is improved, it still remains the main bottleneck

---

Key Insight (Stage 2 → Stage 3)

Compared to Stage 2, Stage 3 aims to improve multi-hop reasoning through graph message passing.

• ✔ Improved modeling of entity interactions
• ✔ Better handling of cross-sentence dependencies
• ❗ Still limited performance on long-range reasoning (Inter F1 gap)

---

Output Files

• Predictions: results/stage3/v6/predictions.json
• Submission File: results/stage3/v6/result.json

---

Notes

• This model is evaluated under the DocRED standard evaluation protocol
• Results may vary depending on threshold tuning and preprocessing pipeline
• Inter-sentence performance highlights the importance of graph-based reasoning modules