dacr-paper

Synthetic Traces, Real Reasoning

How Procedurally Generated Document Challenges Transfer to Real-World Scientific Papers

Author: Botoshi | Date: April 2026

---

Download paper.pdf

---

Abstract

We present evidence that fine-tuning a 7B parameter language model on fully synthetic, procedurally generated document reasoning traces produces substantial gains on real-world scientific paper comprehension. A Qwen 2.5 7B model trained on 4,421 synthetic traces more than doubles its accuracy on real arXiv papers in DACR-Bench, from 18.9% to 40.0%, and increases the full-challenge pass rate from 0% to 44%. We also observe large gains in Causal Authority Resolution Score (CARS), from 7.7% to 46.2%, indicating improved ability to resolve conflicting claims by evidential provenance. In contrast, single-hop extraction remains flat, which suggests the model learns transferable reasoning procedures rather than additional lookup capacity.

We introduce DACR-Bench (Document Analysis with Causal Reasoning), a benchmark for multi-hop causal reasoning over real technical documents. DACR-Bench combines real arXiv papers with procedurally generated questions and deliberately planted conflicting information, requiring models to identify which claims carry greater evidential authority. We define the Causal Authority Resolution Score (CARS), a metric that isolates a model's ability to resolve contradictions by tracing evidential provenance.

The training data originates from a decentralized challenge network in which multiple frontier AI agents (GPT-5.4, Claude Haiku, Codex) independently solve structured document reasoning challenges. Each challenge is graded by deterministic automated verifiers, not human annotators. The resulting trace corpus exhibits natural diversity in reasoning style and problem decomposition.

Key Results

All results reported on 9 real arXiv document challenges (90 questions). Synthetic engine-generated challenges excluded from evaluation.

Overall Performance

Skill-Specific Transfer

Domain Transfer (no training data from any evaluation domain)

Training Configuration

• Base model: Qwen 2.5 7B Instruct
• Method: QLoRA (4-bit NF4, rank 32, alpha 64)
• Loss: Completion-only (assistant response only)
• Data: 4,421 reasoning traces from 4 domains, 3 frontier models
• Training time: 2.8 hours on 1x H100
• Epochs: 1

Related Repositories

Citation

@article{botoshi2026synthetic,
  title={Synthetic Traces, Real Reasoning: How Procedurally Generated Document Challenges Transfer to Real-World Scientific Papers},
  author={Botoshi},
  year={2026},
  note={Available at \url{https://huggingface.co/botcoinmoney/dacr-paper}}
}

License

Apache-2.0