scholastic-llm-sft-v2

scholastic-llm SFT-v2 @ iter 800 (Phase 2 final)

⚠ NOTICE — research experiment, not theological authority

This is a personal portfolio / research project exploring how small open-weights LLMs can be fine-tuned to adopt a specific historical register and citation style. The trained model is not a reliable source of Catholic doctrine, biblical interpretation, or philosophical truth. It can hallucinate citations, misrepresent the Catechism, and confidently err. Outputs must not be cited as catechetical instruction, theological argument, or spiritual direction. For doctrinal questions, consult the actual Catechism of the Catholic Church, a qualified priest, or a trained theologian.

What this is

LoRA adapter for Qwen/Qwen2.5-7B-Instruct, trained to respond to philosophical and theological questions in a scholastic / Latin-inflected register grounded in the Catechism of the Catholic Church (CCC, 1992) and modeled after Aquinas's Summa Theologica and Augustine's Confessions / City of God.

Phase 2 final weights at iter 800 (twice the iter-400 best checkpoint). Mildly overfit: leans more strongly Augustinian but loses some scholastic-register and structure scores compared to iter 400. Useful as a 'maximum drift' reference.

For full background, recipe, and Phase 1+2 results, see:

• Paper: https://pleyva2004.github.io/scholastic-llm/main.pdf
• GitHub: https://github.com/pleyva2004/scholastic-llm

Variants (all four published)

(Strict total $= \textsc{reg} + \textsc{aug} + \textsc{ccc} + \textsc{str}$, max 120. Balanced total $= \max(\textsc{reg},\textsc{aug}) + \textsc{ccc} + \textsc{str}$, max 90; introduced in Phase 2 because the strict total penalizes appropriate register switching.)

How to load (MLX)

from mlx_lm import generate, load

model, tokenizer = load(
    "Qwen/Qwen2.5-7B-Instruct",
    adapter_path="pleyva2004/scholastic-llm-sft-v2",
)

prompt = "How do you reconcile divine foreknowledge with free will?"
messages = [{"role": "user", "content": prompt}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(generate(model, tokenizer, prompt=text, max_tokens=300))

Requires mlx-lm (≥ 0.27) on Apple Silicon. For other inference engines you will need to convert the adapter manually; the repo contains MLX-format weights only.

Training

Training data was generated by Claude Sonnet 4.6 as teacher, per a strict system prompt requesting scholastic register and CCC citations, applied to ~50–150 cleaned source chunks scraped from the Catechism, the Summa, and Augustine's works. See the GitHub repo's scripts/generate_training_pairs.py for the exact prompt.

Evaluation

Rubric-based evaluation on 10 held-out philosophical prompts (not seen during training). Four dimensions, each scored 0–3 per prompt, summed across 10 prompts (per-dimension max 30).

Full per-prompt scores and qualitative samples are in the paper.

Data licensing

Training data sources:

The training-data JSONL itself is not redistributed with this adapter; only the LoRA weights and this card. See DATA_LICENSING.md for the full posture.

License

• This adapter (LoRA weights): MIT — see LICENSE in the repo.
• Base model (Qwen/Qwen2.5-7B-Instruct): Apache 2.0 (governed by the base-model card on Hugging Face).
• Source corpus: terms above.

Limitations & ethics

• Hallucinated citations. The fine-tuned model confidently emits CCC paragraph numbers with the surface form of ground truth. Many citations do not correspond to the actual content of the cited paragraph. Always verify against the actual Catechism.
• No human evaluation. Reported numbers come from a regex/keyword rubric. The rubric measures lexical and structural surface form, not theological correctness.
• Small held-out set (N=10). Confidence intervals are wide; the +49-point delta is large relative to noise but not bootstrapped.
• No doctrinal authority. The model speaks in a voice culturally associated with magisterial authority. It has none. It can confidently err and should not be relied upon for spiritual direction.

Citation

@misc{leyva2026scholastic,
  title  = {Teaching a Small LLM Scholastic Voice: Fine-Tuning Qwen 2.5 on the Catechism, Summa, and Augustine via Local MLX},
  author = {Pablo Leyva},
  year   = {2026},
  url    = {https://github.com/pleyva2004/scholastic-llm},
  note   = {Independent Research}
}