cyber-witten-7b-GGUF

cyber-witten-7b (GGUF, Q4_K_M)

A QLoRA fine-tune of Qwen2.5-7B-Instruct distilled for grounded, cite-or-fail question answering over Edward Witten's papers. It is the local model behind Cyber-Witten, a retrieval-augmented QA system with a citation validator, a live guardrail, and an expert-scored evaluation stack.

⚠️ RAG-only: do not use as a bare chatbot

This model was trained exclusively on prompts that contain retrieved passages (the Cyber-Witten serving format: a question plus eight source passages, with the instruction to cite passage IDs inline). Its distinctive behavior — citing paper IDs unprompted — is an instinct, not knowledge of what was retrieved:

• With retrieval (the intended use), citations are checkable and the measured ungrounded-citation rate is low (see numbers below).
• Without retrieval (bare chat), the same instinct produces plausible-looking citations to papers that are not in front of it — confident, ungrounded citation that looks grounded. This is the worst failure mode of the base model, amplified.

Use it inside the Cyber-Witten pipeline (or any harness that supplies passages and validates citations), not as a standalone assistant.

Training

• Data: 2,050 (question, passages, cited-answer) triples generated by a DeepSeek-V4 teacher over the Witten corpus, each gated by a citation validator (grounded, ≥1 citation, no grounding violation); evaluation gold questions strictly held out, with an n-gram overlap guard.
• Method: QLoRA (nf4, r=16, ~2 epochs, assistant-only loss, max_seq_len 8192) on a single RTX 4090; merged and quantized to Q4_K_M.
• Total cost: ≈ $14 (teacher API + GPU rental).

Measured (23 held-out expert questions, automated citation metrics)

The citation reflex transferred: naked, it cites unprompted and beats the guardrailed base model. The guardrail still earns its keep by zeroing the raised naked ungrounded-citation rate. Known regression: the training data contained no refusal demonstrations, and refusal on false-premise questions eroded (it will assert a fictional follow-up paper while citing a real one); Cyber-Witten covers this with a pre-generation refusal gate.

Use

With ollama:

ollama run hf.co/xingyang-yu/cyber-witten-7b-GGUF

then set num_ctx ≥ 10240 for RAG use (the serving prompt is ~5-6k tokens, and guardrail retries were measured to overflow 8192):

# Modelfile
FROM hf.co/xingyang-yu/cyber-witten-7b-GGUF
PARAMETER num_ctx 10240

Full pipeline (retrieval + guardrail + UI): github.com/xingyang-yu/cyber-witten.

adapter/ contains the LoRA adapter (safetensors) for use on the unquantized base.

Provenance & license

Base model Apache-2.0 (Qwen2.5-7B-Instruct); teacher outputs generated under DeepSeek's terms permitting training use; this repository redistributes model weights only — no corpus text. Trained by Xingyang Yu as part of the Cyber-Witten project.