jev-distill-corpus

Jev-Gate Student B — LoRA Memory-Relevance Judge

A LoRA adapter (r=16, α=32, on q_proj/v_proj) on Qwen/Qwen2.5-0.5B-Instruct, distilled from the Jev typed-judgment API into a compact local judge for agent-memory gating.

What it does: given a query and a candidate memory passage, outputs P(relevant) as the calibrated yes probability read from the final-token logits of yes vs no. Used to filter which vector-recalled memories get injected into agent context (vector recall → cross-encoder band gate → this judge for gray-band cases).

Training: LoRA, 1 epoch, LR 1e-4, on rows sampled from SargeDev/jev-distill-corpus — paired relevance judgments (Jev graded 0–7 expected-value scores + binary labels) distilled from typed judgment calls by a larger teacher model. Prompt format:

Memory: {passage ≤600 chars}
Query: {query}
Question: Is this memory relevant for answering the query? Answer yes or no with confidence.

Full 10k Held-Out Benchmark (Sep 20, 2026)

A 10,000-row held-out test set (test_10k.jsonl, sampled from SargeDev/jev-distill-corpus v2, seed 42, hard-disjoint from all 142,909 train/dev/test rows by (query, text[:600]) key). Gold labels = 32B teacher scores rescaled /7. All three judges scored with the identical prompt above, yes/no logit softmax.

Distillation fidelity (the key number)

Student B vs its own teacher (Jev 1.13): 86.4% agreement, Pearson r = 0.824, MAE 0.144 across all 10k rows. The 1.1M-parameter LoRA reproduces a production System One model's judgments within ~3 points of binary agreement, at 43× lower latency and zero API cost.

Where they disagree (1,364 items), Jev is right vs gold 60% of the time — a real but modest teacher edge. The gray band (teacher y ∈ [0.3, 0.7]) is where both models live in production (the band gate escalates only these cases); student gray-band MAE 0.275 vs teacher 0.205.

Small test slices (earlier evals)

Takeaways

1. Distillation works: the LoRA captures most of the teacher's judgment; the win is from distillation, not model size (vanilla 1.5B barely helps).
2. Gray band is the battleground: both models are most uncertain there; that's exactly where the cascade routes escalation.
3. Latency economics: local student ≈ sub-second for a 25-candidate cascade judgment (batched); API teacher ≈ 1 s/item — fine for referee calls, wrong for every-candidate scoring.
4. 10k-set construction: seeded sample from fresh (never-trained) manifest rows, disjoint by content key — reproducible via seed 42.

How the pipeline works (production)

Qdrant vector recall (25 candidates)
  → bge-reranker-base int8 ONNX cross-encoder (local band gate: keep ≥0.08 / drop ≤0.01)
  → gray band [0.01, 0.08] → Student B (this adapter) judges locally
  → final TOP-K injection (default 8)
Fail-open everywhere: any stage error → keep everything (no forgetting).

Every decision is logged with (vector_score, local_score, student_score, verdict) to a JSONL log — that log is the retraining corpus for the next student iteration (see v2 dataset below).

What's published

Future directions

1. Retrain Student B on v2 corpus (~155k rows incl. 7k production cascade judgments with real agent-query distribution — current training was 60k rows from the pre-production corpus).
2. Scale the student: the same recipe on Qwen2.5-1.5B-Instruct (LoRA r=16–32) should close most of the remaining ~3-point gap to Jev; a Qwen3-4B / Qwen3-8B LoRA is the next rung — expect MAE < 0.15 and agreement > 90% if the trend holds. All still local-friendly (1.5B ≈ 3 GB bf16).
3. Hybrid referee cascade: keep Student B local for the gray band; route only true near-ties (student score ∈ [0.45, 0.55]) to the Jev API for a final call. Expected: teacher accuracy at ~2% of the call volume.
4. Multi-judge ensemble: Jev API + Student B + bge-reranker — average scores for the injected set, use disagreement as an uncertainty signal.
5. Merge to GGUF: merge the LoRA into base weights for llama.cpp single-file serving (removes PeftModel dependency; warm path already ~1 s, GGUF would make it ~50 ms on CPU-only hosts).

License

Apache-2.0 (matches base model).