qwen3-1.7b-cogs-ingest-lora

qwen3-1.7b-cogs-ingest-lora

LoRA adapter for Qwen/Qwen3-1.7B, fine-tuned as the student model for the cogs ingest pipeline. It replaces a large teacher LLM as a local, OpenAI-compatible provider for four structured-output tasks, emitting compact JSON exactly as the cogs runtime parses it:

Output-format fidelity (strict, schema-exact JSON) is the objective, not prose.

Training

• Method: LoRA SFT (TRL) — r=16, α=32, dropout=0.05, targets = all attention
  • MLP projections (q,k,v,o,gate,up,down).
• Data: produced by cogs distill from 256 real ingests — 1,921 train / 192 valid chat-format pairs (assistant turn = compact JSON target). ~2.8M tokens/epoch; avg 1,462 tok/example.
• Schedule: 2 epochs, effective batch 16 (2 × grad-accum 8), max_seq 8192, lr 1e-4 cosine, warmup 0.03, bf16, gradient checkpointing.
• Hardware: NVIDIA DGX Spark (GB10), ~107 min, ~870 tokens/s.
• Loss: train 2.55 → 1.41; eval 1.125 → 1.118 (still decreasing at 2 epochs). Eval token-accuracy 0.756.

⚠️ Serving: avoid pure greedy decoding

The model learned the schema well, but under pure greedy (temperature 0, no penalty) it degenerates on the long list-valued tasks (extract, suggest_links): it keeps appending list items and never emits <|im_end|>, leaving valid-but-unterminated JSON. Fix with either:

• repetition_penalty ≈ 1.1 (keeps determinism), or
• Qwen non-thinking sampling: temperature=0.7, top_p=0.8, top_k=20.

With either, a 5-sample / 4-task JSON parse eval is 5/5 (100%) schema-exact (vs. 3/5 at plain greedy). Use enable_thinking=False and stop on <|im_end|>.

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "Qwen/Qwen3-1.7B"
model = AutoModelForCausalLM.from_pretrained(base, dtype="bfloat16", device_map="cuda")
model = PeftModel.from_pretrained(model, "lewisdog/qwen3-1.7b-cogs-ingest-lora").eval()
tok = AutoTokenizer.from_pretrained("lewisdog/qwen3-1.7b-cogs-ingest-lora")

enc = tok.apply_chat_template(
    messages, add_generation_prompt=True, enable_thinking=False,
    return_tensors="pt", return_dict=True,
).to(model.device)
out = model.generate(**enc, max_new_tokens=2048, do_sample=False,
                     repetition_penalty=1.1, pad_token_id=tok.pad_token_id)
print(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))

To merge for serving (e.g. mlx_lm.convert -q for Apple-silicon / omlx):

merged = PeftModel.from_pretrained(
    AutoModelForCausalLM.from_pretrained(base, dtype="bfloat16"),
    "lewisdog/qwen3-1.7b-cogs-ingest-lora",
).merge_and_unload()
merged.save_pretrained("merged")

Framework versions

• PEFT 0.19.1 · TRL 1.7.1 · Transformers 5.13.0 · PyTorch 2.12.1+cu130 · Datasets 5.0.0