qwen3-1.7b-cogs-ingest

qwen3-1.7b-cogs-ingest

Merged full model (Qwen/Qwen3-1.7B + LoRA, weights fused, bf16) — the student model for the cogs ingest pipeline. Use this repo when you want a standalone model to serve or convert; the standalone LoRA adapter is at lewisdog/qwen3-1.7b-cogs-ingest-lora.

It replaces a large teacher LLM as a local, OpenAI-compatible provider for four structured-output tasks, emitting compact JSON exactly as the cogs runtime parses it:

Apple silicon (MLX / omlx)

# converts + 4-bit quantizes into an MLX model dir
python3 -m mlx_lm.convert --hf-path lewisdog/qwen3-1.7b-cogs-ingest -q --mlx-path qwen3-cogs-ingest-mlx
# then drop into ~/.omlx/models and point cogs [llm] at it

⚠️ Serving: avoid pure greedy decoding

The model learned the schema well, but under pure greedy (temperature 0, no penalty) it degenerates on the long list-valued tasks (extract, suggest_links): it keeps appending list items and never emits <|im_end|>, leaving valid-but-unterminated JSON. Fix with either:

• repetition_penalty ≈ 1.1 (keeps determinism), or
• Qwen non-thinking sampling: temperature=0.7, top_p=0.8, top_k=20 (this model's bundled generation_config.json already samples at temp 0.6, which sidesteps the issue — just don't override it to temp-0 greedy).

With either, a 5-sample / 4-task JSON parse eval is 5/5 (100%) schema-exact (vs. 3/5 at plain greedy). Use enable_thinking=False and stop on <|im_end|>.

Transformers usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "lewisdog/qwen3-1.7b-cogs-ingest", dtype="bfloat16", device_map="auto"
).eval()
tok = AutoTokenizer.from_pretrained("lewisdog/qwen3-1.7b-cogs-ingest")

enc = tok.apply_chat_template(
    messages, add_generation_prompt=True, enable_thinking=False,
    return_tensors="pt", return_dict=True,
).to(model.device)
out = model.generate(**enc, max_new_tokens=2048, do_sample=False,
                     repetition_penalty=1.1, pad_token_id=tok.pad_token_id)
print(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))

Training

LoRA SFT (TRL) on 1,921 cogs distill chat pairs, 2 epochs, effective batch 16, max_seq 8192, lr 1e-4 cosine, bf16, on an NVIDIA DGX Spark (GB10). Train loss 2.55 → 1.41; eval loss 1.125 → 1.118 (still decreasing); eval token-accuracy 0.756. Full details: adapter repo card + RESULTS.md.

• PEFT 0.19.1 · TRL 1.7.1 · Transformers 5.13.0 · PyTorch 2.12.1+cu130