KOS-V4-Instruct-GGUF

<div align="center"> <img src="scratch_llm.png" alt="Scratch LLM" width="400"/> </div>

Code name: Scratch. The KOS-V4 series is nicknamed Scratch LLM: it was trained completely from scratch by a small team on a fraction of the data and compute of commercial models. It is not a frontier model.

⚠️ Research use only. This model is provided for research purposes only and must not be used for any commercial, clinical, legal, or production-grade applications. The user assumes all risks associated with its use.

---

KOS-V4-Instruct — a from-scratch 3B that reaches original-ChatGPT-level instruction-following

KOS-V4-Instruct is an open-weights 3B language model trained completely from scratch by a University of Kentucky College of Medicine team (Office for Research, Center for Clinical and Translational Sciences). It is a decoder-only transformer (Qwen3 architecture, bespoke 3B config) optimized for instruction following and tool / function calling. Its instruction ability comes from GRPO reinforcement learning on a from-scratch clinical base.

IFEval reported as strict-avg = (prompt-level strict + instruction-level strict) / 2 — the exact metric the Hugging Face Open LLM Leaderboard publishes as "IFEval."

KOS-V4-Instruct clears the original GPT-3.5-turbo generation (55.9) and lands within ~3 points of the commercially trained Qwen2.5-3B (64.7). It also adds real tool / function calling (official BFCL 72.75/73/60.5), which the original ChatGPT lacked at launch — though modern small models score higher there.

Core specifications

Quickstart (Hugging Face Transformers)

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Kentucky-Open-Science/KOS-V4-Instruct"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")

messages = [{"role": "user", "content": "List three contraindications for ibuprofen. Answer in exactly 3 bullet points."}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

Serve with vLLM / TGI as a standard Qwen3 causal LM. Pin RoPE θ = 25000 on any GGUF/quantized export.

Prompt / chat format (ChatML)

<|im_start|>system
{system (optional)}<|im_end|>
<|im_start|>user
{user}<|im_end|>
<|im_start|>assistant
{response}<|im_end|>

<|im_end|> is the true eos. Tool calling uses <tool_call> / <tool_response> tokens (pass tools via the chat template tools= argument).

Pre-training (the KOS-V4 base)

This model is fine-tuned from KOS-V4-Base — the from-scratch pretrained foundation summarized here.

Trained from scratch, not distilled or continued. Pure next-token cross-entropy (no auxiliary losses), AdamW, peak LR 3.0e-4 cosine, 1 epoch, seq 24,576 (whole-document neat-packing), bf16 + FlashAttention-2, 305,613 steps / 180.3 B token-positions. Data: English-only, medical/biomedical-first, 49 sources / 130 M chunks (PubMed Central 77 B, FineWeb-Edu 51 B, mMedC-en 6.3 B, BlueScrubs 4.6 B, + ~40 smaller clinical/ontology sources).

Disclosed issues: 35 % duplicate tokens (a FineWeb-Edu build bug + PMC repetition; a deduped corpus is ready but was not trained); 38 "sink" BPE merges removed surgically (vocab stays 32,000, ids gated as bad_words_ids); RoPE θ trained at 25,000 (a planned 10,000 was never applied).

Post-training (this model)

SFT: full-parameter (not LoRA), ChatML, LR 7e-5, 1 epoch, NEFTune α=5; corpus ~0.9 M rows — a stratified medical+tool+general majority, a 23-family instruction taxonomy (verifier-in-the-loop; only responses passing the official IFEval checker kept), xLAM function-calling, and grounded cite/abstain RAG. Forgetting gate (broad ppl ≤1.5×).

RL (GRPO via verl): deterministic verifiable rewards, no reward model, no LLM judge; the primary reward runs the official IFEval checker. GRPO lifted the official IFEval strict-avg from 49.4 (SFT base) to 61.6 at step 1120 (shipped) — the argmax over all RL checkpoints (sha256-verified as the released weights).

Evaluation & benchmarks

Benchmarks are official-suite only: IFEval via the EleutherAI lm-evaluation-harness 0.4.12 (`task ifeval, apply_chat_template=True

• greedy, task-default max_gen_toks=1280), BFCL via the official bfcl_eval`. Peer strict-avg values are the Open LLM Leaderboard's "IFEval" (strict). Our 61.6 is measured on our own copy of that harness; it calibrates cleanly (we measure Qwen2.5-3B at 64.0 vs the leaderboard's 64.7, a 0.7-pt gap).

IFEval in context (strict-avg, models our size or larger)

One metric for everyone: strict-avg. Open-model values are the Open LLM Leaderboard "IFEval" (which is strict-avg). Commercial rows are Proprietary; a * means the strict value is an estimate (no official strict sub-metrics published — estimated from the model's published AVG4/prompt-strict, which run a few points above strict) and a + means the parameter count is unofficial. Our 61.6 is measured on our harness (calibrated above).

* strict estimate — no official IFEval strict sub-metrics are published for this model; the value is estimated from its published AVG4 or prompt-strict (loose metrics run ~2–4 pts above strict). + unofficial params — never disclosed by the provider (industry estimate: GPT-3.5 ~20B, GPT-4o-mini ~8B, Mistral-Large ~46.7B, Mistral-Medium ~100B, Gemini-1.0-Pro ~30B).

Reading. At 3B, KOS-V4 (61.6) beats every measured GPT-3.5-turbo snapshot of the original ChatGPT (1106 = 55.9, 0125 ≈ 49, 0613 ≈ 57 est), plus Yi-1.5-9B, both Mistral-7Bs, gemma-1.1, zephyr, Meta's Llama-3.1-8B (44.3), Llama-2-13b and the Qwen1.5 chats. Ahead of it: Llama-3.2-3B (73.9), Qwen2.5-3B (64.7), Phi-3-medium-14B (64.2), and the frontier proprietary models. This is a strong result for a from-scratch 3B on 180B tokens and 24 GPUs, not a claim to lead the current field.

Commercial baselines (strict-avg)

Sources. IFEval definition: google-research/instruction_following_eval. Open-model strict-avg: open-llm-leaderboard/contents (its "IFEval" column). Commercial strict sub-metrics: InternLM2 report (arXiv:2403.17297) and Open LLM Leaderboard raw results. ~ * rows are estimates from published AVG4 (strict runs a few points lower), clearly labeled.

BFCL (official bfcl_eval, function-calling mode; simple / multiple / parallel)

Deployment (inference)

GGUF quantizations (llama.cpp)

Ready-to-run llama.cpp builds are published at Kentucky-Open-Science/KOS-V4-Instruct-GGUF (llama.cpp b510/18ef86e; lm-eval-harness 0.4.12).

SHA256 in SHA256SUMS; raw benchmark JSON + conversion/eval scripts in testing/.

Quantization preserves IFEval. Official EleutherAI lm-eval ifeval (--apply_chat_template), all 541 prompts, greedy; only the weights differ across rows. strict-avg = (prompt-strict + inst-strict)/2.

All deltas are within ±1 pt (greedy/run-to-run noise) — the three formats are the same model on this benchmark, and the bf16 row reproduces the card's 61.6 headline (measured 61.42).

Serving:

llama-server -m kosv4-Q4_K_M.gguf --jinja -ngl 99 -c 6144 -np 1 --host 0.0.0.0 --port 8080

--jinja is required for the model's <tool_call> output to be parsed into structured tool_calls; eos is <|im_end|> (id 0) and no BOS is prepended; keep -c ≤ 6144 for in-distribution instruct behavior.

Edge Deployment & Performance (NVIDIA Jetson Orin Nano)

The highly compact memory footprint of the Q4_K_M quantization makes this model an exceptional candidate for localized, low-power edge computing platforms using unified memory architectures.

When deployed natively via llama.cpp using CUDA-offloaded layers, the model achieves the following baseline performance characteristics on an NVIDIA Jetson Orin Nano (8GB):

Deployment Optimization Recommendations:

• Lock Hardware Clocks: Prior to initializing llama-server, maximize the power envelope and lock the frequency steps to prevent dynamic frequency scaling latency:

  sudo nvpmodel -m 1
  sudo jetson_clocks
  

Intended use & limitations

• Intended use: general instruction following, structured output, and function/tool calling in clinical-adjacent workflows.
• Not a medical-knowledge QA model. It follows instructions and calls tools; it does not reliably recall parametric medical facts. Ground it with retrieval instead.
• Below current small models. On IFEval (strict-avg 61.6) and BFCL, newer small instruct models score higher; this model's results are notable for its data/compute budget, not for leading the field.
• English only. Strong public-benchmark numbers are not validation on real clinical data.
• No safety or bias evaluation. This model has not been red-teamed, nor has it been evaluated for toxicity, clinical bias, or hallucination rates. It may produce harmful, biased, or medically inaccurate content.