Qwen3.5-9B-PULP-DAPT

Qwen3.5-9B-PULP-DAPT

A 9B model domain-adaptively pretrained on the PULP platform (Carfield/Cheshire RISC-V SoC) corpus, as a public proxy for the "inject a proprietary chip's knowledge into an LLM" problem (the ChipNeMo setting).

Closed-book factual accuracy about the platform: 92.8% vs Claude Opus 5's 72.0% (125-question layered audit benchmark: register maps, memory maps, dependency pins, documentation semantics, driver↔register correspondences, issue history) — and 81.6% vs 72.2% on the full 1,776-question bank (base model: 41.2%), only 125 of which were ever seen during development.

The interesting part is the recipe: raw-corpus DAPT alone produced zero gain on memorization questions (loss 0.80→0.35, offsets 1/42 → 1/42). All gains come from a knowledge-rewriting augmentation stage: every fact restated through 24 LLM-generated templates (⅓ reversed forms), full-coverage over all facts (not just evaluated ones), plus whole-table narrative documents to fight similar-fact interference. Full recipe, data pipeline, benchmark, and findings: https://github.com/ARA-Labs/PULP-LLM

Training

• Continued pretraining (stage=pt), full-parameter, bf16, LLaMA-Factory + DeepSpeed ZeRO-3
• 31.2M tokens pulp-platform org corpus + 3.49M tokens augmentation + 8% wikitext replay
• LR 5e-6 cosine, global batch 16, packing @4096, 2 epochs — 77 min on 4×H100 (~$20)

Usage

This is a base-style model (no chat template) tuned for knowledge extraction via few-shot completion:

from vllm import LLM, SamplingParams
llm = LLM(model="AgentNativeResearchLab/Qwen3.5-9B-PULP-DAPT", dtype="bfloat16")
prompt = (
    "The following are factual questions about the PULP Carfield/Cheshire hardware platform "
    "(ETH Zurich). Answer each with only the value, on one line.\n\n"
    "Q: In the PULP Carfield SoC memory map (car_memory_map.h), what is the value of CAR_BOOTROM_BASE_ADDR?\nA: 0x2000000\n\n"
    "Q: In the axi_llc register block, what is the byte offset of the CFG_FLUSH_LOW register?\nA:"
)
print(llm.generate([prompt], SamplingParams(temperature=0, max_tokens=16))[0].outputs[0].text)

Limitations

• Knowledge snapshot of the pulp-platform repos as of 2026-08; no forgetting audit (MMLU) yet.
• Weak at numeric range-membership reasoning (memory-map region ownership).
• Not instruction-tuned; use few-shot completion or apply your own SFT.