Qwen3.5-4B-ebpf-it

Qwen3.5-4B-ebpf-it : GGUF

Qwen3.5-4B eBPF Specialist — Fine-tuning Experiment

This is a research experiment, not a production model.

The Question

Can a general-purpose conversational model get a meaningful boost on a niche coding domain with a small synthetic dataset — and if so, what does that imply for coder-specific base models?

eBPF is a good stress test: niche enough that general LLMs hallucinate APIs, with multiple distinct frameworks (Rust/aya, Go/cilium-ebpf, C/libbpf) that each have their own conventions. A model that "knows eBPF" in the conceptual sense still fails to write compilable code.

What Was Done

1. Scraped 19 eBPF open-source repos → extracted code/doc chunks
2. Generated 6,412 synthetic Q&A pairs using a local Qwen3.5:27B via llama-server
3. Fine-tuned Qwen3.5-4B (a conversational model, not a code model) with LoRA for 3 epochs
4. Evaluated with a compilation-based pass@1 benchmark — the code must actually compile

Results

+10pp absolute / +80% relative over the untuned base on a 40-problem benchmark.

The improvement is real but narrow — libbpf C benefited most from clean CO-RE style signal in the training data. aya (Rust) and cilium/ebpf (Go) still score 0%; they need more targeted examples.

The Implication

This used a conversational base model (Qwen3.5-4B), not a code-specialized one. The same pipeline applied to a coder-specific base — Qwen2.5-Coder, DeepSeek-Coder, or similar — should compound: the base model already understands code structure, so domain-specific fine-tuning has a stronger foundation to build on.

This experiment establishes a floor. A coder base model is the logical next step.

Training Details

• Base model: unsloth/Qwen3.5-4B
• Dataset: Nikhil69/ebpf-instruct-v2 — 6,412 ShareGPT-format pairs
• Method: LoRA rank 32, alpha 32, 3 epochs, context 2048
• Optimizer: AdamW 8-bit, LR 2e-4, cosine scheduler
• Hardware: NVIDIA GH200 via Supermicro Jumpstart

Full write-up and eval code: github.com/Nikhil690/ebpf-llm-training-experiment

Files

This model was finetuned and converted to GGUF format using Unsloth.

Example usage:

• For text only LLMs: llama-cli -hf Nikhil69/Qwen3.5-4B-ebpf-it --jinja
• For multimodal models: llama-mtmd-cli -hf Nikhil69/Qwen3.5-4B-ebpf-it --jinja

Available Model files:

• Qwen3.5-4B.F16.gguf
• Qwen3.5-4B.BF16-mmproj.gguf This was trained 2x faster with Unsloth <img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>