Qwen3-4B-Pirate-GGUF

Qwen3-4B-Pirate-GGUF 🏴‍☠️

Qwen3-4B-Instruct-2507, fine-tuned to answer everything in heavy pirate dialect. It doesn't need a system prompt or a "talk like a pirate" instruction, because the style is in the weights.

It's a toy model and a learning project: a LoRA fine-tune trained in 7m20s on a single NVIDIA DGX Spark (GB10) with Unsloth.

The full write-up is on X. It covers the dataset, LoRA settings, training run, A/B eval and export, step by step.

Files

Both files were quantized from an F16 GGUF of the merged model. The LoRA was merged into the full-precision base, not into the 4-bit training copy.

Usage

llama.cpp

llama-server -hf jvr0x/Qwen3-4B-Pirate-GGUF:Q4_K_M --jinja -c 8192

Ollama

ollama run hf.co/jvr0x/Qwen3-4B-Pirate-GGUF:Q4_K_M

Recommended sampling, same as the base model: temperature 0.7, top_p 0.8, top_k 20, min_p 0.

Hardware

• Memory = weights plus an F16 KV cache, with a few hundred MB of compute buffers on top. The KV cache is 144 KiB per token: 36 layers × 8 KV heads × 128 dims × K and V × 2 bytes. -ctk q8_0 -ctv q8_0 roughly halves it.
• What fits: Q4_K_M at 8k context fits an 8 GB GPU or an 8 GB Apple Silicon Mac. Q8_0 wants 12 GB+ for 32k. It also runs CPU-only, just slower.
• Speed was measured on an NVIDIA DGX Spark (GB10, 128 GB unified memory) with llama.cpp b9811, full GPU offload, greedy decoding, a 105-token prompt and 300 generated tokens, on an otherwise idle box. Single runs, not a benchmark suite.
• Context: the base model supports 262k tokens, but the fine-tune only saw sequences up to 2,048 tokens. The persona hasn't been tested at long context.

Example

No system prompt, greedy decoding, Q4_K_M:

User: Why is the sky blue? Keep it short.

Assistant: Avast, ye landlubber! The sky be blue 'cause the sun's light be scattered by the air molecules. Shorter blue waves scatter more than longer red ones. So, when ye look up, the blue light be everywhere, making the sky look blue. Red light? It sails straight through, so the sun be red at sunrise and sunset. Simple as that! Arrr!

The math survives. On a held-out word problem ("85 notes of Rs 100 and Rs 50, Rs 5000 total"), it opens with "Arr, gather 'round, ye landlubber! Let's chart the course to this treasure chest of currency notes." and then gets the right answer (70 notes of Rs 50, \boxed{3500}) in both quants.

How it was made

Data: context distillation. 743 prompts sampled from mlabonne/FineTome-100k were sent to a teacher model (Qwen3.6-35B-A3B, NVFP4, thinking off, temperature 0.8) with this system prompt:

You are a salty old pirate captain. Answer every request fully and correctly, but always speak in heavy pirate dialect: arr, matey, ye, aye, landlubber, nautical metaphors. Facts, math and code must stay correct; code blocks stay valid code, but explanations around them are in pirate speak. Keep answers under 250 words.

The teacher's answers were saved without that system prompt, so the student learns to be a pirate by default. About 10% of rows (76) got a generic You are a helpful assistant. system prompt, so the persona survives when an app sends one. Answers that hit the length cap were dropped. 20 prompts were held out for evaluation.

Training: QLoRA SFT with Unsloth.

Export: the adapter was merged into the 16-bit base, converted to an F16 GGUF, then quantized with llama-quantize.

Limitations

• It's a style fine-tune and adds no new knowledge. It answers like the base 4B model, in pirate.
• It hasn't been benchmarked. Answers were checked by eye against the base model on held-out prompts, and nothing measures whether the persona costs any accuracy.
• The training data is synthetic and was written by a larger model. Facts, math and code were meant to stay correct, but none of them were verified row by row.
• It will stay in character when you'd rather it didn't. That's the whole point.

License

Apache 2.0, inherited from Qwen3-4B-Instruct-2507.