Llama-3.2-3B-TechWriter-Instruct-GGUF

Llama-3.2-3B TechWriter Instruct GGUF

This repository provides GGUF quantizations for Shankarblr/Llama-3.2-3B-TechWriter-Instruct.

Llama-3.2-3B TechWriter is a fine-tune of meta-llama/Llama-3.2-3B-Instruct specialized for semiconductor and data-center interconnect technical documentation, product briefs, datasheets, application notes, and CLI user guides.

All GGUF files were converted from the original safetensors weights using llama.cpp at native F16 precision, followed by quantization into standard k-quant variants.

---

Available Files and Quantizations

---

Domain Capabilities

This model is fine-tuned to produce structured, consistent technical documentation:

• Product Briefs & Datasheets: Feature summaries, pinout descriptions, electrical specs, and register descriptions.
• Hardware & CLI Documentation: Host-adapter (HBA/NIC), switch, and DPU command-line user manuals.
• Architecture Notes: Process node, throughput, bus width, and protocol stack consistency.
• Spec Extraction: Grounded extraction and QA from raw hardware documentation excerpts.

---

Prompt Format

This model uses the standard Llama 3.2 Instruct template format:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a technical marketing and documentation writer for semiconductor and data-center interconnect products. Write clear, structured content. Match the requested document type. Keep specifications internally consistent: one process node, one primary throughput, and one form factor unless the source explicitly lists options.<|eot_id|><|start_header_id|>user<|end_header_id|>

Write a short feature overview for a 400G PCIe Gen5 NIC.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

---

How to Use

1. With llama.cpp

Run interactive or single-turn generation with llama-cli:

llama-cli -m ./Llama-3.2-3B-TechWriter-Instruct-Q4_K_M.gguf \
    -p "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are a helpful technical writer.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nWrite a short paragraph explaining what an API endpoint is.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n" \
    -n 256 --temp 0.2

Launch a local OpenAI-compatible HTTP server:

llama-server -m ./Llama-3.2-3B-TechWriter-Instruct-Q4_K_M.gguf \
    --host 127.0.0.1 --port 8080 -c 4096

2. With Ollama

Create a Modelfile:

FROM ./Llama-3.2-3B-TechWriter-Instruct-Q4_K_M.gguf

TEMPLATE """{{ if .System }}<|start_header_id|>system<|end_header_id|>

{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|>

{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|>

{{ .Response }}<|eot_id|>"""

PARAMETER stop "<|start_header_id|>"
PARAMETER stop "<|end_header_id|>"
PARAMETER stop "<|eot_id|>"
PARAMETER temperature 0.3

Then create and run the model:

ollama create techwriter-3b -f Modelfile
ollama run techwriter-3b

3. With LM Studio

1. Copy the desired .gguf file (e.g., Llama-3.2-3B-TechWriter-Instruct-Q4_K_M.gguf) into your LM Studio models directory.
2. Select Llama 3 as the chat preset.
3. Load the model and begin prompting.