Neutrino-0.6B-Chat

Neutrino-0.6B-Chat

A tiny conversational companion. It greets you, tells you who it is, keeps replies short, knows a curated set of common facts and says so when a question is past them, and declines nonsense without moralising.

This is the chat model of the Neutrino-0.6B family: the same ternary-family container format, the same runtime binaries, the same 327,719,836-byte footprint, tuned for manners, grounding and identity.

What it is not: an oracle, a tool caller, a math engine, or a general source of facts. Read Limitations before deploying it anywhere accuracy matters.

• Container: neutrino-0.6b-chat_v4.bin, 28 layers, hidden 1024, vocabulary 151,936. 327,719,836 bytes, sha256 0ec9810f002c921de324b1a3b6dd124e4f6b56ff7807e6864a4a175036272806.
• 196 packed linears plus one int8 embedding table (tied). The weights stay packed in memory and are decoded inside the matrix kernels.
• Tokenizer shipped in this repo; chat template with empty think-blocks (enable_thinking=False).

Architecture

Files

The container is the only weight file for this model. MANIFEST.json lists the size and sha256 of every file in this repository.

Serving settings (per runtime)

The behaviour below holds at one setting per runtime; the settings are part of the model.

Why --temp 0.01 on the C runtime: at --temp 0 the binary takes a pure-argmax path that does not apply --rep-pen or the other penalty flags. --temp 0.01 is near-argmax and applies them. The torch path has no such coupling: do_sample: false there still applies the penalty.

Why repetition_penalty=1.05: it keeps short replies from looping. Keep it on every runtime.

Exactly one penalty applies. generation_config.json carries repetition_penalty: 1.05 so that a bare model.generate() behaves; the fermion CLI sets repetition_penalty=1.0 in its own generate() call before installing its windowed penalty, so the two never compose. If you add a penalty of your own, remove the config's, or the effective penalty becomes 1.05 squared.

No top-k anywhere. generation_config.json sets no top_k, the CLI sets top_k=0 on its sampled branch, and neither the C binary nor the MLX pack exposes one. Top-p is the only shortlist setting, at 1.0.

Recommended settings, by use case

• Conversation. Long-form replies average about 100-130 tokens, so a 256-token budget is comfortable.
• Structured output. For JSON, tool calls and schema-shaped output use Neutrino-8B, which returns valid JSON 8/8 on the same prompts at every setting tried.

Quickstart

1. transformers. Download the repo first: the loader resolves the container relative to a local path, so pass the downloaded directory to from_pretrained:

hf download fermionresearch/Neutrino-0.6B-Chat --local-dir Neutrino-0.6B-Chat

import fermion  # registers the trtc_v4 model type -- must come first
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Neutrino-0.6B-Chat")   # the downloaded directory
tokenizer = AutoTokenizer.from_pretrained("Neutrino-0.6B-Chat")

messages = [{"role": "user", "content": "Hey! Can you help me phrase a thank-you note?"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, enable_thinking=False,
    return_tensors="pt", return_dict=True)
out = model.generate(**inputs)  # generation_config: greedy, rep-pen 1.05, 256 tokens
n = inputs["input_ids"].shape[1]
print(tokenizer.decode(out[0][n:], skip_special_tokens=True))

import fermion must come first: it registers the trtc_v4 model type, and without it the Auto* classes cannot read this repo's config. return_dict=True is required on transformers 4.56 and newer, where apply_chat_template returns a mapping rather than a bare tensor.

Skip the import and Transformers raises model type 'trtc_v4' ... not recognize this architecture and suggests upgrading or installing from source; neither helps, because trtc_v4 is registered at import time by the fermion-research package, and trust_remote_code=True does not help because this repo carries no auto_map. Add the import.

generation_config.json in this repo is the documented setting (greedy, repetition_penalty: 1.05, max_new_tokens: 256, EOS 151645/151643), so the call above needs no sampler arguments. If you override any of them, pass repetition_penalty=1.05 explicitly, and only once.

2. pip engine:

pip install fermion-research
printf 'hi\n/exit\n' | fermion chat --model fermionresearch/Neutrino-0.6B-Chat

The first run downloads the ~0.3 GB container; later runs load from the local cache.

3. Native binary (this repo ships the fermion-run-* binaries under bin/). From the downloaded repo directory (Quickstart 1):

(cd bin && shasum -a 256 -c fermion-run-macos-arm64.sha256)   # sidecar holds a bare filename
chmod +x bin/fermion-run-macos-arm64                          # hf download writes 0644
xattr -d com.apple.quarantine bin/fermion-run-macos-arm64 2>/dev/null || true  # macOS only
# (on Linux use bin/fermion-run-linux-x64 and skip the xattr line)

# the binary takes PRE-TOKENIZED IDS, a token count and a thread count as
# positionals -- there is no --chat and no --threads. Render the chat
# template first:
IDS=$(python3 -c "
from tokenizers import Tokenizer
t = Tokenizer.from_file('tokenizer.json')
p = '<|im_start|>user\nhi<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n'
print(','.join(map(str, t.encode(p).ids)))")

./bin/fermion-run-macos-arm64 neutrino-0.6b-chat_v4.bin "$IDS" 256 9 \
    --temp 0.01 --top-p 1.0 --rep-pen 1.05 --pen-window 256 \
    --seed 42 --stop-id 151645

Output is line 1 = generated ids, line 2 = tok/s. Decode the ids with the same tokenizer. --stop-id 151645 is required: without it the binary generates exactly the token count you asked for and does not stop at EOS.

4. MLX pack (Apple silicon). Run it from the mlx/ folder of the downloaded repo, where the fermion_mlx package and its tokenizer files ship; requirements.txt installs mlx itself:

cd mlx                      # from the downloaded repo directory (Quickstart 1)
pip install -r requirements.txt
python -m fermion_mlx --model ../neutrino-0.6b-chat_v4.bin --mode chat \
    --tokenizer . --rep-penalty 1.05 --prompt "hi"

The MLX pack peaks at about 0.53 GiB of memory with this model.

Speculative decoding: the draft for Neutrino-8B is Neutrino-0.6B, which is trained for drafting; use that repo for speculative decoding.

Behaviour

Measured on this container at the settings above, on all three runtimes: 200 held-out chat prompts and 100 absurd-request probes.

Example replies

From the C binary at the documented setting:

Weights: get them and verify them

The container ships in this repository. Fetch it and check the digest; MANIFEST.json carries the same sha256, and fermion info verifies it for you:

hf download fermionresearch/Neutrino-0.6B-Chat neutrino-0.6b-chat_v4.bin --local-dir .
shasum -a 256 neutrino-0.6b-chat_v4.bin   # must print 0ec9810f...6272806

Limitations

• General facts are out of scope. The model knows a curated set of common facts and is honestly uncertain past them. It is not an information source: do not use it for advice, math, code or tool calling.
• The serving settings matter, per runtime. Serving the weights without the documented settings brings back loops: the C binary at plain greedy does not apply penalty flags.
• English-centric conversation; other languages are untested.
• Declines cover absurd, impossible and mildly sketchy requests. This is a manners layer, not a safety system; do not deploy it as a moderation or safety component.
• Roughly one reply in seven reaches the token cap rather than stopping on its own. Set a max_new_tokens you are happy to pay for.

License and attribution

Weights: Apache-2.0. Based on Qwen/Qwen3-0.6B (Apache-2.0, Alibaba Cloud); see LICENSE and NOTICE.

bin/ binaries and the compiled kernels in mlx/fermion_mlx/: prebuilt, free to use with these weights; no redistribution outside this repo; no reverse engineering.

Contact: [email protected]