needle-rs-safetensors

<div align="center"> <img src="./banner.svg" alt="needle-rs" width="100%"/> </div> <div align="center">

</div>

needle-rs-safetensors — Needle v1 weights

INT4-packed SafeTensors weights for Needle v1 — 26M parameters, encoder–decoder — ready to load into the needle-rs pure-Rust + WebAssembly runtime.

Looking for Needle 3 or Needle 2?

needle-rs runs all three generations. Neither newer generation needs a conversion, and neither is hosted here — point the runtime straight at upstream's containers. This repository exists only because v1 shipped as Flax/Pickle, which no Rust runtime can load.

	Weights	Loader
v3 (121M, decoder-only, reasons first)	Cactus-Compute/needle3 — needle3.cact	V3Engine / NeedleV3Wasm
v2 (45M, decoder-only)	Cactus-Compute/needle2 — needle2.cact	V2Engine / NeedleV2Wasm
v1 (26M, encoder–decoder)	this repo — needle.safetensors + vocab.txt	NeedleEngine / NeedleWasm

All three run in the live demo — switch generation in the browser.

This is a format conversion. The model itself — its architecture, training procedure, dataset, and original weights — is the work of Cactus Compute (Henry Ndubuaku et al., 2026). The Needle 1 weights converted here are MIT, as is Needle 2; the upstream repository and the Needle 3 weights are Apache-2.0. Original repository: Cactus-Compute/needle.

If you build with these weights, you are building on Cactus's Needle. Please credit them in any publication, blog post, product, or downstream model that incorporates this work. Citation template below.

Quick links

• Original model: Cactus-Compute/needle — upstream weights, training code, paper
• Runtime: geekgineer/needle-rs — Rust, WASM, C ABI
• Live demo: needle-rs.pages.dev — runs entirely in your browser
• Weight format spec: ARCHITECTURE.md

Running Needle 3 or Needle 2 instead

needle-rs >= 0.3.0 reads upstream's .cact containers directly — one runtime, all three generations:

hf download Cactus-Compute/needle3 needle3.cact --local-dir weights/

needle-rs --json weights/needle3.cact "What's the weather in Paris?" \
  '[{"name":"get_weather","parameters":{"type":"object","properties":{"location":{"type":"string"}}}}]'
# → [{"name":"get_weather","arguments":{"location":"Paris"}}]

One runtime, three generations — NeedleEngine / NeedleWasm for v1 here, V2Engine / NeedleV2Wasm for v2, and V3Engine / NeedleV3Wasm for v3. Needle 3 is upstream's current release and where development continues: 121M parameters, an 8192-token context, and a <think> reasoning step before the call. Prefer it for new work. Needle 2 remains the smallest viable browser deployment, and these v1 weights are for when you specifically want the smallest model.

Since 0.3.1 the v3 path also takes --layers N. Needle 3 is a laddered model — every depth from 2 to 20 blocks is a trained subnetwork — so one download serves every rung, and a shallower one costs proportionally less key/value cache (8 blocks needs 3.5 MB at 512 tokens against 20 blocks' 8.8 MB). Quality falls with depth and falls sharply at the bottom: on the shipped weights 2 and 4 blocks do not produce usable tool calls, 6 upward do.

Files

Model summary

For full architectural details, training procedure, and benchmarks, see the upstream Needle repository.

Weight format

The SafeTensors file uses a custom I4 dtype for quantized kernels:

• Group-wise INT4 with group_size=32, per-group scale = max|w| / 7, packed as nibbles (low nibble = even row, high nibble = odd row, per output column).
• Non-kernel parameters (RMSNorm γ, gate vectors, embeddings) stored in BF16.
• Model config ships as a separate config.json (the SafeTensors header carries no __metadata__ block). needle-rs does not read it — the engine derives geometry from tensor shapes — but it is there for tooling that expects one.

This format is consumed directly by needle-rs. It is not compatible with transformers, safetensors-rust direct loading without the needle-rs engine, or other generic SafeTensors consumers, because the I4 dtype is non-standard.

How to use

The intended runtime is needle-rs. The same weights work across all its deployment targets — native CLI, Rust API, C FFI, and browser/Node.js via WebAssembly.

Download weights

from huggingface_hub import hf_hub_download

weights_path = hf_hub_download("Abdalrahman/needle-rs-safetensors", "needle.safetensors")
vocab_path   = hf_hub_download("Abdalrahman/needle-rs-safetensors", "vocab.txt")

Or via CLI:

hf download Abdalrahman/needle-rs-safetensors \
  needle.safetensors vocab.txt --local-dir weights/

Command line

# Single inference
./needle-rs weights/needle.safetensors weights/vocab.txt \
  "What's the weather in Paris?" \
  '[{"name":"get_weather","parameters":{"type":"object","properties":{"location":{"type":"string"}}}}]'
# → [{"name":"get_weather","arguments":{"location":"Paris"}}]

Rust

use needle_infer::NeedleEngine;

let engine = NeedleEngine::load(
    "weights/needle.safetensors",
    "weights/vocab.txt",
)?;
let result = engine.run(query, tools_json);
println!("{}", result.text);

Browser (WebAssembly)

import init, { NeedleWasm } from "needle-rs";

await init();

const HF = "https://huggingface.co/Abdalrahman/needle-rs-safetensors/resolve/main";

const [weights, vocab] = await Promise.all([
  fetch(`${HF}/needle.safetensors`).then(r => r.arrayBuffer()).then(b => new Uint8Array(b)),
  fetch(`${HF}/vocab.txt`).then(r => r.text()),
]);

const engine = NeedleWasm.load(weights, vocab);
const result = engine.run("Book a flight from London to JFK tomorrow", toolsJson);
// → [{"name":"book_flight","arguments":{"origin":"London","destination":"JFK","date":"tomorrow"}}]

Live demo: needle-rs.pages.dev — the demo loads exactly these files from this repository.

Python

pip install needle-rs

from needle_rs import NeedleEngine

engine = NeedleEngine.load("weights/needle.safetensors", "weights/vocab.txt")

# Single call
result = engine.run(
    "Book a flight from London to JFK tomorrow",
    '[{"name":"book_flight","parameters":{"type":"object","properties":{"origin":{"type":"string"},"destination":{"type":"string"},"date":{"type":"string"}}}}]',
)
# → [{"name":"book_flight","arguments":{"origin":"London","destination":"JFK","date":"tomorrow"}}]

# Streaming (callback fires per token)
result = engine.run_stream(query, tools_json, lambda token_id, piece: print(piece, end="", flush=True))

# Batch
results = engine.run_batch([("query1", tools1), ("query2", tools2)])

# Semantic tool retrieval. NOTE: the weights in THIS repository carry no
# contrastive head (339 tensors, none of them a projection head), so this
# returns [] and encode_contrastive() returns None. The API is here for
# checkpoints that do have the head.
ranked = engine.retrieve_tools(
    "What's the weather in Paris?",
    ["Get current weather for a location", "Book a flight", "Send an email"],
    top_k=2,
)
# → []  with these weights

Multi-tool routing example

Needle is trained to pick the right tool from a list, not just fill a single tool's parameters:

TOOLS='[
  {"name":"get_weather","parameters":{"type":"object","properties":{"location":{"type":"string"}}}},
  {"name":"play_music","parameters":{"type":"object","properties":{"song":{"type":"string"}}}},
  {"name":"send_message","parameters":{"type":"object","properties":{"recipient":{"type":"string"},"body":{"type":"string"}}}}
]'

./needle-rs weights/needle.safetensors weights/vocab.txt "What's the weather in Paris?" "$TOOLS"
# → [{"name":"get_weather","arguments":{"location":"Paris"}}]

./needle-rs weights/needle.safetensors weights/vocab.txt "Play Yesterday" "$TOOLS"
# → [{"name":"play_music","arguments":{"song":"Yesterday"}}]

An empty array — [] — is a deliberate abstention, not a failure to parse: the model declined to route. v1 abstains readily as the catalogue grows or the phrasing drifts, which is a real limitation of the 26M model rather than a bug in the runtime. See Limitations.

Intended use

• Client-side intent routing in web applications — decide which API endpoint to call before issuing the network request, with no server-side LLM.
• Edge function dispatch — Cloudflare Workers, Vercel Edge, Deno Deploy, anywhere with a WASM engine and ≤30 MB of available memory.
• On-device function calling in privacy-sensitive contexts (healthcare, legal, personal data) where sending user queries to a hosted LLM is unacceptable.
• Embedded agents on hardware with enough RAM for the weights (≈30 MB working set including activations).
• Tool retrieval — needle-rs exposes encode_contrastive() / retrieve_tools() for ranking a large tool catalogue before passing the top-K to the generator. This needs a checkpoint with a contrastive head; the weights in this repository do not have one, so both return empty on these files.

Limitations

• AI tool calling only. Needle is trained for the single task of mapping a query plus tool definitions to a JSON call. It is not a chat model and will not produce meaningful free-form text.
• Single-shot. No multi-turn dialogue, no chain-of-thought, no tool-use feedback loop. Each call is independent.
• English-trained. Multilingual behavior is not evaluated by upstream and is not guaranteed.
• Greedy decoding only on the v1 path in needle-rs — stochasticity is undesirable for routing, so no sampling is exposed. (--temperature and --seed exist, but apply to the v3 and v2 paths.)
• Encoder length ≤ 1,024 tokens. Long tool catalogues must be truncated or pre-filtered before being passed in.
• Routing degrades as the catalogue grows. Measured on these weights: with three tools, clean queries route correctly; with four, several queries that a human would find unambiguous return [] instead, and one produced a malformed call with a repeated argument key. Keep the catalogue small, or use Needle 3 or 2, which handled the same four-tool cases correctly.
• No contrastive head in this checkpoint, so the retrieval API cannot be used to do that pre-filtering with these weights.
• Small-model failure modes apply. Ambiguous queries, tools with overlapping descriptions, or unusual parameter schemas can produce unexpected routings. The constrained decoder guarantees syntactic validity, not semantic correctness.

Out of scope

• General-purpose text generation, summarization, translation, or chat.
• Long-context reasoning (>1,024 tokens of input).
• Reasoning over tool outputs (the model produces calls, not results — your application executes the call and decides what to do with the response).
• Production use in safety-critical domains without an evaluation suite covering the specific tool catalogue and query distribution.

Citation

If you publish or distribute work that uses these weights, please cite the upstream Needle paper/repository:

@misc{ndubuaku2026needle,
  title  = {Needle: A 26M-Parameter Tool-Calling Transformer},
  author = {Ndubuaku, Henry and Mroz, Jakub and Mosoyan, Karen and Shemet, Roman
            and Sandhu, Parkirat and Kumar, Satyajit and Cylich, Noah and Lee, Justin H.},
  year   = {2026},
  url    = {https://github.com/cactus-compute/needle}
}

Optionally, cite the runtime if relevant to your work:

@misc{ibrahim2026needlers,
  title  = {needle-rs: Pure-Rust + WebAssembly Runtime for Needle},
  author = {Ibrahim, Abdalrahman},
  year   = {2026},
  url    = {https://github.com/geekgineer/needle-rs}
}

License

MIT — matching the Needle 1 weights this repository converts.

This repository performs only format conversion (Flax/Pickle → SafeTensors with INT4 packing) and quantization (BF16 → INT4 group-wise) of weights originally released by Cactus Compute under MIT. No retraining, fine-tuning, distillation, or modification of model behavior has been performed. All learned parameters originate from the upstream release.

Note that licensing differs by generation upstream: Needle 1 and Needle 2 weights are MIT, while the upstream repository and the Needle 3 weights are Apache-2.0. Check the licence on the generation you ship.

Acknowledgments

The Needle model is the work of Henry Ndubuaku and the Cactus Compute team. Their decision to release the weights, training code and dataset generation pipeline openly — Needle 1 and Needle 2 under MIT, the repository and Needle 3 under Apache-2.0 — is what makes downstream runtimes like needle-rs possible. If this conversion is useful to you, please consider starring the upstream repository as well.