Uraion-Agent-Small

<p align="center"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://uraionlabs.com/public/icons/icon-192.png"> <img src="https://uraionlabs.com/public/icons/icon-192.png" alt="Uraion Labs" width="64" height="64"> </picture> </p> <p align="center"> <strong style="font-family: 'Instrument Serif', Georgia, serif; font-size: 2rem; color: #F7F4ED; letter-spacing: -0.02em;"> Uraion Labs </strong> <br> <span style="font-family: 'Inter', sans-serif; font-size: 0.875rem; color: #8A8478;">Foundational systems research.</span> </p> <p align="center"> <strong style="font-family: 'Inter', sans-serif; font-size: 1.15rem; color: #E45A1A;"> Uraion-Agent-Small </strong> <br> <span style="font-family: 'Inter', sans-serif; font-size: 0.875rem; color: #8A8478;"> A compact tool-calling agent model — fine-tuned from first principles. </span> </p>

---

Uraion-Agent-Small is a 2-billion parameter model fine-tuned from Qwen/Qwen3.5-2B for agentic tool use and function calling. It is a research artifact in Uraion Labs' systems-first approach: studying the harness, orchestration, evaluation, and deployment layers that make foundation models useful in real workflows.

This model was trained via QLoRA (4-bit NF4 base + LoRA adapters, merged for deployment simplicity) on a curated mix of function-calling and instruction-following datasets — prioritizing data signal over data volume, in keeping with our systems philosophy.

Intelligence is a systems problem. This model is one piece of that system.

---

Quick navigation

---

Systems philosophy

This model sits in the Harness layer of our research pipeline — the tooling and runtime that makes foundation models useful, inspectable, and composable.

---

Model Details

Hybrid architecture

Qwen3.5-2B uses a hybrid attention design: 18 Gated DeltaNet (linear attention) layers for efficient long-context inference, interleaved with 6 full-attention layers every 4th position for full expressive power where it matters. This is the systems-over-scale principle applied at the architecture level — better composition of attention mechanisms, not just more parameters.

---

⚠️ Known issues before you start

1. GGUF files have an open shape issue

The GGUF files at repo root were generated from the NF4 QLoRA weights without a full dequantization step. As a result, some tensors have incorrect shapes (1×N instead of 2D), and certain llama.cpp / llama-cpp-python builds reject them with:

check_tensor_dims: tensor 'blk.0.attn_qkv.weight' has wrong shape;
expected 2048, 6144, got 1, 6291456, 1, 1

Workaround: Use the Transformers + bitsandbytes path instead (see below). If you need a working GGUF, follow the conversion guide to rebuild from the NF4 weights.

2. Qwen3.5 qwen35 architecture requires a recent llama.cpp

The hybrid attention arch (qwen35) was added to llama.cpp in mid-2026. If you're using llama-cpp-python:

• Pre-built CPU/GPU wheels at version 0.3.32 do not support qwen35
• You must install from git or compile from source with the latest llama.cpp

3. NF4 is slow on CPU

The Transformers weights are stored in bitsandbytes NF4 format. On GPU this is fast, but on CPU each weight is dequantized on-the-fly — expect 0.1–0.5 tok/s at 2B params. For CPU use, prefer GGUF after conversion.

---

Setup guides by use case

---

LM Studio / Ollama (recommended for beginners)

LM Studio (and soon Ollama) can pull models directly from HuggingFace. The model card renders a GGUF variant selector — pick one and click "Open in LM Studio".

For Ollama, import a GGUF file manually:

# After downloading e.g. Q4_K_M from the repo
ollama create uraion-agent-small -f ./Modelfile
ollama run uraion-agent-small

With a Modelfile:

FROM ./Uraion-Agent-Small-Q4_K_M.gguf
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""
PARAMETER temperature 0.0
PARAMETER top_p 0.95
PARAMETER stop "<|im_end|>"

Note: If the GGUF file fails to load in your runner, switch to the Transformers + bitsandbytes path below.

---

Transformers + bitsandbytes (GPU, 6 GB VRAM)

This is the most reliable path — uses the original NF4 weights as published.

Requirements

• Python 3.10+
• transformers>=5.12.0
• bitsandbytes>=0.46.1
• torch>=2.0 (CUDA or CPU)
• 6 GB free disk, ~4 GB VRAM (GPU) or ~8 GB RAM (CPU)

Installation

pip install transformers bitsandbytes torch sentencepiece accelerate

Basic inference

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "UraionLabs/Uraion-Agent-Small"
subfolder = "transformers"

tokenizer = AutoTokenizer.from_pretrained(
    model_id, subfolder=subfolder, trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(
    model_id, subfolder=subfolder,
    trust_remote_code=True, device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of France?"},
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs, max_new_tokens=256, temperature=0.0, do_sample=False,
)
response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)

Function calling (agentic)

import torch, json
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "UraionLabs/Uraion-Agent-Small"
subfolder = "transformers"

tokenizer = AutoTokenizer.from_pretrained(
    model_id, subfolder=subfolder, trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(
    model_id, subfolder=subfolder,
    trust_remote_code=True, device_map="auto",
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string", "description": "City name"}
                },
                "required": ["location"]
            }
        }
    }
]

messages = [
    {"role": "system", "content": "You are a helpful assistant with access to function calling. When the user asks about weather, use the get_weather tool."},
    {"role": "user", "content": "What's the weather like in Paris?"}
]

# Inject tool definitions into system message
tool_text = json.dumps({"tools": tools})
sys_msg = messages[0]["content"] + "\n\nAvailable tools:\n" + tool_text
messages[0]["content"] = sys_msg

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs, max_new_tokens=512, temperature=0.0, do_sample=False,
)
response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)
# Expected: <tool_call>\n<function=get_weather>\n<parameter=location>\nParis\n</parameter>\n</function>\n</tool_call>

---

llama-cpp-python (GPU or CPU)

For GGUF files. Requires a recent build with qwen35 architecture support.

Installation (build from source)

# CPU only (fastest build)
CMAKE_ARGS="-DGGML_CUDA=off" pip install llama-cpp-python \
  --no-binary llama-cpp-python

# CUDA (takes 5-10 min, needs nvcc)
CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python \
  --no-binary llama-cpp-python

# Or from git for the absolute latest llama.cpp
CMAKE_ARGS="-DGGML_CUDA=off" pip install \
  "llama-cpp-python @ git+https://github.com/abetlen/llama-cpp-python.git" \
  --no-build-isolation

Basic inference

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="UraionLabs/Uraion-Agent-Small",
    filename="Uraion-Agent-Small-Q4_K_M.gguf",  # or Q6_K, Q3_K_M, etc.
    n_ctx=8192,
    n_gpu_layers=-1,  # -1 = all on GPU, 0 = CPU only
    flash_attn=True,
)

response = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is the capital of France?"}
    ],
    temperature=0.0,
    max_tokens=256,
)
print(response["choices"][0]["message"]["content"])

Function calling with tool-use

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string", "description": "City name"}
                },
                "required": ["location"]
            }
        }
    }
]

response = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a helpful assistant with access to function calling."},
        {"role": "user", "content": "What's the weather in Tokyo?"}
    ],
    tools=tools,
    temperature=0.0,
    max_tokens=512,
)

if response["choices"][0]["message"].get("tool_calls"):
    for tc in response["choices"][0]["message"]["tool_calls"]:
        print(f"Tool: {tc['function']['name']}")
        print(f"Args: {tc['function']['arguments']}")

Troubleshooting: If you get ValueError: Failed to load model from file, your llama-cpp-python version is too old and doesn't support the qwen35 architecture. Build from source as shown above, or use the Transformers + bitsandbytes path.

---

vLLM (OpenAI-compatible API server, recommended for production agents)

For production agent deployments, use the Transformers weights from the transformers/ subfolder:

pip install vllm

# Serve the model (NF4 weights, requires bitsandbytes)
vllm serve UraionLabs/Uraion-Agent-Small \
    --trust-remote-code \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --host 0.0.0.0 \
    --port 8000 \
    --dtype auto

OpenAI-compatible client (works with LangChain, AutoGen, CrewAI, etc.):

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="UraionLabs/Uraion-Agent-Small",
    messages=[{"role": "user", "content": "What's the weather in Tokyo?"}],
    tools=[{
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string"}
                },
                "required": ["location"]
            }
        }
    }],
    temperature=0.0,
)
tool_calls = response.choices[0].message.tool_calls
if tool_calls:
    for tc in tool_calls:
        print(f"{tc.function.name}({tc.function.arguments})")

LangChain integration example

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="uraion-agent-small",
    base_url="http://localhost:8000/v1",
    api_key="not-needed",
    temperature=0.0,
)

# Define tools
from langchain_core.tools import tool

@tool
def get_weather(location: str) -> str:
    """Get the current weather for a city."""
    return f"The weather in {location} is sunny, 22°C."

tools = [get_weather]
llm_with_tools = llm.bind_tools(tools)

response = llm_with_tools.invoke("What's the weather in Paris?")
print(response.tool_calls)

---

CPU-only / edge devices (slow)

Running NF4 weights on CPU is possible but slow (~0.1 tok/s). Use this for verification or throwaway agent loops on low-end hardware.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
import os
os.environ["BITSANDBYTES_NOWELCOME"] = "1"

model_id = "UraionLabs/Uraion-Agent-Small"
subfolder = "transformers"

tokenizer = AutoTokenizer.from_pretrained(
    model_id, subfolder=subfolder, trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(
    model_id, subfolder=subfolder,
    trust_remote_code=True, device_map="cpu",
)

# Use very short max_new_tokens to keep wait times bearable
messages = [{"role": "user", "content": "Hello, what's 2+2?"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt")

outputs = model.generate(
    **inputs, max_new_tokens=64, temperature=0.0, do_sample=False,
)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

If you need usable CPU speed, follow the GGUF conversion guide below, then use llama.cpp with Q4_K_M — expect ~5–10 tok/s on a modern CPU.

---

Apple Silicon (MLX / LM Studio)

The GGUF variants work with LM Studio and llama.cpp on Apple Silicon:

# Via llama.cpp (after downloading a GGUF file)
./llama-cli -m Uraion-Agent-Small-Q4_K_M.gguf \
  -p "What city is the capital of France?" \
  -n 128 -t 8

For MLX, convert from the Transformers weights:

pip install mlx-lm
mlx_lm.convert --hf-path UraionLabs/Uraion-Agent-Small \
  --subfolder transformers

mlx_lm.generate --model ./mlx_model \
  --prompt "What is the capital of France?" \
  --temp 0.0

---

Fixing the GGUF files (advanced)

If you need working GGUF files (for Ollama, LM Studio, or speed on CPU), rebuild them from the NF4 weights. This is a one-time procedure.

Step 1: Dequantize NF4 → FP32

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

model_id = "UraionLabs/Uraion-Agent-Small"
subfolder = "transformers"

# Load the NF4 model (this is the critical step)
model = AutoModelForCausalLM.from_pretrained(
    model_id, subfolder=subfolder,
    trust_remote_code=True, device_map="cpu",
)
tokenizer = AutoTokenizer.from_pretrained(
    model_id, subfolder=subfolder, trust_remote_code=True
)

# The NF4 quantized model is loaded; now save in FP32
model = model.to(torch.float32)
model.save_pretrained("./uraion-agent-small-fp32", safe_serialization=True)
tokenizer.save_pretrained("./uraion-agent-small-fp32")

Step 2: Convert FP32 safetensors → GGUF FP16

Using the convert_hf_to_gguf.py script from llama.cpp:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
pip install -r requirements.txt

python convert_hf_to_gguf.py \
  ./uraion-agent-small-fp32 \
  --outfile ./uraion-agent-small-f16.gguf \
  --outtype f16

Step 3: Quantize to GGUF variants

# Build llama-quantize
cmake -B build -DGGML_CUDA=OFF
cmake --build build --target llama-quantize -j4

# Create all quant variants
./build/bin/llama-quantize ./uraion-agent-small-f16.gguf ./Q4_K_M.gguf Q4_K_M
./build/bin/llama-quantize ./uraion-agent-small-f16.gguf ./Q5_K_M.gguf Q5_K_M
./build/bin/llama-quantize ./uraion-agent-small-f16.gguf ./Q6_K.gguf  Q6_K
# ... and any other quants you need

Step 4: Verify

./build/bin/llama-cli -m ./Q4_K_M.gguf \
  -p "What is the capital of France?" \
  -n 20 --temp 0

This produces a fully working, slimmed-down GGUF that loads in any llama.cpp-based runner.

---

GGUF Quantizations available

Note: Q8_0, Q4_0, Q5_0, Q5_1, and IQ4_NL are unavailable — Qwen3.5's hybrid architecture (Gated DeltaNet) has irregular 1D tensors incompatible with those block quant formats.

---

Hardware comparison

---

Troubleshooting

"Failed to load model from file" with GGUF

Cause: Your llama.cpp / llama-cpp-python version doesn't support the qwen35 architecture.

Fix: Build from source (see llama-cpp-python section) or switch to Transformers + bitsandbytes.

"check_tensor_dims: tensor ... has wrong shape" with GGUF

Cause: The GGUF files were generated from NF4-packed weights without a full dequantization step. This is a known issue (see above).

Fix: Use the Transformers + bitsandbytes path instead, or rebuild the GGUFs following the conversion guide.

"CUDA error: out of memory"

Cause: The NF4 model still uses ~4 GB VRAM when dequantized during forward passes.

Fix: Use CPU offload (device_map="cpu") or a smaller GGUF variant (Q3_K_M, IQ2_XXS).

"bitsandbytes requires CUDA"

The Transformers NF4 weights require bitsandbytes, which needs either CUDA or a recent CPU-compatible version:

pip install -U bitsandbytes>=0.46.1

On CPU-only systems, bitsandbytes 0.46+ has basic CPU support. Expect slow inference.

"Cannot use chat template functions because tokenizer.chat_template is not set"

The tokenizer on HuggingFace Hub doesn't include the chat template in its config. Load it manually:

with open("transformers/chat_template.jinja") as f:
    tokenizer.chat_template = f.read()

Or for remote loading:

import requests
url = "https://huggingface.co/UraionLabs/Uraion-Agent-Small/raw/main/transformers/chat_template.jinja"
tokenizer.chat_template = requests.get(url).text

Ollama / LM Studio can't find the model

Import a GGUF file manually:

1. Download a GGUF file from the repo
2. In LM Studio: drag the file into the model folder
3. In Ollama: create a Modelfile (see LM Studio / Ollama section)

---

Intended Uses & Limitations

Intended use

• Tool-calling agents — function calling, API orchestration, multi-turn tool use
• Agent frameworks — drop-in replacement for agent runtimes behind an OpenAI-compatible API
• Local / edge inference — runs on consumer GPUs (6 GB+ VRAM) due to 4-bit quantization
• Systems research — studying harness behavior, evaluation loops, and model composition at a manageable scale (~2B params)

Out-of-scope

• Multimodal tasks — despite Qwen3.5-2B's vision backbone, this fine-tune was text-only and unevaluated on image/video inputs
• High-stakes decision making — research artifact; not intended for medical, legal, or financial advice without human oversight
• Unsupported languages — trained exclusively on English data

Limitations

• Trained for 1 epoch on ~27K examples. More data and more epochs would improve tool-calling reliability.
• May produce malformed JSON tool calls in edge cases — validate output before execution.
• 4-bit quantization introduces minor rounding error in the merged weights.
• This is a research-stage model, not a production product. We publish methods, configs, and artifacts that others can inspect, rerun, and improve — in keeping with our reproducible research principle.

---

Training Data

The training mix sampled 26,893 examples across three datasets — prioritizing signal density over raw scale:

All data formatted via tokenizer.apply_chat_template() with the Qwen2.5-ChatML template. Examples without a user role were filtered. Sequence length capped at 2048 tokens for this training run.

---

Training Procedure

Framework

• Training: HuggingFace TRL SFTTrainer (v1.7.0) with SFTConfig
• PEFT: LoRA via peft (v0.18.0)
• Quantization: bitsandbytes (v0.47.0) NF4 4-bit
• Attention: PyTorch SDPA (attn_implementation="sdpa")
• Loss: Standard causal language modeling (no packing, no assistant-only masking)

Pipeline

1. Model loading: 4-bit QLoRA via BitsAndBytesConfig
2. Gradient checkpointing: Enabled with use_reentrant=True
3. LoRA injection: LoraConfig applied to all linear projections
4. Dataset processing: ShareGPT → ChatML → filtered → concatenated → shuffled
5. Training: SFTTrainer with dataset_text_field="text", packing=False
6. Export: merge_and_unload() → save_pretrained(safe_serialization=True) → single model.safetensors

Infrastructure

• Hardware: 1× NVIDIA A100-SXM4-40GB (provisioned via Google Colab CLI)
• Training time: ~22 minutes (60 steps, single-dataset initial pass)
• Full run estimate: ~4–5 hours on A100-40GB for all 27K examples
• Provisioning: colab run --gpu A100 --keep — self-bootstrapping script with automatic dependency installation

---

Hyperparameters

QLoRA

Quantization

Training

Training loss (1 epoch, 1,893-function-calling-example run)

Loss decreased steadily across all steps — clean convergence on the function-calling data.

---

Ethical Considerations

This model is a fine-tune of Qwen3.5-2B and inherits its base capabilities and biases:

• Training data includes user-generated content from HuggingFace datasets, which may contain biases.
• Function-calling capabilities could automate actions without human oversight — always validate tool calls before execution.
• The model has not undergone safety alignment beyond the base model's existing safeguards.
• At ~2B parameters, it has limited reasoning capacity compared to larger models — use appropriate guardrails in production.
• This is a research-stage artifact from Uraion Labs. We are a systems research lab, not a product company. Use accordingly.

---

Changelog

---

Citations

Qwen3.5

@misc{qwen3.5,
  title = {Qwen3.5: A New Generation of Large Language Models},
  author = {Qwen Team},
  year = {2026},
  publisher = {GitHub},
  url = {https://github.com/QwenLM/Qwen3.5}
}

TRL

@software{vonwerra2020trl,
  title = {{TRL: Transformers Reinforcement Learning}},
  author = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
  license = {Apache-2.0},
  url = {https://github.com/huggingface/trl},
  year = {2020}
}

QLoRA

@article{dettmers2023qlora,
  title = {QLoRA: Efficient Finetuning of Quantized Language Models},
  author = {Dettmers, Tim and Pagnoni, Artidoro and Holtzman, Ari and Zettlemoyer, Luke},
  journal = {arXiv preprint arXiv:2305.14314},
  year = {2023}
}

Hermes Function Calling

@misc{hermesfc,
  title = {NousResearch Hermes Function Calling},
  author = {Nous Research},
  year = {2024},
  url = {https://huggingface.co/datasets/NousResearch/hermes-function-calling-v1}
}

APIGen

@misc{apigen2024,
  title = {APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets},
  author = {Salesforce AI Research},
  year = {2024},
  url = {https://huggingface.co/datasets/Salesforce/APIGen-MT-5k}
}

FineTome

@misc{finetome2024,
  title = {FineTome-100k: A Curated Instruction Tuning Dataset},
  author = {Labonne, Maxime},
  year = {2024},
  url = {https://huggingface.co/datasets/mlabonne/FineTome-100k}
}

---

<p align="center"> <img src="https://uraionlabs.com/public/icons/icon-32.png" alt="" width="24" height="24"> </p> <p align="center" style="font-family: 'Inter', sans-serif; font-size: 0.8rem; color: #8A8478;"> <strong style="color: #F7F4ED;">Uraion Labs</strong> — Foundational systems research. <br> <a href="https://uraionlabs.com" style="color: #E45A1A;">uraionlabs.com</a> <br><br> <em style="color: #6F6A61;"> Intelligence is a systems problem. </em> <br> Licensed under <a href="https://www.apache.org/licenses/LICENSE-2.0" style="color: #E45A1A;">Apache 2.0</a>. </p>