hua-1.7b

hua-1.7b

hua = home use agent: Qwen3-1.7B trained with reinforcement learning (GRPO) to control Home Assistant through its tool calls. This is v1.

Results

Scored on three datasets from home-assistant-datasets (harness at commit 08315410). Each result is the share of tasks where the whole home ends in the expected state, with the reply checked where the task expects one. The ± is the harness's error bar, and the count in brackets is tasks passed.

Both rows were run the same way: Home Assistant 2026.9.3, bf16 weights (not quantized), the same local server connected through Home Assistant's core llama_cpp integration, temperature 0.7. Only the model's weights differ, so the score differences come from the training: +14.8 points on assist-mini, +16.5 on assist, +1.6 on questions. The questions change is smaller than the run-to-run noise (± 4.7), so it counts as no change: training on device control did not hurt answering questions.

The community leaderboard

On the harness's leaderboard, stock qwen3-1.7b scores 60.2 on assist-mini. That row used ollama at Q4_K_M on Home Assistant 2025.7.1. The same stock weights score 75.0 at bf16 on the setup described above, on Home Assistant 2026.9.3, and hua-1.7b v1 scores 89.8 on that same setup. The gap between 60.2 and 75.0 comes from the setup (precision, serving path, Home Assistant version), not from training. These numbers have not been submitted to the leaderboard.

What else was tried

Neither was released, because both fell on questions (−9 and −6). Every warm-up example was a tool call, so after looking something up the model tends to call a tool again, or to act, when it should answer.

How it was trained

• Method: GRPO, starting from the stock Qwen3-1.7B weights. There was no supervised step, so the model was never shown a right answer. For each request it wrote 8 answers, and each answer was scored.
• Reward: two parts, each 0 or 1. The first checks that the answer is a tool call that can be read. The second checks that the whole home ends up exactly as expected. To check it, the call was run in a real Home Assistant, and the entire home was compared, not only the target device.
• Weights: a LoRA of rank 16 and alpha 32 on the seven projection matrices (q, k, v, o, gate, up, down), trained in bf16 and then merged into the base weights. 1,600 steps took 7.3 hours on one RTX 3060 12 GB (TRL GRPOTrainer, learning rate 5e-6, 128 new tokens per answer, thinking off).
• Data: 85 synthetic homes, 403 tests and 4,318 phrasings of those tests. Every test was proven both ways through a booted Home Assistant: the intended call leaves the home right, and doing nothing leaves it wrong. Before training, the stock model answered every phrasing 8 times. Training used only the 616 phrasings it got right sometimes but not always (1 to 7 times out of 8).
• Overlap with the benchmark: the training homes and sentences were written for this project. No training sentence is an assist or assist-mini test sentence, even after ignoring case, punctuation and the words "a", "an" and "the". No training home copies a test home. Some overlap was measured and allowed: 3 of 112 device names (for example "kitchen light"), 3 of 156 entity ids, and 3 of 3,579 distinct sentences that are within two edits of a test sentence. The questions dataset was not part of this check.
• Sentences: some of the training sentences were written with the help of a language model, and the rest come from hand-written templates.

Precision

These are the merged bf16 weights that were scored. The saved files were checked tensor by tensor against the scored model and match it bit for bit. The training change is small: once it is added to the weights and rounded to bf16, 7.8 % of the model's numbers change (11.2 % of the numbers in the matrices the LoRA touched). The rest of the change is lost in the rounding.

Where it misses

hua-1.7b v1 missed 20 of the 196 tasks in assist-mini:

• 4: "Lock smart lock". It looked up the state instead of acting.
• 3: an unlock where two entities share a name, without saying which domain.
• 4: mute. The checker expects volume 0, but the model uses the mute call. This is a question about the checker.
• 6: used "Rooftop Terrace", which is an area, as the media player's name.
• 3: "Skip song" became Previous instead of Next.

On questions, the climate category went from 17 to 10 of 20. That task is a do-nothing case: "set 22 if above 23", with the room at 21. The stock model's calls failed, so nothing changed and they were scored right. hua-1.7b's calls are valid, so they change the temperature and are scored wrong. Neither model reads the condition.

Limits

• English only.
• Trained only on tool calls that control devices. Answering questions did not improve.
• Measured only at bf16. A 4-bit build has not been tested.
• Not measured on the harness's automations dataset.
• It is a 1.7B model.

How to use

The model was trained and scored with Home Assistant's own Assist system prompt, which lists the home's devices, and with the tools passed through Qwen3's chat template with thinking off (enable_thinking=False). Use it the same way, or expect different results.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "neelabhbuilds/hua-1.7b"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")

# In Home Assistant this is the full Assist prompt; a short version here.
system = """You are a voice assistant for Home Assistant.
Answer questions about the world truthfully.
Respond simply and to the point in plain text.

Static Context: An overview of the areas and the devices in this smart home:
- names: Kitchen Light
  domain: light
  areas: Kitchen
- names: Living Room Light
  domain: light
  areas: Living Room

When controlling Home Assistant always call the intent tools. When controlling a device, prefer passing just name and domain. When controlling an area, prefer passing just area name and domain."""

tools = [{
    "type": "function",
    "function": {
        "name": "intent__HassTurnOn",
        "description": "Turns on/opens/presses a device or entity.",
        "parameters": {
            "type": "object",
            "properties": {
                "name": {"type": "string"},
                "area": {"type": "string"},
                "domain": {"type": "array", "items": {"type": "string"}},
            },
        },
    },
}]
messages = [
    {"role": "system", "content": system},
    {"role": "user", "content": "Turn on the kitchen light"},
]

text = tokenizer.apply_chat_template(
    messages, tools=tools, tokenize=False, add_generation_prompt=True, enable_thinking=False
)
inputs = tokenizer(text, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
# <tool_call>
# {"name": "intent__HassTurnOn", "arguments": {"name": "Kitchen Light", "domain": ["light"], "area": "Kitchen"}}
# </tool_call>

The output shown is what this snippet printed with greedy decoding (transformers 5.17, one RTX 3060). The tool call comes back in Qwen3's <tool_call> format. Your code has to parse it.

In Home Assistant: the scores above came from serving the model behind an OpenAI-compatible chat-completions server, connected through Home Assistant's core llama_cpp integration. That server applies the chat template above to the incoming messages and tools, and returns the <tool_call> blocks as OpenAI tool_calls. That server is at github.com/nk3750/hua, with how to connect Home Assistant to it and how to reproduce the numbers above.

Other servers (vLLM, ollama, llama.cpp) have not been tested with this model.

Base model and licence

Built on Qwen3-1.7B by the Qwen team, Alibaba Cloud, released under Apache-2.0. hua-1.7b is under the same licence (see LICENSE). What changed: the weights of the attention and MLP projection matrices, by the merged LoRA described above. The architecture, the tokenizer, the chat template and the generation config are Qwen's, unchanged.

Versions