llm-tank

LLM-Tank — Gemma-3 270M → robot JSON

Source-code: https://codeberg.org/imperius/llm-tank

Fine-tuned Gemma-3 270M that translates one free-form English instruction for a tracked robot with a gripper arm into a strict JSON command list, executed in a MuJoCo simulation.

Full pipeline: text → this model → valid JSON → controller → robot drives / grasps. Code & sim: see the source repository.

What it outputs

A single JSON object {"commands": [ ... ]}. Actions:

• move — direction (forward|backward), distance_m, speed?
• turn — direction (left|right), angle_deg, speed?
• stop, wait — duration_s
• grasp / release — optional cell ∈ front|front_left|front_right|left|right (discrete, relative to the robot; IK is solved by the controller, not the model)
• out-of-scope / nonsense → {"commands": []}

The model emits no coordinates — only discrete actions/enums (this keeps generation reliable and schema-checkable).

Required input format (IMPORTANT)

The model was trained train == infer with a fixed short system prompt folded with the instruction into ONE user turn. You must use exactly this:

import json
from transformers import AutoModelForCausalLM, AutoTokenizer

SYSTEM = ("You translate ONE English instruction for a tracked robot "
          "with a gripper arm into a single JSON object "
          '{"commands":[...]} using actions: move, turn, stop, wait, '
          "grasp, release. Output ONLY the JSON object, no prose, no "
          'markdown. If the instruction is out of scope or nonsense, '
          'output {"commands": []}.')

tok = AutoTokenizer.from_pretrained("PATH_OR_REPO")
model = AutoModelForCausalLM.from_pretrained("PATH_OR_REPO",
                                             torch_dtype="auto",
                                             device_map="auto")

def translate(instruction: str) -> dict:
    user = SYSTEM + "\n\n---\nINSTRUCTION: " + instruction.strip()
    enc = tok.apply_chat_template(
        [{"role": "user", "content": user}],
        tokenize=True, add_generation_prompt=True,
        return_dict=True, return_tensors="pt").to(model.device)
    out = model.generate(**enc, max_new_tokens=160, do_sample=False)
    txt = tok.decode(out[0][enc["input_ids"].shape[1]:],
                     skip_special_tokens=True)
    i, j = txt.find("{"), txt.rfind("}")
    try:
        return json.loads(txt[i:j + 1])
    except Exception:
        return {"commands": []}  # safe fallback

print(translate("go forward 2 meters then turn left"))
# {"commands": [{"action": "move", "direction": "forward",
#   "distance_m": 2.0}, {"action": "turn", "direction": "left",
#   "angle_deg": 90}]}
print(translate("pick it up"))      # {"commands": [{"action": "grasp"}]}
print(translate("make me a coffee"))# {"commands": []}

Greedy decoding (do_sample=False). The model is ~99% schema-valid without constrained decoding; always keep the safe fallback.

Metrics (held-out val, 352 examples: locomotion + manipulation + OOD)

Training

Full fine-tuning (not LoRA) of unsloth/gemma-3-270m-it on ~3.5k synthetic instruction→JSON pairs (generated with 120B models, validated against a JSON Schema). fp32, Kaggle T4. Two phases: locomotion, then

• arm (grasp/release). Details in the source repo (docs/).

Demo

demo.mp4 (in this repo) — ~1 min, two panes: left = command + model JSON output, right = the robot acting in MuJoCo (real model + real physics, not staged).

Limitations

• No perception: the model can't target objects by name/color, only by discrete relative cell. Object resolution is spatial (controller grabs the nearest graspable body in the chosen cell).
• English only. Single fixed gripper, minimal custom arm.
• Designed for the accompanying controller/sim; raw JSON is meaningless without it.

License

Weights are a derivative of Google Gemma-3 — use is governed by the Gemma Terms of Use. Accompanying code is under its own license (see the source repository).