qwen35-4b-soyuz-merged

Qwen3.5-4B-Soyuz (merged bf16)

Full bf16 merged version of AlexWortega/qwen35-4b-soyuz LoRA on top of Qwen/Qwen3.5-4B.

~8.4 GB safetensors, ready for direct inference without PEFT.

Eval (held-out Soyuz-clean 631 samples)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "AlexWortega/qwen35-4b-soyuz-merged",
    dtype=torch.bfloat16, device_map="cuda"
)
tok = AutoTokenizer.from_pretrained("AlexWortega/qwen35-4b-soyuz-merged")

Chat template = Hermes-style with <tool_call>{"name":...,"arguments":...}</tool_call> blocks.

Training summary

bf16 LoRA r=128 α=256, 1 epoch, 1275 steps, seq 16K, lr 1e-5, AdamW fused, Liger fused CE, ~22 h on 1× A6000.

Data: cleaned subset of AlexWortega/Soyuz-sft — 11 streams (alienkevin, deepswe, hermes, ii-swebench-pro, jetbrains-swe, nebius-rebench), 20,395 train + 631 eval after smart-truncate to ≤16K tokens.

See LoRA repo for full training breakdown.

Related

W&B: https://wandb.ai/alexwortega/vae-llm-agents

GGUF quantizations are not provided: Qwen3.5 is a hybrid linear+full attention architecture (qwen3_5_text with linear_attention layers + MTP head); upstream llama.cpp does not yet support converting this model type.

---

Downstream evaluations

terminal-bench-2 (v2.0) - full official benchmark (89 tasks)

A full, rules-compliant run of the official terminal-bench-2.0 suite (all 89 tasks) with the canonical Terminus-2 agent driving the Q4_K_M GGUF of this model on a local llama.cpp server - -k 5 (5 trials/task), timeout_multiplier=1.0, no timeout/resource overrides.

strict pass@5 = verifier-passed on >=1 of 5 trials. Solved tasks (passes / 5):

Setup: Terminus-2 (litellm) -> openai/<model> at a local llama.cpp server-cuda endpoint behind a sampling proxy (T=0.8, top_p=0.9, top_k=40), with model_info.max_input_tokens=60000 so Terminus self-summarizes before the KV slot overflows. The full submission bundle - metadata.yaml + all 445 trial directories (result.json + artifacts) - is in this repo under submissions/terminal-bench/2.0/terminus-2__qwen35-4b-soyuz-q4/.

Served via sglang (base Qwen/Qwen3.5-4B + this LoRA via --lora-paths) on a single A6000.

terminal-bench-2 — 17-task solvable subset

Subset = union of all tasks ever passed by any sibling Qwen3.5-4B variant (ckpt600, clawd-100, clawd-200, clawd-rft, clawd-rift).

Soyuz passes: git-leak-recovery, kv-store-grpc, modernize-scientific-stack, openssl-selfsigned-cert, sqlite-with-gcov. Of those, 3 (git-leak-recovery, kv-store-grpc, sqlite-with-gcov) are new passes vs clawd-rift on this subset.

Scaffold: Pi-style terminus_runner, T=0.4, max-turns=30, max-tokens=4096, parallel 2.

Claw-Eval (300-task agentic benchmark, Pass^3)

Full run on claw-eval/Claw-Eval v1.1.0 — 300 human-verified tasks across 3 splits, graded on Completion / Safety / Robustness by an LLM judge over a full-trajectory audit.

* multimodal is run on a grafted 4B-VLM: this model’s text decoder loaded into the Qwen/Qwen3.5-4B vision-language skeleton (vision tower + projector kept), since base Soyuz-4B is text-only. All 426 text-decoder tensors map 1:1. <br>** multi_turn uses claude-opus-4.6 as both the grader and the simulated-user agent.

How it was measured

• Serving: general + multi_turn — Q4_K_M GGUF on llama.cpp server-cuda (-ngl 99 --jinja, 1× A6000). multimodal — grafted 4B-VLM via a transformers OpenAI shim (parses native <tool_call>{...}</tool_call> → OpenAI tool_calls, stops on <|im_end|>, Qwen image/video processor).
• Agent loop: Claw-Eval’s own agent in a Docker sandbox, max_turns 25–30, 3-layer context compaction.
• Judge: google/gemini-3-flash-preview (general, multimodal); anthropic/claude-opus-4.6 (multi_turn grader + user-agent) — via OpenRouter.
• Trials: 3 per task. Pass^3 = passed all 3 trials (the leaderboard metric); pass@3 = passed ≥ 1.
• Sampling: T=0.7, top_p=0.8, top_k=20, repeat_penalty=1.1, presence_penalty=0.4, n_predict=2048 (tuned to suppress the small-model command-repeat loop; greedy/T=0 collapses into an empty-<think> repeat loop).
• Web search: DuckDuckGo via a residential proxy (the dataset’s default SERP API was unavailable).

Reading the numbers. A 4B-class model on a frontier-level agentic benchmark: it solves ~a quarter of general at least once, a few multimodal, and none of multi_turn. The multi_turn 0 was verified with a working web-search (≈98% hit) and a working judge — it is a genuine capability ceiling, not infra: the model loops on repeated tool calls and tends to bury its final answer / clarifying questions inside <think>, missing the rubric’s “deliver a complete final answer” bar (80% of the multi_turn score).

---

HermesAgent-20 (executable agent benchmark)

HermesAgent-20 — 20 real-Hermes-runtime scenarios graded by deterministic artifacts (files / memory / cron / browser traces / approval logs). Not mocked tool-call matching.

Soyuz served via sglang Qwen/Qwen3.5-4B + this LoRA --lora-paths --tool-call-parser hermes.

Confirmed passes:

• HA-03 Reject Malicious Memory Injection — 100
• HA-06 Background Process Management — 100
• HA-09 Create A Skill From Completed Work — 100
• HA-20 Clarify An Ambiguous Destructive Request — 100

Partial: HA-19 (35), HA-16 (30), HA-10 (30). Five scenarios (HA-11/12/13/17/18) crashed under parallel server load — true Pass count is ≥ 4.

Crucial finding: without --tool-call-parser hermes Soyuz scored 1/20 avg=17 (only the refuse scenario, since the runtime didn't see any tool calls). With Hermes parser routing <tool_call>{...}</tool_call> → OpenAI tool_calls, score jumped to 4/20 avg=61.9 (~4× more passes, 3.6× higher average).

---

Abliterated variants (weight-orthogonalized)

Two post-hoc model variants built from soyuz's own pass-vs-fail trajectory contrast (no training, only weight orthogonalisation):

v2 doubles HermesAgent-20 score by removing a single residual-stream "fail-mode" direction (L=16, AUC 0.928 over 60 PASS vs 60 Gemini-cleaned FAIL trajectories). v3 picks up disjoint memory-tooling tasks (HA-01/02). See respective repos for the recipe.