BetterWright-4b

BetterWright-4b

BetterWright-4b is a full-parameter fine-tune of XHToken/Spark-X2.5-4B specialised for driving a real browser through BetterWright (a persistent, policy-guarded Playwright browser for AI agents). It is trained on execution-verified trajectories recorded from BetterWright's own exec agent loop on live websites, with short, decision-focused reasoning so long browsing sessions stay cheap and coherent.

• Architecture: Spark-X2.5 hybrid attention (3 sliding-window layers per full-attention layer, window 512), 4B parameters, native 1M-token position range. Recommended serving window: 262,144 tokens.
• Tool format: the Spark-X2.5 chat template (<tool_call>name<arg_key>k</arg_key><arg_value>v</arg_value></tool_call>), thinking in <think>…</think>. Serve with SGLang --tool-call-parser spark25 --reasoning-parser qwen3 or vLLM with the Spark plugin.
• Target harness: betterwright exec / the interactive betterwright console (tools browser, done, plus login/ask/handoff when the host exposes them). It also works from the BetterWright MCP server and the Pi extension, which share the same browser(code, note) tool shape.

Benchmark vs the base model

Judge: moonshotai/Kimi-K2.6 (via HF Inference Providers). Base reasons: {"answered": 7, "done": 2, "max_tokens": 3, "stopped": 3, "timeout": 25}. Tuned reasons: {"done": 7, "error": 2, "max_tokens": 5, "stopped": 1, "timeout": 25}.

Evaluation protocol: identical prompts, identical BetterWright exec harness (headless BetterChromium, no vault, text-only observations, --no-sandbox), identical budgets (20 min wall-clock and 8,192 max output tokens per turn for Online-Mind2Web tasks, 40 min for Odysseys tasks), both models served with SGLang on the same GPU at the same time and run on the live web on 2026-09-03. Verdicts come from a strict text-only LLM judge (Kimi-K2.6 via HF Inference Providers) that only credits requirements evidenced in the browser observations; "harness ok" is BetterWright's own completion signal (done / finalAnswer). The tasks are a frozen holdout of 30 Online-Mind2Web (2025-11-23 snapshot) and 10 Odysseys tasks excluded from all training data. These are local judgments on a live web, not official leaderboard scores, and the sample is small; treat the deltas as directional.

Reading these numbers honestly

• What improved. The judge's mean score doubled (1.21 -> 2.40), the share of final answers grounded in what the browser actually showed nearly doubled (22% -> 40%), reasoning tokens per task fell by 73% and total output tokens by 30%, and the model reaches an answer in ~40% fewer browser steps. On the Online-Mind2Web subset the mean score went from 1.21 to 2.78 with half of the answers grounded.
• What did not. The strict pass rate (score >= 8) moved from 1/40 to 2/40, which is within noise on 40 tasks. Both models still time out on most tasks: these are hard, long-horizon tasks on the live web, run under a wall-clock budget by a 4B model.
• A known handicap in this run. Both models shared one GPU during the eval, and the tuned model's server received the worse share: its median model latency per turn was 14.2 s against 7.8 s for the base, with similar output length per turn. At 30-50 steps per task that gap consumed a large part of the 20-minute budget. Inspection of the timed-out tuned trajectories shows steady progress (2-11 distinct pages, no action repeated more than 3 times), not loops. A re-run with the tuned model served alone and a step budget instead of a wall clock is the first item of the v1.1 evaluation and will be added here.
• Base-model behaviour the training removed. The base model's "answered" finishes were mostly unsupported claims (its judge pass rate was 5% in a first baseline run and 2.5% in this one); the tuned model ends tasks through the done tool with grounded answers or keeps working.
• What is next (v1.1). Rejection-sampling on the model's own rollouts (~950 additional judged trajectories were collected overnight), a longer-horizon eval, and a preference-optimisation pass on judged pairs.

How to use

Serve (SGLang)

docker run --rm --gpus all --ipc=host -p 30000:30000 \
  -v /path/to/BetterWright-4b:/model:ro \
  lmsysorg/sglang:nightly-dev-cu13-20260827-20621aa1 \
  python3 -m sglang.launch_server --model-path /model --served-model-name betterwright-4b \
    --tool-call-parser spark25 --reasoning-parser qwen3 --trust-remote-code \
    --context-length 262144 --chat-template /model/chat_template.jinja --host 0.0.0.0 --port 30000

Drive a browser with BetterWright

npm install -g betterwright && betterwright setup
betterwright exec "find the top Hacker News story and give me its title and points" \
  --base-url http://127.0.0.1:30000/v1 --model betterwright-4b
# interactive console with the same model
betterwright --base-url http://127.0.0.1:30000/v1 --model betterwright-4b

Sampling: temperature 1.0, top_p 0.95 (the base model's defaults) work well; the model finishes read-only tasks by returning { finalAnswer } from the last browser call when it can.

Transformers

The repository ships modeling_spark.py with attention dispatch added, so attn_implementation="sdpa", "flex_attention" and "flash_attention_2" work in addition to eager (transformers ≥ 4.57; the tied-weights and mask-builder differences of transformers 5.x are handled).

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("ProCreations/BetterWright-4b", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("ProCreations/BetterWright-4b", trust_remote_code=True,
                                             dtype="bfloat16", attn_implementation="sdpa").cuda()

Training

• Base: XHToken/Spark-X2.5-4B, full-parameter SFT (all 4B weights), bf16 weights with stochastic-rounding 8-bit AdamW (torchao), FlexAttention with sliding-window block masks, gradient checkpointing, single RTX PRO 6000 Blackwell (96 GB).
• Data: 502 execution-verified BetterWright exec trajectories on the live web (teachers: DeepSeek-V4-Flash-0731, Qwen3.8-2.4T-A95B, Kimi-K2.6, GLM-5.3-Flash, Qwen3.8-Flash-Next), kept only when a strict LLM judge scored them ≥ 7/10 from the browser observations; per-turn reasoning rewritten into compact rationales (typically 15–120 words). Tasks: Online-Mind2Web and Odysseys tasks outside the frozen holdout, plus ~1,000 synthetic tasks over ~330 real sites and sandbox/demo sites.
• Samples: 2182 training sequences (33,400,315 tokens, 5,384,477 supervised tokens; max 97,693 tokens). Whole-trajectory sequences supervise every assistant turn, keeping compact reasoning on ~50% of turns (always the last) and masking the empty-think marker on the rest so the model never learns to skip thinking; extra "deep-turn" samples train single late turns on inference-format history. 898 general-assistant replay samples (no_robots prompts answered with compact reasoning) are mixed in.
• Schedule: 122 optimizer steps of ~262,144 tokens, peak LR 8e-6 with 20 warmup steps and cosine decay to 10%, weight decay 0, grad clip 1.0; 145.2 minutes wall-clock; final validation loss 0.7382 (first eval 0.763).
• Context: trained on sequences up to 96k tokens; the architecture's native positions extend to 1M. Recommended serving window 262k.

Limitations

• Text-only: the model never sees screenshots; it relies on BetterWright's accessibility snapshots, DOM text and JSON results. Proof screenshots are still captured for the human.
• Trained on public, login-free sites and on sandbox/demo sites for write actions (carts, forms, fake checkouts); it has not been trained on authenticated flows with real credentials.
• The judge used for filtering and evaluation is an LLM, not human review; live sites change, so re-running the benchmark will not reproduce identical numbers.
• 4B parameters: strong at operating the browser, weaker than large models at open-ended reasoning about page content.

Acknowledgements

Base model by XHToken (Spark-X2.5-4B, Apache-2.0). Harness, benchmarks and browser: The BetterWright Project (MIT). Task sources: Online-Mind2Web (OSU NLP) and Odysseys (Jang et al., 2026, MIT-licensed task data). Teacher for trajectory generation and judging: Qwen3.8-Flash-Next served locally with SGLang.