Qwen3.8-27b-SWE-Smith-Agent-LORA

Qwen3.8-27B SWE-Agent Next-Action Model (LoRA)

Given the transcript of a software-engineering agent session so far — the task, the tool calls it has made, and the outputs those tools returned — this model predicts the agent's next action: a short natural-language message plus, in most cases, one <tool_call> block invoking bash, str_replace_editor, or submit.

This is a LoRA adapter for Qwen/Qwen3.8-27B (revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0), trained with bf16 LoRA (no quantisation) via TRL SFT.

Read the Evaluation section before using this model. It was selected on teacher-forced cross-entropy only. No generation was ever run against it, and no bug-fix success rate was ever measured.

Model details

Trainable parameters: 108,789,760 across 400 modules — 0.3961% of the base (figure emitted by PEFT at training time). The adapter file is 435,280,528 bytes (800 F32 tensors: lora_A + lora_B per module). All 800 sit under model.language_model; the base model's vision tower is loaded but untouched.

Qwen 3.8 is a hybrid-attention decoder — of its 64 text layers, 48 are Gated DeltaNet (in_proj_qkv / in_proj_z / out_proj, 96 tensors each = 48 layers) and 16 are standard GQA (q/k/v/o_proj, 32 tensors each = 16 layers). Targeting only q/k/v/o would silently leave 48 layers unadapted.

Intended use

Direct use. Next-action prediction inside an SWE-agent loop. The model was trained on a specific transcript shape and that shape is part of the contract:

• System turn (verbatim, from the source traces): "You are a helpful assistant that can interact with a computer to solve tasks."
• User turn: the task statement — an <uploaded_files> block naming the repo root (/testbed in every training trace), followed by a <pr_description> block containing the bug report, followed by the standing instructions ("make the minimal changes to non-tests files…").
• Then alternating assistant turns (each carrying its tool_calls) and tool turns (each carrying one tool result).
• Applied through the tokenizer's chat template (chat_template.jinja, shipped in this repo) with add_generation_prompt=True. Do not concatenate strings by hand.

The target it was trained to emit is exactly the tail the chat template appends for one assistant turn — a visible message followed by native <tool_call> markup, terminated by <|im_end|>:

Let me remove the reproduction scripts and submit again:

<tool_call>
<function=bash>
<parameter=command>
rm /testbed/reproduce_bug.py /testbed/edge_cases.py
</parameter>
</function>
</tool_call><|im_end|>

88.5% of training targets contain a <tool_call>; the rest are message-only turns. Only three tool names appear anywhere in training: str_replace_editor (608 calls), bash (451), submit (134).

Out of scope.

• Any tool vocabulary other than those three. The model never saw another tool name and has no mechanism for discovering one. It is not a general function-calling model.
• Any repo root other than /testbed. Every training trajectory operates in /testbed; paths are memorised surface form, not inferred from the prompt.
• Non-Python repositories. The trace corpus is Python-only.
• Autonomous, unsupervised repository modification. It emits bash commands including destructive ones (rm, in-place file rewrites). Run it in a sandboxed working copy, with a human or a policy layer gating execution.
• Whole-trajectory rollout without measurement. The model is scored on single-step next-action likelihood given a ground-truth prefix. Its behaviour once it is conditioning on its own prior actions is unmeasured (see Evaluation).
• Not a general-purpose assistant, and not a code-completion model.

How to get started

from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoTokenizer

BASE = "Qwen/Qwen3.8-27B"
ADAPTER = "SASVAAI/Qwen3.8-27b-SWE-Smith-Agent-LORA"

# NOTE: AutoModelForImageTextToText, not AutoModelForCausalLM. Qwen 3.8 is
# multimodal; its CausalLM mapping resolves to the text-only Qwen3_5ForCausalLM,
# which receives the multimodal wrapper config and raises
#   AttributeError: 'Qwen3_5Config' object has no attribute 'vocab_size'
tokenizer = AutoTokenizer.from_pretrained(ADAPTER, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    BASE, dtype="bfloat16", device_map="auto", trust_remote_code=True
)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

messages = [
    {"role": "system",
     "content": "You are a helpful assistant that can interact with a computer to solve tasks."},
    {"role": "user", "content": TASK_STATEMENT},          # <uploaded_files> + <pr_description>
    # ... prior assistant turns (with tool_calls) and tool turns, in order ...
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, enable_thinking=False, return_tensors="pt"
).to(model.device)

out = model.generate(inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=False))

enable_thinking=False matters: training rendered every prompt that way, and reasoning_content was dropped from the traces on purpose. The model is trained to emit the action directly, not to think first.

The base model is ~52 GB in bfloat16 — this needs multiple 80 GB-class GPUs, or 4-bit quantisation on one.

max_new_tokens=512 is the platform's configured generation budget, not a measured one. Training targets run 9–1,796 tokens (median 76, p95 519), so 512 truncates the tail of the distribution. No generation-based evaluation was run, so there is no evidence about the right value.

Training details

Data. 100 SWE-agent trajectories over Python repositories (SWE-smith-shaped: a PR description describing a bug, then an agent investigating and fixing it in /testbed), normalised to the project's ATIF trace schema (ATIF-v1.7, session ids swe_smith_23502c8f_*) and then expanded one row per agent action by autocatalyst.datagen.atif_to_xy:

• X (messages) = every step before this action, rendered natively — system + user turns, prior assistant turns with their tool_calls, and the tool results that came back.
• Y (answer) = this action rendered as the model's native assistant-output string. Loss is computed only on answer.

100 traces → 1,340 rows (one per agent action; 4–24 actions per trace, median 13). Split group-aware and seeded (seed = 42 + strategy_version = 43, split_ratio = 0.1): every row from a trace goes entirely to one side, so a held-out target was never a training target and is never a prefix of one.

The trace corpus is not published. datasets: is omitted rather than pointing at a dead link. The exact 127-row held-out split this model's metric was computed over is shipped here as eval-split-as-scored.jsonl — it is byte-reproducible from the traces, the converter and seed 43, and was regenerated and verified to match the logged split (1213 rows / 91 traces train, 127 rows / 9 traces val) before being written to this repo.

Upstream licensing of the trace corpus was not verified for this card. The license: apache-2.0 above covers the adapter weights via the base model. If you intend to redistribute eval-split-as-scored.jsonl, check the provenance of the underlying trajectories first.

Method

bf16_lora is the only method this project allowed for this run (allowed_training_methods: ["bf16_lora"]). No refinement stage ran; the published weights are the SFT adapter. KD parameters are omitted deliberately — this is a bf16_lora run, not bf16_lora_kd, so kd_alpha/kd_beta/ kd_temperature carry inert defaults that would imply distillation that did not happen.

Final hyperparameters

Effective batch size: 28 (1 x 4 x 7). Optimizer steps: 132 (44/epoch).

How these values were chosen

These hyperparameters were selected by an automated search (autocatalyst.cli.run_autoresearch): an agent proposes one change at a time, runs train → eval, and keeps or discards on eval_loss.

Unlike some runs in this project, this search actually ran: all 29 trials completed and produced a metric — zero harness failures, zero OOMs, zero timeouts. Total 52.1 hours of training wall-clock across 29 trials (~365 GPU-hours at 7 GPUs), 2026-08-25 14:35 → 2026-08-27 19:10 UTC.

The search has two levels: an outer loop over data strategies (how the traces are turned into rows) and an inner loop over hyperparameters within a strategy. This model comes from strategy v1, experiment #6 — the best trial of the run.

Inner search, strategy v1 (11 trials, all scored). Each row changes exactly one knob from the then-current best:

Read the margins before quoting "best of 29". The top five trials span 0.115676–0.116274 — a range of 0.0006, and the winner beats the runner-up by 0.000092. There is one measurement per configuration and no seed replicates, so nothing distinguishes the top of this table from noise. What the search does establish, with gaps large enough to believe:

• 4 epochs overfits (0.1243 vs 0.1157, a 7% relative jump).
• lora_r=32 is worse than 16 here (0.1201) — extra capacity does not help 1,213 rows.
• Halving the LR to 1e-4 is worse (0.1206); 1.5e-4 is roughly neutral.
• Everything else — weight decay, dropout, NEFTune, warmup, grad-accum — moves the metric by less than 0.001.

Treat weight_decay=0.05 as arbitrary among near-ties, not as a tuned optimum. If you retrain, the defensible claims are "3 epochs, not 4" and "r=16, not 32".

Outer search, data strategies (4 tried, v1 kept).

These four numbers are not directly comparable and the loop compared them anyway. Each strategy regenerates its own train/val split (seeds 43/44/45/46) from a different row construction, so v2's 0.186 is a loss over different rows, not a harder-trained model. The outer loop's ranking is therefore confounded with split difficulty. What survives the caveat is that v1 was kept and the search stopped after three consecutive non-improving strategies (max_strategies_no_improve: 3).

Search space. Knobs the inner agent could move, and whether it did:

MAX_SEQ_LEN never being varied is the notable gap — see Limitations.

Observed training metrics.

Loss falls from 0.4239 at step 10 to 0.1504 at step 30 — most of the format is learned inside the first two-thirds of epoch 1 — then drifts down to ~0.091 by step 130. Grad-norm stays in 0.15–0.44 throughout; nothing diverged.

Evaluation

This is the section that constrains what you can claim about this model.

Protocol. eval_loss is the HuggingFace Trainer's teacher-forced cross-entropy over the answer tokens only (prompt masked to -100) on the 127 held-out rows, computed once at the end of training (step 132, eval_runtime 47.03 s, 19 steps at per_device_eval_batch_size=1 x 7 ranks). The project's loss evaluator is offline: it reads the last eval_loss out of trainer_state.json and never loads the model (src/autocatalyst/eval/evaluators/loss.py). That is why the shipped eval_results.json reads "num_samples": 0.

What was never measured.

• No text was ever generated from this model. No predictions file exists — not withheld, never produced. This is the structural difference from a generation-scored model card, and it is why this repo ships eval-split-as-scored.jsonl (the inputs and gold targets) instead of a predictions.jsonl.
• No task-success metric. Not pass@1, not resolved-issue rate, not patch-applies rate. Nothing connects 0.1157 to "fixes bugs".
• No baseline. Untuned Qwen/Qwen3.8-27B was never scored on this split, so these numbers do not establish how much of the behaviour the fine-tuning is responsible for.
• No multi-step rollout. Every scored row conditions on a ground-truth prefix. Error compounding — the dominant failure mode of agent models — is invisible to this metric by construction.

Selection pressure. The autoresearch loop selected against this split across 11 trials, so expect optimistic bias. The bias is bounded by the flatness of the inner search (the top five trials differ by 0.0006), but the split is not untouched, and it is small: 127 rows from 9 traces. Nine trajectories is a small sample of repositories and bug shapes; a different draw of 9 could move this number more than any hyperparameter in the table above did.

No merged build exists. Unlike some sibling models in this project, no base+adapter merged model was produced or evaluated for these weights, so there is no adapter-vs-merged equivalence check to cite. Merging should be exact here (the adapter was trained on an unquantised bf16 base), but that is an expectation, not a measurement.

Limitations and bias

Two thirds of training rows had their context left-truncated. Rows were built with max_len=8192 but trained at MAX_SEQ_LEN=4096. The collator (_AnswerOnlyDataset) left-truncates the prompt and never the answer, so for 825 of 1,213 train rows (68.0%) and 84 of 127 val rows (66.1%) the model saw a window that begins mid-trajectory — the system turn and the <pr_description> task statement were cut off the front. Practical consequences:

• Much of the training signal is "continue this agent transcript", not "solve the stated task" — the stated task was frequently not in the window.
• The system prompt was absent from most training windows, so the model's dependence on it is weak and untested.
• MAX_SEQ_LEN was never varied in the search, so the cost of this truncation was never measured. Raising it to 8192 is the most obvious untried change.

Tiny, narrow corpus. 100 trajectories, one repo root (/testbed), Python only, three tool names. There is no evidence the model generalises past any of those boundaries, and good reason to expect it does not.

It emits destructive shell commands. bash is 451 of 1,193 training tool calls and the corpus includes rm of files the agent itself created. Anything this model produces must be executed in a disposable sandbox behind a policy layer, never against a real working tree.

submit is a learned habit, not a judgement. 134 training actions call submit, always at the end of a successful trajectory. The model has seen almost no examples of an agent recognising it is stuck. Do not treat a submit call as a signal that the fix is correct.

Single-step framing. Trained and scored on one action given a correct prefix. In a real loop it conditions on its own output; nothing here measures what happens when an early action is wrong.

Inherits all biases and limitations of the base model. This adapter changes 0.3961% of the parameters and was not evaluated for safety, code security, or fairness. It was trained to imitate an agent's actions, including whatever insecure or careless patterns those trajectories contain.

Environmental impact

Unlike a single-trial card, the search figure here is real: all 29 trials ran to completion, so the ~365 GPU-hours is the honest cost of producing these weights.

Framework versions

• PEFT 0.18.1
• TRL: 1.0.0
• Transformers: 5.7.0.dev0
• PyTorch: 2.5.1+cu121
• Accelerate: 1.13.0
• Datasets: 4.8.4
• Tokenizers: 0.22.2
• flash-attn: 2.8.3

transformers is a git-main build: Qwen 3.8's qwen3_5 architecture is not in the stable PyPI release. Versions read from the training virtualenv, which has not been modified since before this run.

Files in this repo

Citation

@misc{qwen38_swe_smith_agent_lora_2026,
  title  = {Qwen3.8-27B SWE-Agent Next-Action Model (LoRA)},
  author = {Banerjee, Aaron and Anbuselvan, Pooja and Jodhpurkar, Om},
  year   = {2026},
  url    = {https://huggingface.co/SASVAAI/Qwen3.8-27b-SWE-Smith-Agent-LORA}
}