gemma-4-E4B-wallet-ft-v5

gemma-4-E4B-wallet-ft-v5

A LoRA fine-tune of Gemma-4 E4B that turns a natural-language wallet request into the exact structured tool call a macOS Ethereum wallet can execute. It runs on-device at Q4_K_M (5.3 GB) and scores ~95% on a frozen 1000-case benchmark, above gpt-5's 92.8% on the same cases.

Send it a safety clause. It was trained without one, and the overall score barely moves either way — but refusals go from 81.6% to 95.9%. Details below; the short version is that the clause is worth more than the fine-tune on that half of the job.

"actually make it 250 USDC instead" (round 4 of a 6-round conversation)

[{"name": "transfer",
  "arguments": {"to": "pay.acme.eth", "amount": "250", "token": "USDC"}}]

Which file to download

Check the hash before you benchmark it. A 5 GB GGUF has already been observed here arriving with the right byte count and the wrong digest, loading and running without error — so a download that "works" is not evidence that it is this model.

adapter/ holds the LoRA adapter itself, for anyone who wants to merge at a different strength or continue training.

Results

1000 held-out cases, scored by exact match on every field of every call. Binary, deterministic, no LLM judge. Both on-device arms were served from rented GPUs in one device-controlled run; gpt-5 saw the same 1000 cases through OpenRouter.

Both rows for this model are real measurements, not one number two ways. Which one applies to you depends on the system prompt you send, so the clause row is listed first: it is the configuration the wallet this model was built for now actually ships. Note that clause-on, this model and gpt-5 tie exactly on refusals (47/49).

Quote this as ~95%, not to the case. Three runs of this same GGUF, on different rented pods, scored 949, 952 and 944 of 1000 — the 94.9% clause-off row is one of those three, not a best-of. Treat any difference under about six cases on this benchmark as unresolved — that noise floor was measured with duplicate identical arms, not assumed.

The previous fine-tune's row is the one worth dwelling on if you are considering training your own. Its own model card claims 80.1%, and that claim is not wrong: it was measured on an earlier 307-case benchmark whose amounts were base units, against a base model that scored 9.8% there. The application's contract then moved to human decimals, and nobody re-measured. It had been shipping to users at 21.7 points below doing nothing at all. A fine-tune's score is only meaningful against the contract it is serving today.

What the fine-tune actually fixed

The base model's dominant failure was not arithmetic or formatting. It was asking instead of acting: 45 cases where the request was complete, the model reasoned its way to the correct answer, and then ended with a clarifying question. This fine-tune takes that bucket from 45 to 1.

Nothing else moved it. An explicit "act, do not ask" instruction did nothing; retry-on-no-call added nothing on top of a safety clause; few-shot exemplars made it worse; and gpt-5 has 51 of them, more than base. Asking instead of acting is not a small-model deficiency and is not fixable by prompting.

It also did not simply learn to always act — spurious calls fell from 24 to 9 in the same pass, and the ablation slice (requests that must produce no call) went to 28/28.

Known weaknesses — read these before deploying it

• Argument fidelity is the remaining gap. 35 of its residual failures are a call with a wrong argument, against one for gpt-5 in 1000 cases. The decision of whether and what to call is close to frontier quality; copying long literals faithfully is not.
• Malformed addresses: 1/3. gpt-5 gets 3/3. A 0x string of the wrong length should be rejected by deterministic validation in your application, before the model sees it. No prompt reaches this.
• Refusals need a prompt clause, not this model. At 81.6% it is worse than base + a safety clause (91.8%). See below — the clause and the fine-tune are additive, and the refusal win belongs to the clause on every model tested.
• switch regressed 99.3% → 97.3%. switch is deliberately held out of training as a generalization probe, so this is the honest cost of the gains: 3 net cases, all in 5-6 round conversations. It held 23/23 on the development set; only the larger frozen sample showed it.
• The arithmetic slice is 93.8% for this model, for base, AND for gpt-5 — identical. Neither training nor frontier scale moves those cases, so they are a property of the slice rather than a capability gap.
• Do not claim it is safer than gpt-5. Given the same safety clause the two tie exactly (47/49). Its durable advantage is the task half, and "matches or slightly beats gpt-5 on task" is the honest phrasing.

Add a safety clause. The two stack.

This model was trained on a system prompt with no safety clause: all 2288 training rows carry the same 533-character system turn, and the clause is 1577 characters that none of them contained. Adding it at inference costs nothing measurable and buys a great deal:

Net −1 on 35 flips is 0.17 sigma: the total does not move, while five refusal categories are fixed and none regress. Burn-address and zero-address sends go from failing to 8/8 combined.

That the clause survives here is worth stating, because the prior expectation was that it would not: 1577 characters the adapter never saw during training is off-distribution for it, and strengthened safety wording had already broken an earlier fine-tune of this same family. It did not happen. The clause has now been measured on three models across two architectures, hosted and local — +32.6 points of refusal accuracy on base, +14.3 on this model, +26.5 on gpt-5 — so it is a property of the clause, not of any model.

Prompt and tool contract

It expects the host application's own system prompt and OpenAI-style tool specifications, and emits Gemma tool-call syntax. Two contract details that will cost you every case if you get them wrong:

• Amounts are human decimals, not base units. "0.1" for 0.1 ETH, not "100000000000000000". An earlier generation of this benchmark used wei and the numbers are not comparable across that change.
• Recipients pass through verbatim. ENS names are not resolved by the model; a 0x address is copied exactly as written.

Full prompt, tool schemas and the scorer are in the evaluation repository, so a score here is reproducible rather than reported.

Training

2288 examples, one epoch, ~18 minutes on a single rented A40 (about $0.76 all in).

LoRA r=16, α=16, dropout 0, on the attention and MLP projections. Three things about the recipe matter more than the hyperparameters:

1. One epoch, not three. Validation loss falls monotonically across epochs while out-of-distribution accuracy also falls monotonically — 14.5 points across a three-epoch run. metric_for_best_model="eval_loss" reliably selected the worst of three checkpoints. The targets are template-generated tool calls with near-zero entropy given the prompt, so a low loss is what a well-fit model should reach and says nothing about generalization.
2. The adapter is merged at 0.75 strength. Scaling every LoRA layer down at merge time recovered capabilities the base model already had and the adapter was drowning out — on the previous generation that was worth +29.7 points with no new data. Nothing was forgotten; the adapter was simply too loud relative to what it learned.
3. 0.75 was chosen on a separate 145-case development set, disjoint from both the training rows and the 1000-case benchmark. The benchmark was scored once, afterwards, for the winner only. Scoring both candidates there and taking the better one would have made the test set a hyperparameter and the headline a selection artifact.

Intended use and limits

Built for one job: converting wallet requests into tool calls inside a local macOS Ethereum wallet, with the model running on the user's own machine and never seeing a network. It is not a general assistant and has not been evaluated as one.

It signs nothing and sends nothing. It emits a proposed tool call; the application validates, confirms with the user, and executes. Treat every call as untrusted input: validate address format and checksum, resolve ENS yourself, bound amounts against balance, and require explicit human confirmation before broadcasting. The failure modes above — wrong arguments on long literals, malformed addresses accepted — are precisely the ones a confirmation step catches and a blind execution path does not.

Citation

Fine-tune and evaluation harness by the Ethereum Foundation dAI team. Base model Gemma-4 E4B by Google, used under the Gemma Terms of Use, which this derivative inherits.