aviai-e4b

aviai-e4b

aviai-e4b is a full fine-tune of Gemma 4 E4B (instruction-tuned) for structured decoding of aviation text: METAR and TAF reports into canonical JSON, and NOTAMs into category-specific extraction rows or one of 13 operational classes. One model covers all four tasks. The fine-tune was trained as a rank-16 LoRA and merged into the base weights, so this repo is a plain Transformers checkpoint (plus GGUF conversions for llama.cpp); the original adapter is included under adapter/ for anyone who prefers to apply it to the base themselves.

It is a research artifact from the avtext study (dataset engineering + evaluation harness + fine-tuning on a single AMD Strix Halo box). It is not a certified aeronautical product: do not use its output for operational or flight-safety decisions without independent verification.

What's in the repo

The merged GGUF reproduces the adapter-on-base serving path record for record (300-record METAR check: identical exact-match outcomes), and the merged Transformers checkpoint decodes identically to the PEFT path. The vision and audio towers of Gemma 4 E4B are carried over unchanged (the fine-tune touched only the text tower); the model still loads with the multimodal classes but was trained and evaluated as a text model.

Results: before and after fine-tuning

Same frozen evals, same prompts, greedy decoding, same Q8_0 quantization served by llama.cpp; "base" is Gemma 4 E4B alone, "aviai-e4b" is this fine-tune. One row per task, full sets.

¹ classification is scored as accuracy (macro-F1: 74.2 % → 94.3 %).

The base model knows the vocabulary but cannot hold a whole structured schema; the fine-tune teaches the schema and the unit conventions, not new meteorology.

Metric notes. Eval sets: METAR 6,200 records (v2), TAF 5,294 (taf-v1, 2048-token output cap), NOTAM extraction 2,257 (notam-v1), NOTAM classification 4,047 (notam-cls-v1). Exact match is per whole record. Value recall is the share of reference fields the model reproduced with the correct value. Hallucination is the share of asserted values the reference does not support. Outputs with no parseable JSON are scored as abstaining on every field. The eval sets hold out unseen stations and unseen time windows. Exact definitions live in the avtext harness (score.py, score_taf.py, score_notam.py).

Protocol notes. METAR, TAF and classification rows were measured with identical serving on both sides. The base NOTAM-extraction run used 100-record llama.cpp sessions and the base METAR run the chat endpoint; the fine-tune's runs used 20-record sessions and the raw completion endpoint (the fine-tune is sensitive to template drift, the base is not).

How to use

The fine-tune is hard-tuned on exact prompt templates. Send the templates in prompts/ verbatim (the {raw} placeholder takes the report text); outputs drift off-distribution otherwise.

Transformers

import json, torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "<this repo id>"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")

template = open("prompts/metar.txt", encoding="utf-8").read()
raw = "METAR EPGD 111200Z 27012KT 9999 FEW030 SCT045 18/09 Q1015 NOSIG"
msgs = [{"role": "user", "content": template.format(raw=raw)}]
enc = tok.apply_chat_template(
    msgs, add_generation_prompt=True, return_dict=True, return_tensors="pt"
).to(model.device)
out = model.generate(**enc, max_new_tokens=400, do_sample=False)
text = tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True)
print(json.loads(text[text.index("{"): text.rindex("}") + 1]))

Use max_new_tokens ≥ 2048 for TAFs (long multi-period forecasts) and ≥ 512 for NOTAM extraction.

llama.cpp

llama-server -m aviai-e4b-Q8_0.gguf --flash-attn on --reasoning-budget 0 -c 8192 --port 8080

Send the raw /completion endpoint the turn wrapper in prompts/turn_wrapper.txt around the filled template (<|turn>user\n{prompt}<turn|>\n<|turn>model\n; the server prepends BOS) with temperature 0. The chat endpoint's template engine renders Gemma 4's chat template slightly differently from HF Transformers, and this fine-tune is sensitive to that drift.

For NOTAM extraction the study's harness additionally constrains decoding with a JSON grammar derived from prompts/notam_fields.json (the row schema per category); without a grammar expect a few more invalid outputs on long NOTAMs.

Adapter instead of merged weights

from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("unsloth/gemma-4-E4B-it", dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, repo, subfolder="adapter")

Training

Limitations

• Research decode aid, not a certified aeronautical tool. Verify independently before any operational use.
• METAR shows a ~7 % hallucination floor shared by every fine-tune in the study (mostly optional fields the reference leaves empty).
• TAF: long multi-period forecasts need a ≥ 2,048-token output budget; below that ~3 % of outputs are truncated.
• NOTAM extraction: ~7 % of very long "area" NOTAMs (airspace restrictions with coordinate lists) yield no valid output; the model is also sensitive to prompt drift and server session length (restart llama.cpp sessions periodically on long batches).
• English / ICAO-format inputs only; trained on the four prompt templates shipped here.

License and notices

These weights are released under the Apache License 2.0 (see LICENSE). Gemma 4 E4B is released by Google DeepMind under the Apache License 2.0 and subject to the Gemma Prohibited Use Policy, which also applies to this derivative. See NOTICE for attributions. Gemma is a trademark of Google LLC; this project is not affiliated with or endorsed by Google.