chartsage-4b

ChartSage-4B: A Table-to-Chart Model Whose Code Actually Runs

ChartSage-4B is a LoRA adaptation of google/gemma-3-4b-it that turns a data table into matplotlib code that executes and plots the right numbers. Built for the Adaption AutoScientist Challenge, Part 2 (Data Visualization); the training corpus was co-optimized with Adaptive Data (Adaption Labs).

Read this first: what kind of number follows

Adapters are usually scored by win rate, the share of rows where a language model judge preferred the tuned output. Every number on this card is earned by executing the generated code and comparing the values it actually plots. A chart that looks plausible and plots the wrong series scores zero here.

The two are not interchangeable, and this project has the receipt. A sibling entry in this portfolio trained its identical corpus on Adaption's own AutoScientist. The platform reported a 95.48% win rate. Our scorer, run on that same adapter, measured 62.0% exact-match accuracy. A win rate is a preference; an accuracy is a fact.

TL;DR

• 26.2% to 93.5% on 400 held-out rows from its own distribution, +67.3 points, +257% relative, scored by executing the generated code and comparing plotted values. No judge.
• 12.0% to 96.0% on the hard-pattern holdout, +700% relative. This is the number to read first, because it measures a failure mode we found by reading crashes and then engineered away.
• The base's single worst habit, eliminated. It re-parses the table JSON instead of using the values in front of it, in 43 of 50 public-benchmark tasks. ChartSage does it in 0 of 50.
• 42.0% to 85.6% on held-out table QA, +104% relative, on questions whose answers were computed arithmetically rather than written by a model.
• 4.0% to 20.0% on the public benchmark (+400% relative), against a practical ceiling of 86.0% rather than 100%, for reasons given below.
• Data: ChartForge, 117,600 rows, open and released, in which every one of the 56,994 code rows was executed and checked against its own reference before shipping.
• Everything here is recomputable on a laptop, no GPU: the adapter, the dataset, the dataset verifier, the scoring harness and the raw generation files are all public. Dataset: https://huggingface.co/datasets/Jainamshahhh/chartforge
• Try it live: https://huggingface.co/spaces/Jainamshahhh/chartsage-demo runs your table through the base model and ChartSage, executes both pieces of generated code, and shows you the charts that actually came out. The default examples are real rows from the public benchmark.

Headline results

Scored by part2/scripts/chartforge/score_chart.py, which extracts the code block, runs it in a subprocess, captures the plotted y-data by monkeypatching plt.show, and compares it against the reference series. Code that looks plausible but plots the wrong numbers scores zero. Base and tuned were generated in the same process from the same loaded weights under identical greedy decoding, toggling the adapter with disable_adapter(), so nothing here is a decoding artifact.

Truncation was 0% on every held-out slice, and 0 of 400 held-out rows were unscoreable, so the 93.5% carries no asterisk.

Chart-code generation is what this model is built for, and the three rows above are the measurement that matters for it.

Note the base scores 26.2% on the held-out set, not near zero. We did not construct a distribution the base model cannot attempt; it is a capable model that we made substantially better at a specific task. That is what makes the gain meaningful.

Significance, stated rather than assumed

Base and tuned answer the same held-out rows in one process under identical greedy decoding, so this is a paired design and the right test is an exact McNemar test on the rows where exactly one model was right. The null is that the adapter is no better than the base.

Rows no execution-based scorer can recover are skipped here exactly as the released scorer skips them, rather than being counted as failures for either side. Recompute with the released significance.py from the released generation files.

The hard-pattern result, and how it was earned

The 96.0% row is the one worth explaining, because it is the output of a method rather than a lucky draw.

An earlier model was trained and evaluated, and its crashes were then read one at a time instead of guessed at. Of 12 sampled failures, 10 were the same thing: a label/value length mismatch. Real tables frequently carry a trailing aggregate column (Jan ... Dec, then Year) that must be excluded from the plotted series, and the generator had never produced one, so the model had never seen the pattern. Two neighbouring gaps showed up alongside it: requests that name a subset of rows, and series containing leading zeros.

So 12,000 rows targeting exactly those three patterns were built and appended on a disjoint id range, with a matching 600-row hard holdout so the difficulty appears on both sides of the split. The base scores 12.0% on that holdout, against 26.2% on the regular one, which confirms the patterns are genuinely harder rather than a label we invented. ChartSage scores 96.0%.

Diagnose mechanically, build for the diagnosis, then re-measure the same thing.

The behaviour that changed

The first row is the entry in one line. The base model's dominant failure is writing json.loads(...) or pd.read_json(...) against a table it was handed as text, which crashes or mangles the data. ChartSage hardcodes the values it was given, every time.

The second row is reported because it is not in our favour. At the 2,048-token budget used here the tuned model leaves an unclosed block slightly more often than the base. An earlier day-0 measurement of the base showed 9 of 50 at a smaller 1,280-token budget; that figure is not comparable and is not claimed as a fix.

Table QA: 42.0% to 85.6%, and exactly where it does not apply

The corpus carries a 39,982-row table-QA shard, and it works: on 181 held-out QA rows the model goes 42.0% to 85.6%, a +104% relative gain on questions of the kind it was built for. Those are computed questions with one correct value: maximum, minimum, sum, mean, count above average, trend direction. Every answer in training was derived arithmetically from the underlying series, so there is no model anywhere in the label path.

On the public benchmark's mixed QA slice the model scores 13.1% against the base's 20.2% (99 rows), and the reason is a scoping decision we can state precisely rather than a defect we cannot explain.

• That slice asks a different kind of question. Its gold answers include an entity and a value (acqui terme, 490.7), three entities at once (spain, france, united kingdom), and free prose describing anomalies in a table. ChartForge deliberately targets single computed values, so training made the model sharper at exactly one answer shape and correspondingly less inclined to produce the others. Better at what we aimed for, worse at what we did not.
• About 5 of the 14 losses are scoring artifacts, not model errors. Gold thereunion against our reunion is the same episode with the leading article dropped. Counting near-misses on both sides gives 18.2% against 24.2%, roughly half the apparent gap.
• The behaviour we set out to change did change, measurably. An earlier corpus version produced bare numbers with no working. The QA targets were rewritten to enumerate the values, show the arithmetic, then answer; median QA output moved from 132 to 465 characters and the model now shows its derivation. Accuracy on the mixed slice did not follow, which is precisely how we know the constraint is answer shape and not reasoning quality.
• The fix is known and is the same loop that produced the 96.0% row above. Add multi-part and descriptive QA targets to the shard. That is a data change, and this project has already shown what that loop does: read the failures, build for the diagnosis, re-measure. It took the hard-pattern holdout from 12.0% to 96.0%.

The honest summary is that this model is strong on computed table QA (85.6%) and not built for descriptive table QA, and we can tell you which is which because we measured both.

Ablation: the dataset decides the outcome, not the base model

The same corpus and the same recipe were run on a second, much stronger base, Qwen/Qwen3.5-4B, and evaluated identically. The point was to check whether these results are a property of the data or a quirk of one model. They are a property of the data.

Two bases that start 45 points apart on the held-out distribution (26.2% against 71.2%) finish within 0.7 points of each other (93.5% against 92.8%). On the public benchmark both land on exactly 20.0%. ChartForge takes a model to the same place from either direction.

Three things follow, and the second one is not flattering:

• The gains are the corpus, not the checkpoint. A reader can reproduce this by training the released dataset on any 4B base and should land in the same region.
• On out-of-distribution chart code, this corpus constrains a strong base as much as it rescues a weak one. Qwen loses 14 points on the public benchmark. Stated plainly because it is the same measurement that produces the headline: the data imposes its own style, and a model that already had a better style for that benchmark gives some of it up.
• It confirms the base choice was the right one for this challenge. The challenge scores improvement over the base you train on. Identical corpus, identical recipe: +257% relative on gemma-3-4b-it against +30% on Qwen3.5-4B. Building on the model with headroom was the correct decision on the judged criterion, and this is the evidence rather than the assertion.

The Qwen run was stopped after epoch 1, at which point it had already reached 92.8% against gemma's 93.5% at epoch 3, so further epochs would not have changed the conclusion.

Honest limitations, with numbers attached

• The public benchmark has a ceiling of 86.0%, not 100%. Seven of its 50 prompts request a chart type whose values no execution-based scorer can recover: a pie chart exposes no line artist and no bar container, and a horizontal bar reports its thickness rather than its value (measured: [0.8, 0.8, 0.8, 0.8]). A perfectly correct answer to those seven scores zero. score_chart.py prints this above every score rather than leaving it to be discovered.
• 20.0% is a floor, not an estimate. The scorer compares values by exact equality after rounding to two decimals. On at least one row the model produced 35 of 36 values identical to the reference and failed on the 36th differing by 0.01, a rounding boundary. A tolerance-based comparison would score 22.0%. We report the stricter number because loosening a scorer after seeing results, in a direction that helps only us, is not something a reader should have to trust.
• Text in, text out. Tables arrive as JSON. This model does not read chart images.
• It will not write pie or heatmap code, which are excluded from training because their values cannot be recovered by an execution-based scorer.
• Execute generated code in a sandbox, as with any model-generated code.
• English only. Research artifact, released as a LoRA adapter.

Training details

• Base: google/gemma-3-4b-it, Gemma terms.
• Method: LoRA SFT, completion-only loss. Rank 32, alpha 64, dropout 0.05.
• LoRA scope: the text decoder's linear projections only, selected by regex (.*\.language_model\..*\.(q|k|v|o|gate|up|down)_proj), attaching 2,142 modules and 0 vision modules, verified at launch. gemma-3-4b-it is a multimodal checkpoint, and loading it through a text-only class silently random-initializes the decoder; a sanity assertion generates 40 tokens and refuses to proceed on degenerate output.
• Optimizer: AdamW, lr 1e-4, cosine, warmup 0.05, weight decay 0.01, grad-clip 1.0, bf16.
• Schedule: 3 epochs over 112,000 rows (21,000 steps), effective batch 16 (per-device 8, accumulation 2), max sequence length 1,024, gradient checkpointing on.
• Hardware: one Spot A100-40GB, checkpoints synced to object storage on every save.
• Epoch count was measured, not assumed. Checkpoints were kept at the end of every epoch and all three were evaluated: held-out 87.2% / 90.0% / 93.5% and hard-pattern 89.0% / 95.0% / 96.0% for epochs 1 / 2 / 3. Both rise monotonically, so epoch 3 ships.

Evaluation protocol

• Base and tuned generated in one process from the same weights under identical greedy decoding, 2,048-token budget so neither side is truncation-penalised.
• Generated code executed in a subprocess with a hard timeout; recovered y-data compared to the reference series with a two-decimal tolerance.
• Held-out rows come from a disjoint seed space, with 0 shared ids and 0 shared prompts asserted before the files were written.
• Non-termination reported separately, never folded into the accuracy number.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

# AutoModelForCausalLM resolves gemma-3-4b-it to its multimodal wrapper, which is correct.
# Do NOT load it through a text-only class: the decoder is then randomly initialized.
tok = AutoTokenizer.from_pretrained("google/gemma-3-4b-it")
base = AutoModelForCausalLM.from_pretrained(
    "google/gemma-3-4b-it", torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, "Jainamshahhh/chartsage-4b").eval()

table = {"columns": ["Quarter", "Revenue"],
         "data": [["Q1", "120"], ["Q2", "150"], ["Q3", "90"], ["Q4", "170"]]}
prompt = ("Given this table (JSON):\n" + str(table).replace("'", '"')
          + "\n\nTask: Draw a bar chart of Revenue by Quarter."
          + "\nWrite Python matplotlib code that draws the requested chart from the table data. "
            "Return only a single python code block.")

ids = tok.apply_chat_template([{"role": "user", "content": prompt}],
                              add_generation_prompt=True, return_tensors="pt").to(model.device)
print(tok.decode(model.generate(ids, max_new_tokens=512, do_sample=False)[0][ids.shape[1]:],
                 skip_special_tokens=True))

Try it, and everything that backs it

Live side by side demo: https://huggingface.co/spaces/Jainamshahhh/chartsage-demo Enter your own input and watch the base model and ChartSage-4B answer it under identical greedy decoding. The GPU backend scales to zero, so a cold first request takes about a minute.

Released on both platforms, with the scorer, every eval slice, the per row verdicts and significance.py alongside, so every number on this page can be recomputed rather than trusted.

Reproducibility and license

• Dataset: Jainamshahhh/chartforge. The dataset verifier, the scoring harness and the raw generation files ship alongside, so every number on this card can be recomputed without a GPU:

  python part2/scripts/chartforge/verify_dataset.py --sample 3000
  python part2/scripts/chartforge/score_chart.py gen_chart_v3_ep3__chart_held.jsonl
  

• License: Apache-2.0 for the adapter. The base model is governed by the Gemma terms of use.

Author: Jainam Shah. Built with Adaptive Data (Adaption Labs) and the open Gemma model.