chart-parsing-vlm

Nutrient Chart Parsing · commercial

Read a chart image and return its data and structure as one JSON object. The Nutrient chart-parsing model takes a rendered chart (bar, line, pie, scatter, area, box, bubble, heatmap, histogram, treemap, funnel, radar, violin, stem, step, candlestick, combo) and returns the title, chart type, data table, value representation, encoding, axes, series, legend, and data-label state in a single pass. Values are what a reader can recover from the pixels: printed labels verbatim, shares for pie and donut charts, axis-scaled estimates otherwise.

The runnable weights are commercial and are not downloadable from this repository. This public page is the product specification and scorecard; see License and access to evaluate or deploy it.

• 🧪 Try it: nutrientdocs/chart-parsing-demo
• 🏆 Leaderboard: nutrientdocs/chart-parsing-leaderboard
• 📊 Benchmark: nutrientdocs/chart-parsing-benchmark

Headline result

Systems are ranked by mean per-chart cell F1 at 3% tolerance on the frozen 2,500-chart benchmark: every chart's data cells are matched by row identity and column, a value counts when it is within 3% of the chart's value range (3 percentage points on share charts, exact when the chart prints data labels), and the per-chart F1 is averaged so every chart counts once. Strict F1 demands values within 0.1%. Every entry sees the same charts and the same prompt.

The Nutrient model reaches 0.888, ahead of the strongest cloud model here, Gemini 3.8 Flash (0.710), and it recovers the right table shape on 96.3% of charts against 80% for the best cloud model. Its lead is in the data cells, not in the labels: cloud models read titles and chart types about as well. Seconds per chart are measured on one NVIDIA A100 80GB with vLLM at 64-way concurrency and compare only within that setup; provider APIs give no comparable latency. Full per-field and per-type breakdowns are on the leaderboard.

Result by chart type

Mean per-chart cell F1 for the Nutrient model at the three tolerance levels, with the number of benchmark charts of each type.

Per-field accuracy on the same set: chart type 0.983, title 0.989, x and y axis labels 0.987 and 0.986, axis scales 0.975 and 0.985, legend visibility 0.990, data-label visibility 0.989, column types 0.989, column roles 0.988, axis bounds 0.835. Set F1 over encoding pairs 0.974 and over series pairs 0.993.

Output

One JSON object per chart. data.rows is the table; every other key describes how the chart shows it.

{
  "title": "Programming Language Popularity",
  "chartType": "pie",
  "data": {
    "columns": [
      {"name": "Language", "type": "categorical", "role": "category", "unit": null},
      {"name": "Popularity", "type": "numeric", "role": "value", "unit": null}
    ],
    "rows": [["Python", 21.6], ["Java", 19.4], ["C++", 17.7]]
  },
  "valueRepresentation": {"mode": "percentage_of_total", "format": "0.0%"},
  "encoding": {"category": "Language", "value": "Popularity", "color": "Language"},
  "axes": null,
  "series": [{"name": "Popularity", "field": "Popularity"}],
  "legend": {"visible": false, "position": null, "entries": null},
  "dataLabels": {"visible": true, "format": "percentage"}
}

• chartType is one of bar, line, area, pie, scatter, bubble, histogram, box, violin, heatmap, treemap, funnel, radar, candlestick, step, stem, combo.
• Column type is categorical, numeric, or temporal; column role is one of category, value, series, x, y, z, size, color.
• valueRepresentation.mode is percentage_of_total for pie, donut, treemap, and 100%-stacked charts, otherwise absolute.
• axes holds {label, unit, scale, min, max} per axis with the visible limits, or null for charts without axes.
• Any field the image does not support is null.

The data block converts to CSV, HTML, or a spreadsheet with plain code; the model is not asked to emit those formats.

Intended use and limits

• Use it for: turning rendered chart images (exports from plotting libraries, report figures, dashboard screenshots) into a data table and a description of how the chart shows it, for indexing, analysis, re-plotting, or conversion to spreadsheets.
• Values are visual estimates. When a chart prints no data labels the model reads values off the axis scale; expect errors within a few percent of the value range, and check the strict versus 3% columns above for how much that costs per chart type. Pie and donut charts return shares of the total, never the underlying absolute values.
• Weaker chart types: bubble (size legends are hard to invert), radar, and candlestick charts, and axis bounds in general (0.747 field accuracy). Pie charts read printed shares well but estimating unlabeled wedges is imprecise.
• Validate downstream. The output is well-formed JSON in practice, but parse it defensively and check that data.rows has the width data.columns declares before feeding it to automated pipelines.

License and access

The model weights are offered under a commercial Nutrient license and are not published here. The benchmark charts derive from ChartGen-200K, CC BY 4.0. The benchmark is public and reproducible through nutrientdocs/chart-parsing-benchmark.

📩 Get access

Nutrient Chart Parsing is commercial. To evaluate or deploy it on-prem, with your documents staying in your infrastructure, contact Nutrient: nutrient.io/contact-sales.

About the author

<a href="https://nutrient.io/"> <img src="https://avatars2.githubusercontent.com/u/1527679?v=3&s=200" height="80" /> </a>

This project is maintained and funded by Nutrient - The deterministic document infrastructure enterprises run their highest-stakes workflows on: replayable output, clear exceptions, and full audit trails on the messy, regulated documents where AI alone breaks.