opengraph-image-gemma4-e4b-v1

Model Card for opengraph-image-gemma4-e4b-v1

<!-- Provide a quick summary of what the model is/does. -->

A fine-tuned Gemma 4 E4B that turns images into a valid, schema-faithful knowledge graph (graph.json) — objects, attributes, scene context, text spans, and the relationships between them — ready for AI agents to query and reason over. Built by OpenGraph AI as the small, cheap, on-device alternative to calling a frontier vision API for every extraction.

Model Details

Model Description

The model does one thing extremely well: image → knowledge graph, on-schema, every time. It was trained on verified image→graph gold pairs so that its output always conforms to OpenGraph's graph.json contract — stable snake_case node IDs with type prefixes (entity_, concept_, event_, attr_), typed edges, and cross-image-mergeable entities. Compared to prompting a general frontier model, it is dramatically cheaper per extraction, runs on a single consumer GPU (or laptop, quantized), and produces structurally consistent output that downstream graph tooling can rely on.

• Developed by: OpenGraph AI
• Model type: Multimodal (image-text-to-image), decoder-only transformer; QLoRA fine-tune of Gemma 4 E4B
• Language(s): English
• License: Apache 2.0 (inherited from Gemma 4; use is additionally subject to Google's Gemma terms of use)
• Finetuned from model: google/gemma-4-E4B

Model Sources

• Repository: https://github.com/OpenGraphAI/opengraph-ai
• Demo: [placeholder] <!-- link the graph.html shareable demo when live -->

Uses

Direct Use

Feed the model images plus the OpenGraph extraction system prompt; it returns a complete graph.json — nodes for detected objects (fine-grained labels, normalized bounding boxes), one scene node, attribute nodes, transcribed text spans, and the edges wiring them together. Useful anywhere images need to become structured, queryable knowledge: visual search indexes, dataset annotation, scene understanding, and document/diagram parsing.

Downstream Use

The model's intended home is inside the opengraph-image MCP server: register it with Claude Desktop, Cursor, or any MCP-compatible agent, and the agent gains persistent, queryable visual memory — including multi-hop questions across many images ("which components appear in both photos, and what changed between them?"). It also serves as a local extraction backend for robotics scene memory, where per-frame frontier API calls are too slow and expensive.

Out-of-Scope Use

• Safety-critical decisions without human review (e.g., equipment maintenance, medical, or navigation decisions made solely from the extracted graph).
• General chat, reasoning, or text generation — the fine-tune specializes the model for extraction; general capabilities may be degraded relative to the base model.
• Images far outside the training distribution (see Limitations).
• Any use prohibited by Google's Gemma prohibited use policy.

Bias, Risks, and Limitations

• Extraction errors are silent. The model can hallucinate objects, miss objects, or assign wrong relationships while still producing perfectly valid JSON — structural validity is not factual accuracy.
• Distribution sensitivity. Accuracy degrades on image types unlike the training data (domains, camera angles, lighting, languages in text spans). [More Information Needed — characterize after evaluation]
• Inherited bias. The model inherits biases from Gemma 4's pretraining data, from the public image datasets used for fine-tuning, and from the frontier models used to generate a portion of the training labels.
• Schema lock-in. Output follows OpenGraph's graph.json schema; it is not a general-purpose captioner and will not follow arbitrary output formats reliably.

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. Validate every output with the schema validators shipped in the OpenGraph repo, keep a human in the loop for consequential decisions, and spot-check extractions when applying the model to a new image domain.

How to Get Started with the Model

Use the code below to get started with the model.

# pip install -U "transformers>=5.10.1" torch torchvision accelerate

from transformers import AutoProcessor, AutoModelForImageTextToText
from PIL import Image

MODEL_ID = "OpenGraphAI/opengraph-image-gemma4-e4b-v1"

processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForImageTextToText.from_pretrained(
    MODEL_ID, dtype="auto", device_map="auto"
)

SYSTEM_PROMPT = """[More Information Needed — paste the OpenGraph extraction system prompt]"""

image = Image.open("your_image.jpg").convert("RGB")
messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": "Extract the knowledge graph from this image."},
    ]},
]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=2048)
graph_json = processor.decode(
    outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True
)
print(graph_json)  # -> valid graph.json

Or skip the code entirely and use it through the MCP server:

[More Information Needed — one-line MCP install command]

Training Details

Training Data

Trained on [placeholder] verified image→graph.json gold pairs (OpenGraphAI/opengraph-image-gold-v1), assembled from two sources: (1) public scene-graph datasets (e.g., Visual Genome) converted programmatically into the graph.json schema, and (2) unannotated images labeled by two independent frontier vision models, auto-accepted where both models agreed and human-reviewed otherwise. Every pair passed the OpenGraph Pydantic schema validators before inclusion.

Training Procedure

Supervised fine-tuning (SFT) with QLoRA: the base model frozen in 4-bit NF4 quantization, with LoRA adapters (rank 16, all linear layers, plus lm_head/embed_tokens) trained via Hugging Face TRL's SFTTrainer, following Google's official Gemma 4 vision QLoRA guide.

Preprocessing

Each example is formatted as a three-turn conversation (system = schema instruction, user = image + extraction request, assistant = gold graph.json) and templated with the official Gemma 4 chat template. Images are processed at their native aspect ratio; image tokens are masked out of the training loss.

Training Hyperparameters

• Training regime: bf16 mixed precision (4-bit NF4 quantized base, bf16 compute)
• LoRA: r=16, alpha=16, dropout=0.05, target_modules=all-linear
• Epochs: 3
• Learning rate: 2e-4 (constant schedule)
• Per-device batch size: 1
• Max grad norm: 0.3

Speeds, Sizes, Times

[More Information Needed — fill after training: total training time, adapter size, merged checkpoint size]

Evaluation

Testing Data, Factors & Metrics

Testing Data

A held-out test split (5%) of OpenGraphAI/opengraph-image-gold-v1, never seen during training.

Factors

Results are disaggregated by image source/domain (converted scene-graph data vs. frontier-labeled robot/inspection frames). [More Information Needed — add further factors after evaluation]

Metrics

• Schema-valid extraction rate — % of outputs that parse as JSON and pass the OpenGraph Pydantic validators on the first attempt. Chosen because downstream graph tooling hard-fails on invalid output; this is the reliability number that matters in production.
• Node F1 / Edge F1 — precision and recall of predicted nodes and edges against the gold graph, matched on normalized label + type. Measures whether the content of the graph is right, not just its shape.
• Cost per 1,000 images & p50 latency — the practical case for a small fine-tune over a frontier API.

Results

All numbers produced by the open eval harness and reproducible from the linked script.

Summary

[More Information Needed — 2–3 honest sentences: where the fine-tune wins, where it still trails the frontier baseline]

Technical Specifications

Model Architecture and Objective

Gemma 4 E4B: a decoder-only transformer (~4.5B effective parameters, ~8B with embeddings) with a dedicated vision encoder, hybrid local/global attention, and a 128K-token context window. Fine-tuning objective: supervised next-token prediction on gold graph.json completions, with prompt and image tokens masked from the loss.

Compute Infrastructure

Hardware

[More Information Needed — e.g., 1× NVIDIA L4 24GB (Google Colab Pro)]

Software

Python, PyTorch, Hugging Face transformers>=5.10.1, trl, peft, bitsandbytes, datasets.

More Information

OpenGraph AI is open-source, MCP-first infrastructure for turning heterogeneous data (images, tables, text, audio, video) into semantic knowledge graphs that AI agents can query and reason over. ⭐ Star the repo — and contribute schemas, test images, or extraction edge cases.

Model Card Authors

OpenGraph AI team

Model Card Contact

team@opengraphai.io · GitHub issues