granite4-3b-ec-magento

granite4-3b-ec-magento

A QLoRA adapter for ibm-granite/granite-4.1-3b specialized in e-commerce catalog tasks with Magento 2 conventions: attribute extraction to JSON, product Q&A, search-relevance classification, and product ranking.

Fifth model in a controlled comparison — five base models, identical frozen training data, identical recipe, one eval harness — alongside qwen3.5-4b-ec-magento, ministral3-3b-ec-magento, phi4-3b-ec-magento and gemma4-e2b-ec-magento. It is the strongest of the five on both held-out sets.

What it does

Trained on 56,253 instruction samples: 70% ECInstruct (generic e-commerce) + 30% synthetic Magento-schema data generated from the Magento Luma sample catalog (products fully disjoint between train and eval). Four task shapes:

Attribute extraction — product text (or a raw Magento custom_attributes payload) → JSON:

target attribute: size
product title: Puma Suede green sneakers size 43
→ [{"attribute": "size", "value": "43"}]

Absent attributes are reported as "None" rather than hallucinated.

Product QA — a question answered strictly from given product data. Relevance classification — query + product → graded relevance option (ESCI-style A–D). Relevance ranking — query + lettered product list → ranked letters (B,A,C).

Usage — adapter (unsloth / peft)

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    "gabrielgts/granite4-3b-ec-magento", max_seq_length=2048, load_in_4bit=True)
FastLanguageModel.for_inference(model)

# No system message — the model was trained without one, and Granite's template
# injects nothing when no tools/documents are passed. Greedy decoding recommended.
messages = [{"role": "user", "content":
    "Extract the value of the target attribute from the given product information "
    "and output it as JSON. If the attribute is not present, output None as the value.\n\n"
    "target attribute: size\nproduct title: Puma Suede green sneakers size 43"}]
text = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = tokenizer(text=text, return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
# [{"attribute": "size", "value": "43"}]

Plain peft also works: PeftModel.from_pretrained(base_model, "gabrielgts/granite4-3b-ec-magento").

Usage — GGUF / Ollama

hf download gabrielgts/granite4-3b-ec-magento --include 'gguf/*' --local-dir .
cd gguf && ollama create granite4-3b-ec-magento -f Modelfile
ollama run granite4-3b-ec-magento "target attribute: color ..."

Quantization note: the shipped GGUF uses --token-embedding-type q8_0 --output-tensor-type q8_0. Granite ties its embeddings over a 100,352-token vocabulary, and tied large-vocab embeddings degrade at plain 4-bit — producing correctly-formatted but wrong answers. If you requantize, keep the embeddings at q8_0 and verify answer content, not just JSON validity.

Training recipe

Evaluation

Greedy decoding, identical prompts across all five models; base models evaluated zero-shot with the same harness. All five were trained on the same frozen dataset (sha256 afb7e664…), so the base model is the only variable.

Magento held-out set (2,969 samples, 475 products never seen in training). Bold = best of the five.

ECInstruct held-out set (2,000 samples):

Long-list ranking (500 held-out ESCI queries, 5–8 candidates with graded hard negatives; chance top-1 = 0.167). This set is much harder than the ranking task above, which is 62% two-candidate:

Read that 0.516 against its own baseline, not against the 0.655 above: the easy set is mostly binary A/B choices, so a high score there flatters the model. Ranking is this model's relative weak spot.

Limitations — read before relying on the numbers

• The Magento eval is synthetic-on-synthetic. Eval tasks were generated with the same templates as the training data (products fully disjoint). It validly measures schema adherence — JSON format, Magento attribute vocabularies, the None-when-absent rule — but overstates production quality on real catalogs and real user queries.
• The recipe was transplanted, not tuned per model. r=8 / 2e-4 / 1 epoch was validated on a different base. These results mean "best under this fixed recipe", not an intrinsic ranking of the base models.
• The base model's low structured scores are dominated by format non-adherence; they understate its underlying capability, though prose output is itself a blocker for programmatic use.
• English only; fine-tuned on rigid structured data — expect degraded general chat and multilingual ability versus the base model (drop the adapter to recover it).
• Answer-generation F1 is low by construction (free-form text scored against a single reference); use it comparatively, not absolutely.
• Use greedy decoding (do_sample=False / temperature 0) — that is how it was evaluated.

Provenance

References

• Granite 4.1: ibm-granite/granite-4.1-3b
• ECInstruct / eCeLLM: arXiv:2402.08831
• Amazon ESCI (Shopping Queries): arXiv:2206.06588
• QLoRA: arXiv:2305.14314