granite-4.2-8b-symbolic-cot

Granite 4.2 8B — Symbolic CoT

A fine-tune of ibm-granite/granite-4.2-8b that rewrites the model's thinking mode into a compact symbolic notation instead of verbose natural-language deliberation. Final answers stay in normal English.

Base Granite 4.2 thinks like this:

Okay, let's try to figure out this problem step by step. First, I need to find
the total time from departure to arrival, then subtract the total stop time...
Wait, better to do minutes: 14:20 is 14*60 + 20 = 860 minutes...

This model thinks like this:

t1=14:20, t2=17:05
diff=2h45min=165min
stops=2*12=24
moving=165-24=141
.: 141 min

Both produce the same final answer. The second uses roughly a quarter of the tokens.

Model Details

• Developed by: rolandwhere
• Model type: Dense decoder-only transformer (GraniteForCausalLM), 8B parameters
• Language: English (base model supports 12; this fine-tune was trained on English only)
• License: Apache 2.0
• Finetuned from: ibm-granite/granite-4.2-8b

Model Sources

• LoRA adapter: rolandwhere/granite-4.2-8b-symbolic-cot-lora
• Training data: rolandwhere/granite-4.2-8b-symbolic-cot-traces

Uses

Direct Use

Latency- or cost-sensitive reasoning where the chain of thought is an internal cost rather than a user-facing product: batch processing, agent loops with many turns, long-context work where thinking tokens compete with the context budget.

The notation is deliberately terse. It uses =, -> (implies), <- (because), .: (therefore), &, |, !, and single-letter variable bindings. It is meant to be machine-cheap, not human-pleasant.

Out-of-Scope Use

• Anywhere the reasoning trace is shown to users or audited. Compressed traces are harder to read and harder to check. If you need interpretable reasoning, use the base model.
• Non-English prompts. Training data was English-only; multilingual behaviour is untested and likely degraded relative to base.
• Tool calling and agentic workflows. The base model was RL-trained for these; this fine-tune did not include agentic traces and their behaviour after training is unmeasured.
• Long-horizon reasoning. Compression was tuned on short problems (≤80 thinking tokens). Problems needing genuinely long deliberation may be hurt by a prior toward brevity.

Bias, Risks, and Limitations

• Compression trades against correctness. A model trained to think briefly will sometimes think too briefly. See Evaluation for the measured trade-off, and read the caveats there before quoting the numbers.
• The base model's alignment may be affected. The LoRA touched all attention and MLP projections across all 40 layers. Safety behaviour was not re-evaluated after training. IBM recommends pairing Granite with Granite Guardian; that recommendation carries over.
• Non-thinking and low-effort modes shifted too. The training targeted enable_thinking=True, but the weights are shared, so all three modes moved. Behaviour in the other two modes is not what IBM shipped.
• Teacher-model artifacts. Traces were distilled from MiniMax M3/M2.7. Where the teacher reasoned badly, the notation faithfully encodes bad reasoning.
• Inherits all limitations of the base model.

How to Get Started with the Model

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

path = "rolandwhere/granite-4.2-8b-symbolic-cot"
tok = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, dtype=torch.float16, device_map="auto")

msgs = [{"role": "user", "content": "How many r's are in 'strawberry'?"}]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True,
                               enable_thinking=True)
ins = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**ins, max_new_tokens=1024, do_sample=True,
                     temperature=1.0, top_p=0.95)
print(tok.decode(out[0][ins.input_ids.shape[-1]:], skip_special_tokens=False))

Use temperature=1.0, top_p=0.95 as IBM recommends for the base model. The compressed trace appears inside <think>...</think>; the answer follows it.

Note on system prompts: training used an empty system turn on every example. Serving with a long system prompt is a distribution shift and may weaken the compression. If you need one, test it — or fine-tune further with your system prompt in place.

Training Details

Training Data

Roughly N synthetic reasoning traces (see the dataset repo for the exact row count), distilled via OpenRouter's free auto-router. Per the OpenRouter activity log, generation landed roughly 50/50 across MiniMax M3 and M2.7. The teacher was used only to produce training traces — it plays no part in the model at inference.

Because the router selects models dynamically, the exact data-generation run is not bit-reproducible. The notation spec and filter criteria below are what actually determine the output.

Two seed sources:

• GSM8K train — grade-school math, ~77% of the set. These carry gold answers, so traces whose final answer disagreed with the reference were discarded automatically.
• HuggingFaceH4/no_robots — open QA, brainstorming, classification, extraction. No gold answers; kept for style coverage so the model doesn't collapse into math-only behaviour.

The teacher was prompted to emit two fields: a compressed symbolic think and a normal-English answer. Only the reasoning is compressed.

Preprocessing

Traces were filtered on four criteria before training:

1. Trace ≤ 80 tokens under the Granite tokenizer
2. Terminal .: conclusion line present
3. No natural-language filler (let me, okay, wait, we need, hmm, …)
4. For math seeds, the gold answer appears in the reply

Prompts were rendered with the base chat template at enable_thinking=True, so training overwrites full-thinking behaviour specifically. Loss was masked to the completion only.

Training Hyperparameters

Training loss fell from 1.63 to ~0.60 over the run.

Speeds, Sizes, Times

Single NVIDIA T4 (16GB), ~15 minutes. Adapter is 396 MB; merged weights are ~18 GB in fp16.

Evaluation

Testing Data, Factors & Metrics

40 held-out questions from GSM8K test. Exact-match on the final numeric answer. Four conditions on identical hardware and quantization, with the base conditions obtained by disabling the adapter on the same loaded model. Sampling at temperature=1.0, top_p=0.95, generation capped at 768 new tokens.

Metrics: mean thinking tokens (the quantity being optimized), wall-clock seconds per example, and accuracy.

Results

* These two numbers are measurement artifacts, not properties of the base model. Read the summary before using this table.

Summary

The compression result is solid: 4× fewer thinking tokens (176 → 44) and 4.7× lower latency (25.7s → 5.5s) versus base thinking mode.

The accuracy column is not trustworthy as measured, and I would rather say so than let the numbers flatter this model:

• Base thinking mode at 0.48 is wrong. The 768-token generation cap truncated the base model's longer chains before they reached an answer, scoring them as failures. IBM's own card recommends 8192 tokens for thinking mode. The true base figure is substantially higher, and this model very likely does not beat it on accuracy.
• Base non-thinking at 23 seconds is impossible for a condition producing no reasoning, indicating a harness bug in that row.
• Base low-effort at 0.93 is the meaningful comparison and it is the strongest condition tested. IBM's built-in low-effort mode achieves most of the compression benefit with better accuracy than this fine-tune. If you want short reasoning from Granite 4.2, try low_effort=True first — it is free and requires no fine-tuned weights.

The honest claim for this model is a 4× reduction in thinking tokens at an accuracy cost that has not yet been properly quantified. A re-run with a 2048+ token budget is needed before the accuracy column means anything. Treat this release as a working demonstration of the compression technique, not as a validated accuracy claim.

Environmental Impact

• Hardware: 1 × NVIDIA T4
• Hours used: ~1 (including dataset generation and evaluation)
• Cloud Provider: Kaggle
• Carbon Emitted: negligible at this scale; estimable via the ML Impact calculator (Lacoste et al., 2019)

Technical Specifications

Model Architecture and Objective

Dense decoder-only transformer. 40 layers, hidden size 4096, GQA with 32 query heads and 8 KV heads, SwiGLU MLP (hidden 12800), RMSNorm, RoPE. Standard causal language modelling objective, loss masked to the assistant completion.

Compute Infrastructure

Kaggle notebook, 1 × T4 (16 GB), fp16 compute with a 4-bit NF4 quantized base. Unsloth for training; PEFT and llama.cpp for merging and GGUF conversion.

Citation

If you use this model, cite the base model:

@misc{granite2026,
  title  = {Granite 4.2 Language Models},
  author = {{Granite Team, IBM}},
  year   = {2026},
  url    = {https://huggingface.co/ibm-granite/granite-4.2-8b}
}

Glossary

Notation used inside <think>:

Model Card Authors

rolandwhere

Contact

Questions, results, or corrections are welcome — open a discussion on this repo, or reach out via LinkedIn.

If you re-run the evaluation with a larger token budget, I would genuinely like to see the numbers.

Acknowledgements

Built on IBM's Granite 4.2 (Apache 2.0) and trained with Unsloth on free Kaggle T4 hardware.