amad-vlm6-technical-report

amad-vlm6: What Merging Two Arabic OCR Models Actually Fixes — and Breaks

A technical report on TIES-merging two independently fine-tuned Arabic OCR specialists, with an error-type decomposition, behavior statistics, and bootstrap intervals that separate what merging repaired from what it transferred and what it damaged.

Author: Murtadha · amad-iq Model: amad-iq/amad-vlm6 · GGUF

TL;DR — amad-vlm6 is a TIES merge of amad-vlm5 and DIMI-Arabic-OCR-V2 on their shared Qwen2.5-VL-7B-Instruct base. Against amad-vlm5 on all 13 KITAB-Bench ocr-eval datasets it improves macro CER (0.254 → 0.235) and CHrF (81.05 → 81.68) and worsens macro WER (0.361 → 0.463); on the 10 datasets free of amad-vlm5's training overlap it cuts CER by 26%. On the 7 datasets where all three models were scored, it beats both parents on every macro metric. Decomposition shows the mechanism is behavioral, not perceptual: the merge suppresses one parent's runaway reasoning (thinking rate 45–98% → 5–18%, truncated outputs 10% → 1%) while reading accuracy itself regresses on 10 of 13 datasets, improving only on adab and evarest. The merge does not reliably cancel the other parent's hallucination: on isippt and muharaf, insertion rates jump 0.006 → 0.471 and 0.012 → 0.423. Four per-dataset wins and three losses are statistically significant; five are not. The natural control — amad-vlm5 with inference-side loop and think-budget fixes — is running and will be added; until it lands, "the merge is the best way to get this fix" is unproven.

Key numbers at a glance

¹ The 7 are the datasets on which the merge beat amad-vlm5 — a selection on the outcome, made to test whether DIMI alone explained those wins. DIMI-V2's remaining 6 datasets are being scored and the 13-set columns will be filled.

1. Why merge these two?

The parents fail in complementary, behavioral ways.

amad-vlm5 is a thinking OCR model: on dense page-level documents it reasons inside <think>…</think> before transcribing. Measured on the benchmark, it opens a reasoning block on 45–100% of page-level inputs (§5.1); on hindawi and historicalbooks 10% of those blocks never close, consuming the whole generation budget and producing no transcription. Greedy decoding also loops: one khattparagraph page yielded 16,000 characters of a repeated phrase against a 361-character ground truth.

DIMI-Arabic-OCR-V2 never reasons and reads printed text competently, but over-generates: its outputs run 1.2–1.65× the ground-truth length on every dataset (§5.1), continuing past the page with fluent invented Arabic. On one 84-character line it produced 4,888 characters.

Both were fine-tuned from the same base, which makes their differences exactly expressible — and mergeable — as task vectors.

2. Method

Each fine-tune is its task vector, the delta it added to the shared base:

τ_amad = W_amad − W_base        τ_dimi = W_dimi − W_base

TIES-Merging (Yadav et al., 2023):

1. Trim — keep the top-density fraction of each τ by magnitude (density 0.5), zeroing the rest.
2. Elect — per parameter, the sign of the summed trimmed vectors wins.
3. Disjoint mean — average the vectors agreeing with the elected sign (sum ÷ number of agreeing vectors), then W = W_base + λ · τ_merged, λ = 1.

Implementation note: the merge was run with an in-house script that computes exactly the steps above (disjoint mean, not sum); the mergekit recipe in §8 reproduces this with normalize: true and is illustrative — mergekit's support for Qwen2.5-VL's vision tower should be verified before relying on it.

Empirical properties of this pair, each worth checking before any merge:

• Both task vectors are low-rank. amad-vlm5's has rank ≤ 32 and DIMI-V2's rank ≤ 16 (both LoRA fine-tunes). Magnitude-trimming a rank-16 delta at density 0.5 discards half of a very structured update; results may be more density-sensitive than for full fine-tunes.
• The language-model task vectors are near-orthogonal. On language-model modules (vision excluded), cosine similarity ≈ 0.001 with ~50% per-parameter sign disagreement — the statistics of independent random directions. Election therefore interleaves the two updates rather than amplifying a shared signal.
• DIMI-V2's vision delta is exactly zero. All 160 vision-tower tensors its adapter targets carry zero weights, and it does not target the vision–language merger at all. Every vision parameter in amad-vlm6 is amad-vlm5's, unchanged.
• DIMI-V2 is a continuation of DIMI-V1. Cosine 0.97–0.99 between τ_V1 and τ_V2 is exactly what continued training produces (τ_V2 = τ_V1 + δ) — the tensors confirm the model card's lineage. Stacking both would apply nearly the same update twice; the merge uses V2 alone.

3. Evaluation protocol

• KITAB-Bench ocr-eval: 13 datasets, 3,760 images, unmodified repository metrics and Arabic normalization.
• Prompt: the benchmark's own — Extract the text in the image. Give me the final text, nothing else. — for all three models. DIMI-V2's model card prescribes an Arabic instruction (استخرج النص العربي والأرقام الموجودة في هذه الصورة بدقة عالية); it was not used here, so DIMI's standalone numbers are provisional. A rerun with its own prompt is queued.
• Engine: Unsloth FastVisionModel, 4-bit NF4 weights, HuggingFace generate, greedy, no repetition penalty, batch 2. Image resolution: the processor defaults of the Qwen2.5-VL checkpoint (no min/max_pixels override).
• Budget: 4,096 new tokens for amad-vlm5 and amad-vlm6 (context 16,384). DIMI-V2 was capped at 1,024 after its continuations made 4,096 impractical; §5.3 measures where that cap bit.
• Scoring: reasoning inside <think>…</think> is stripped before any metric; an unclosed block scores as an empty prediction. Ground truth is never modified.
• Uncertainty: 95% bootstrap intervals over images (1,000 resamples of the corpus-level CER). Everything here is a single configuration (density 0.5, λ 1.0); TIES and greedy decoding are deterministic, so there is no seed variance to report — only configuration variance, which was not explored.

4. Results

4.1 Per-dataset, amad-vlm6 vs. amad-vlm5 (all 13)

CER with 95% bootstrap CI; ✓ = amad-vlm6 better, ✗ = worse, ~ = intervals overlap.

amad-vlm6 wins CER on 7 of 13 — 4 significant (adab, evarest, khattparagraph, hindawi), 3 not. It loses 6 — 3 significant (isippt, muharaf, onlinekhatt), 3 not. Note historyar: intervals of [0.09, 4.3] and [0.10, 3.3] mean its CER is set by one or two repetition-loop rows in each model; no conclusion should be drawn from it.

WER worsens on 7 of 13 — but on the 7 datasets where CER improved, WER improves on 6 (all but historyar). The macro WER increase (0.361 → 0.463) comes from three datasets — historyar (1.06 → 2.04), muharaf (0.14 → 0.91), isippt (0.21 → 0.63) — i.e. the same catastrophic rows that drive their CER, not from a general segmentation regression.

Macro over 13: CER 0.254 → 0.235, CHrF 81.05 → 81.68, WER 0.361 → 0.463. Excluding the three datasets overlapping amad-vlm5's training data (552 images, "leak"): CER 0.321 → 0.236, CHrF 77.35 → 82.46.

4.2 Against published KITAB-Bench baselines (under our protocol)

Baselines are the published numbers (run at the benchmark's default 500-token budget); our three rows use the protocol in §3 with a 4,096-token budget. Sorted by CHrF.

Under our protocol amad-vlm6 ranks #1 by CHrF, #3 by CER, #4 by WER among the published systems. The budget difference means this is not a like-for-like leaderboard entry; under the benchmark's default 500-token budget amad-vlm5 scores CHrF 65.22, and amad-vlm6 has not been run at 500 (it still reasons on 5–18% of page-level inputs, §5.1, so the budget matters for it too).

5. Why it works — and where it doesn't

5.1 Behavior statistics

Per model and dataset: share of outputs that open a <think> block, share that never close it, share that hit the generation cap, share scored as empty, and output length relative to ground truth (final answer only).

Line-level datasets (patsocr, adab, evarest, synthesizear, khatt, onlinekhatt) show no thinking and ~1.0 length ratios for both amad models and are omitted except where noted.

Reading the table: the merge cut the thinking rate 3–9× on page-level sets, eliminated unclosed blocks and empty outputs on hindawi and historicalbooks (10% → 0–1%), and brought output length to ≈ ground truth where amad-vlm5 under-produced (hindawi 0.82 → 0.97). That is the "format fix", now measured. But on isippt and muharaf the merged model over-generates by 36–44% — DIMI's signature — where amad-vlm5 did not. The merge still thinks on 5–18% of page-level inputs (100% on historicalbooks), so it cannot be evaluated at the benchmark's 500-token default without the same truncation problem as its parent.

5.2 Error-type decomposition

Character edit operations against the normalized ground truth:

SUB = substitutions / reference chars   → reading errors
INS = insertions    / reference chars   → over-generation (loops, hallucination)
DEL = deletions     / reference chars   → missing output (truncation, under-transcription)
CER = SUB + INS + DEL

All 13 datasets, amad-vlm5 → amad-vlm6:

Three-way on the 7 outcome-selected datasets (amad-vlm5 / amad-vlm6 / DIMI-V2):

² DIMI's hindawi score is unreliable — see §5.3.

Macro over these 7: CER 0.441 / 0.249 / 0.389 · CHrF 70.92 / 80.35 / 58.22.

What the decomposition establishes:

(a) The gain is behavioral; reading did not improve — it mostly regressed. Substitution rate rises on 10 of 13 datasets (relative +16% historyar, +33% khattparagraph, +41% historicalbooks and hindawi, +50% isippt, 4× arabicocr, 5× muharaf). It improves only on adab (−31%) and evarest (−51%) and is unchanged on patsocr. Where amad-vlm6 wins, it wins because insertions and deletions collapsed — khattparagraph INS 0.379 → 0.022 and DEL 0.319 → 0.050 — not because it reads better. A merge partner with no reasoning acted as a behavioral regularizer on the reasoning parent.

(b) DIMI-V2 is not simply the weakest model. By CER it beats amad-vlm5 on 4 of these 7 datasets (evarest, historyar, khattparagraph, hindawi) and on the 7-set macro (0.389 vs 0.441). It is the weakest by CHrF (58.2) because its insertions are large and fluent — CHrF's character n-gram recall punishes invented text harder than CER's per-character count does. On the line-level sets (patsocr, adab) its over-generation makes it clearly worst.

(c) Hallucination did not cancel — it moved. The merge removed DIMI-style over-generation on the page-level sets, but on isippt (INS 0.006 → 0.471, 500 images) and muharaf (0.012 → 0.423) it introduced it where amad-vlm5 had none. On historyar the amad-vlm5 loop behavior survived (INS 1.042). Any claim that "the merge cancels the parents' failure modes" is therefore true on 5 datasets and false on 3.

(d) Real transfer on two datasets — with an open question. On adab and evarest (1,000 images, short scene-text lines) the merged model's substitution rate beats both parents. DIMI-V2's training data is not published; whether it saw adab- or evarest-like material cannot be audited, so "transfer" on these two may partly be DIMI-side memorization crossing the merge. The leak audit in this report covers amad-vlm5's training data only.

5.3 Was DIMI-V2's 1,024-token cap fair?

On the line-level sets the cap never binds (0–0.8% hit it, ground truths are ≤ 43 tokens), so DIMI's deletions there (adab 0.109) are genuine under-transcription and the cap could only have removed insertions. On hindawi, ground truths reach 927 tokens and 26% of DIMI's outputs hit the cap: real transcription was almost certainly truncated, which raises DEL and — because CHrF is recall-weighted — lowers CHrF. DIMI's hindawi row should be read as a lower bound on its quality; historicalbooks (10%) is borderline. The claim that the cap "can only flatter DIMI" holds for five of the seven datasets, not all.

5.4 The missing control

If the merge's gain is mostly behavioral, the natural baseline is not "no merge" but amad-vlm5 with inference-side fixes: a repetition penalty against loops and a think budget that forces </think> after N tokens so an answer is always emitted. If that recovers most of the gain without the isippt/muharaf regressions, merging is the wrong tool for this fix. The same applies to DIMI-V2 with a stopping criterion. This control is running under the protocol of §3 (repetition penalty 1.05, think budget 1,024) and this section will be updated with its full 13-dataset comparison. Until then the report shows that merging fixed the behavior, not that merging was necessary.

6. Limitations

• Reading regresses almost everywhere. SUB is worse on 10 of 13 datasets (§5.2a); behavioral gains dominate the totals. Treat amad-vlm6 as a model with better output discipline, not better perception.
• Two datasets collapse from imported hallucination. isippt (0.054 → 0.541) and muharaf (0.046 → 0.600) are the largest regressions and both are insertion-driven; muharaf also loses memorized reading (SUB 5×). Do not use amad-vlm6 on presentation-slide or muharaf-style handwritten inputs without checking.
• Several results are not statistically distinguishable. historyar, historicalbooks (n = 10), arabicocr (n = 50), synthesizear and khatt have overlapping bootstrap intervals; historyar's is set by single loop rows.
• DIMI-V2's standalone numbers are provisional — English benchmark prompt rather than its card's Arabic one, a 1,024-token cap that bit on hindawi, and 6 datasets not yet scored.
• Single configuration. Density 0.5 and λ 1.0 were not swept; with near-orthogonal low-rank parents the density choice may matter more than usual.
• One benchmark is a narrow window. KITAB-Bench measures transcription accuracy on 13 document styles. It says nothing about photographed pages in poor light, skew, mixed Arabic–Latin text, tables and layout fidelity, multi-turn use, instruction following, latency, or robustness to inputs far from its distribution. The merge may carry real strengths and regressions in all of those dimensions that this report cannot see.
• Protocol caveats inherited from the parents: 4,096-token budget vs. the benchmark's 500, and amad-vlm5's 552-image training overlap on three datasets (both documented on the amad-vlm5 model card).

7. Lessons for practitioners

1. Decompose CER before crediting a merge. SUB / INS / DEL is fifteen lines of code; here it turned "synergy" into "behavioral fix with reading regression on 10 of 13 sets".
2. Then check behavior directly. Thinking rate, unclosed-block rate, cap-hit rate and length ratio (§5.1) are what a "format fix" claim actually rests on.
3. Expect failure modes to migrate, not cancel. Hallucination left the page-level sets and appeared on isippt and muharaf. Test the datasets where the other parent was weak.
4. Check lineage with tensors and read them correctly. cos ≈ 0.98 between V1 and V2 deltas confirms continued training; stacking both would double-apply the update.
5. Zero-delta modules are free wins — DIMI never trained vision, so the merge could not damage amad-vlm5's vision tower. Inventory per-module norms to know what is contested.
6. Memorization does not survive merging — a usable leak detector: clean skills survive averaging; memorized test images do not.
7. Run the inference-side control before publishing a merge. If a repetition penalty and a think budget buy the same behavior, the merge is costing you reading accuracy for nothing (§5.4).

8. Reproduction

Illustrative mergekit recipe — the published weights were produced by an in-house TIES implementation (trim → elect → disjoint mean) as described in §2; verify mergekit's handling of the Qwen2.5-VL vision tower before treating this as an exact reproduction:

merge_method: ties
base_model: Qwen/Qwen2.5-VL-7B-Instruct
models:
  - model: amad-iq/amad-vlm5
    parameters: {weight: 1.0, density: 0.5}
  - model: AhmedZaky1/DIMI-Arabic-OCR-V2   # V2 only — V2 continues V1
    parameters: {weight: 1.0, density: 0.5}
parameters: {lambda: 1.0, normalize: true}   # true = disjoint mean, as used
dtype: bfloat16

Evaluation: KITAB-Bench ocr-eval, unmodified metrics.py; protocol in §3; edit-operation decomposition via Levenshtein.editops on the same normalized strings; bootstrap over images.

9. Availability

• amad-vlm6: amad-iq/amad-vlm6 (bf16 safetensors) · amad-iq/amad-vlm6-GGUF (Q8_0 + F16 vision projector).
• Parents: amad-iq/amad-vlm5 (+ GGUF), AhmedZaky1/DIMI-Arabic-OCR-V2.
• Base: Qwen/Qwen2.5-VL-7B-Instruct. All Apache-2.0.

Citations

@inproceedings{yadav2023ties,
  title  = {TIES-Merging: Resolving Interference When Merging Models},
  author = {Yadav, Prateek and Tam, Derek and Choshen, Leshem and Raffel, Colin and Bansal, Mohit},
  booktitle = {NeurIPS}, year = {2023}
}
@inproceedings{heakl2025kitab,
  title  = {KITAB-Bench: A Comprehensive Multi-Domain Benchmark for Arabic OCR and Document Understanding},
  author = {Heakl, Ahmed and Sohail, Abdullah and Ranjan, Mukul and others},
  booktitle = {ACL}, year = {2025}
}

September 2026 · Murtadha · amad-iq · revised after external review