Llama-xReadable-11B-zhTW-OCR-v3

Llama-xReadable-11B-zhTW-OCR-v3

Built with Llama

云碩科技 · xCloudinfo　·　系列：Meta Llama 3.2 Vision · 云碩繁體中文OCR

以 meta-llama/Llama-3.2-11B-Vision-Instruct 為基底，由云碩科技在自有AI算力資源池上微調的繁體中文（台灣）OCR模型。專注台灣場景的文字辨識——公文、票據、文件與各類筆記的中英混寫內容，可地端部署、資料不外流。xReadable：讓文件被讀懂。

v3相對v2的變化：合成語料的字型多樣性由5個設計家族擴充到 16個家族／36個字型檔（另保留8家族／17字型檔全程不進訓練，專供泛化量測）。印刷辨識在四項量測上全面優於v2。

能力與量測

所有數字皆為逐字元錯誤率（CER，越低越好）。評測集的字型與訓練字型完全不重疊——保留字型從建池起就切開，從未進入訓練。

印刷／掃描文件

保留字型集為逐張配對比較（同200張圖）：v3逐張勝51 / 負21 / 平手128，平均改善 −0.52pp（95% CI [−1.02, −0.03]），完全正確張數124 vs 115。

外部對照：nvidia/nemotron-ocr-v2 為偵測＋辨識管線（非語言模型），在相同影像、相同CER計算下量測。

手寫

手寫的64~88% 是影像難度地板，非模型天花板。 我們以架構完全不同的開源OCR（NVIDIA nemotron-ocr-v2，偵測＋辨識管線）在同一組真實手寫影像上交叉驗證，逐張CER與本模型幾乎一致（某淡鉛筆掃描頁兩者皆約88%、某清晰頁皆約45%）。兩種原理迥異的系統得到相同的逐張結果，代表該數字主要由影像本身的可辨識度決定，而非單一模型的弱點。

字型分層方法

訓練字型（16家族／36檔）：Noto Sans／Serif CJK、文鼎UMing／UKai、Chiron Hei／Sung／GoRound、Zev Hei、GenKiGothic、GenKiMin、GenWanMin、cwTeX明／楷／圓、jf-openhuninn、辰宇落雁體。

保留字型（8家族／17檔，永不進訓練）：GenRyuMin、GenSekiGothic、GenSenRounded、cwTeX仿宋、EvilSung、LXGW文楷、芫荽Iansui、Cubic 11。

同一套設計的所有字重與正／斜體一律歸同一家族（例：ChironSungHKVF與ChironSungHKItVF；GenRyuMin與其注音版Bpmf、台語版Taigi衍生字型），避免同一設計同時出現在訓練與評測兩側而高估泛化能力。

用途與限制

• 適用：台灣繁中印刷／掃描文件的地端OCR、文件數位化、票據與筆記辨識前處理；長篇手寫筆記轉錄。
• 輸出正規化：本模型輸出統一為繁體中文（訓練標註採繁體正規化）；輸入含簡寫字時亦會輸出對應繁體。
• 限制：
  • 手寫辨識仍在迭代；潦草、淡筆、低對比字跡可能出錯，重要用途請務必人工複核。
  • 純英文段落已納入訓練但樣本量仍有限。
  • 表格／複雜版面建議先切行／切格再辨識。
  • 輸出僅供辨識參考，不宜作為法律、醫療、財務等關鍵決策的唯一依據。

量測方法提醒：自行評測時，請先確認測試影像的字型確實含有待測字元。部分字型檔（如macOS Songti.ttc 的第0個字面為簡體字面）缺繁體字時，PIL等渲染器會靜默畫成空白而非豆腐字，導致模型被錯誤扣分。建議渲染後逐字驗證非空白再用於評測。

格式與相容性

本repo提供 bf16 safetensors（transformers／vLLM原生格式），未提供GGUF：Llama 3.2 Vision的 mllama 架構llama.cpp官方尚未支援轉換／載入，故llama.cpp／LM Studio／Ollama目前無法運行本模型；transformers 與 vLLM 可原生載入。上游支援後將補上官方相容GGUF。

硬體需求

bf16權重約21GB，推論建議單張 ≥ 24GB VRAM的GPU（RTX 3090/4090、A10、L4、A100等），或以 device_map="auto" 多卡分攤；記憶體不足可用 bitsandbytes 做int8/int4量化。

用法（transformers）

from transformers import MllamaForConditionalGeneration, AutoProcessor
from PIL import Image
import torch

model_id = "xCloudinfo/Llama-xReadable-11B-zhTW-OCR-v3"
model = MllamaForConditionalGeneration.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto"
).eval()
processor = AutoProcessor.from_pretrained(model_id)

image = Image.open("your_document.jpg").convert("RGB")
prompt = "請辨識圖片中的所有文字，逐字完整輸出，只輸出文字本身。"
messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": prompt}]}]
text = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=768, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

用法（vLLM）

vllm serve xCloudinfo/Llama-xReadable-11B-zhTW-OCR-v3 --dtype bfloat16 --max-model-len 4096

授權與來源聲明（Llama 3.2 Community License）

本模型為Meta Llama 3.2的衍生模型，散布須遵守 Llama 3.2 Community License Agreement：

• Built with Llama。 本模型以Meta Llama 3.2建構。
• 基底模型：meta-llama/Llama-3.2-11B-Vision-Instruct（Meta Platforms, Inc.）。
• 模型命名：依授權要求，衍生模型名稱以「Llama」開頭。
• 授權文本：本散布隨附Llama 3.2 Community License Agreement副本（見本repo LICENSE）；須提供該授權副本並保留本聲明。
• 可接受使用政策：使用須遵守Meta Acceptable Use Policy。
• 歐盟（EU）限制：Llama 3.2授權規定，設立地或主要營業地在歐盟的個人或公司，不被授予使用Llama 3.2多模態模型所含之多模態能力的授權。
• 大規模服務門檻：月活躍使用者超過7億須另向Meta申請授權。
• 微調與散布由 云碩科技xCloudinfo 進行；不得用於非法用途；須遵守中華民國相關法律。

字型：合成訓練語料所用字型均為開放授權（SIL OFL 1.1、jf開放授權、GPL+字型例外等），僅用於產生訓練影像，本repo不散布字型檔本身。

關於云碩科技

云碩科技（xCloudinfo Corp. Limited）專注於地端私有化人工智慧解決方案。xVITA是云碩自研模型的品牌名稱。

其他模型請見 https://hf.co/xCloudinfo。

---

Built with Llama. 由 云碩科技xCloudinfo於自有AI算力資源池微調、散布；基底模型之權利屬Meta Platforms, Inc.。

---

Llama-xReadable-11B-zhTW-OCR-v3 — English

Built with Llama

xCloudinfo (云碩科技)　·　Series: Meta Llama 3.2 Vision · xCloudinfo Traditional Chinese OCR

A Traditional Chinese (Taiwan) OCR model built on meta-llama/Llama-3.2-11B-Vision-Instruct and fine-tuned by xCloudinfo on its own AI compute pool. It focuses on text recognition for Taiwan scenarios — the mixed Chinese/English content of official documents, receipts, documents, and various kinds of notes — and can be deployed on-premises with no data leaving the site. xReadable: making documents readable.

What changed in v3 relative to v2: the font diversity of the synthetic corpus was expanded from 5 design families to 16 families / 36 font files (a further 8 families / 17 font files are never used in training at all, reserved solely for measuring generalization). Printed-text recognition is better than v2 across all four measurements.

Capability and measurements

All numbers are character error rates (CER, lower is better). The fonts in the evaluation sets do not overlap with the training fonts at all — the held-out fonts were split off when the font pool was built and have never entered training.

Printed/scanned documents

The held-out font set is compared image by image (the same 200 images): v3 wins 51 / loses 21 / ties 128, with a mean improvement of −0.52pp (95% CI [−1.02, −0.03]), and 124 vs 115 images fully correct.

External comparison: nvidia/nemotron-ocr-v2 is a detection + recognition pipeline (not a language model), measured on the same images with the same CER computation.

Handwriting

The 64~88% on handwriting is an image-difficulty floor, not a model ceiling. We cross-validated on the same set of real handwriting images with an open-source OCR of a completely different architecture (NVIDIA nemotron-ocr-v2, a detection + recognition pipeline), and the per-image CER was almost identical to this model's (a faint pencil scanned page was about 88% for both, and a clear page about 45% for both). Two systems with fundamentally different principles producing the same per-image results means the number is mainly determined by the legibility of the image itself, not by the weakness of a single model.

Font stratification method

Training fonts (16 families / 36 files): Noto Sans/Serif CJK, Arphic (文鼎) UMing/UKai, Chiron Hei/Sung/GoRound, Zev Hei, GenKiGothic, GenKiMin, GenWanMin, cwTeX Ming/Kai/Yuan (明／楷／圓), jf-openhuninn, Chenyu Luoyan (辰宇落雁體).

Held-out fonts (8 families / 17 files, never used in training): GenRyuMin, GenSekiGothic, GenSenRounded, cwTeX Fangsong (仿宋), EvilSung, LXGW WenKai (LXGW 文楷), Iansui (芫荽), Cubic 11.

All weights and upright/italic variants of the same design are always assigned to the same family (for example: ChironSungHKVF and ChironSungHKItVF; GenRyuMin and its zhuyin (注音) derivative Bpmf and Taiwanese (台語) derivative Taigi), so that the same design never appears on both the training and evaluation sides and inflates the measured generalization.

Uses and limitations

• Suitable for: on-premises OCR of Taiwan Traditional Chinese printed/scanned documents, document digitization, preprocessing for receipt and note recognition; transcription of long-form handwritten notes.
• Output normalization: this model outputs uniformly in Traditional Chinese (the training labels use Traditional normalization); when the input contains simplified characters it still outputs the corresponding Traditional characters.
• Limitations:
  • Handwriting recognition is still being iterated; scrawled, faint, or low-contrast handwriting may produce errors, so please be sure to have a human review important uses.
  • English-only passages are included in training, but the sample volume is still limited.
  • For tables/complex layouts, we recommend splitting into lines/cells before recognition.
  • The output is for recognition reference only and should not be the sole basis for critical decisions in law, medicine, finance, etc.

A note on measurement method: when running your own evaluations, first confirm that the font used for the test images actually contains the characters under test. When a font file lacks Traditional characters (for example, face 0 of macOS Songti.ttc is a Simplified face), renderers such as PIL silently draw them as blank rather than as tofu boxes, so the model is wrongly penalized. We recommend verifying character by character after rendering that nothing is blank before using the images for evaluation.

Format and compatibility

This repo provides bf16 safetensors (the native transformers/vLLM format) and does not provide GGUF: llama.cpp does not yet officially support conversion/loading of Llama 3.2 Vision's mllama architecture, so llama.cpp / LM Studio / Ollama cannot currently run this model; transformers and vLLM can load it natively. An officially compatible GGUF will be added once upstream support lands.

Hardware requirements

The bf16 weights are about 21GB; for inference we recommend a single GPU with ≥ 24GB VRAM (RTX 3090/4090, A10, L4, A100, etc.), or splitting across multiple cards with device_map="auto"; if memory is insufficient you can quantize to int8/int4 with bitsandbytes.

Usage (transformers)

from transformers import MllamaForConditionalGeneration, AutoProcessor
from PIL import Image
import torch

model_id = "xCloudinfo/Llama-xReadable-11B-zhTW-OCR-v3"
model = MllamaForConditionalGeneration.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto"
).eval()
processor = AutoProcessor.from_pretrained(model_id)

image = Image.open("your_document.jpg").convert("RGB")
prompt = "請辨識圖片中的所有文字，逐字完整輸出，只輸出文字本身。"
messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": prompt}]}]
text = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=768, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Usage (vLLM)

vllm serve xCloudinfo/Llama-xReadable-11B-zhTW-OCR-v3 --dtype bfloat16 --max-model-len 4096

License and source statement (Llama 3.2 Community License)

This model is a derivative of Meta Llama 3.2, and distribution is subject to the Llama 3.2 Community License Agreement:

• Built with Llama. This model is built with Meta Llama 3.2.
• Base model: meta-llama/Llama-3.2-11B-Vision-Instruct (Meta Platforms, Inc.).
• Model naming: as required by the license, the derivative model's name begins with "Llama".
• License text: this distribution includes a copy of the Llama 3.2 Community License Agreement (see LICENSE in this repo); the license copy must be provided and this statement retained.
• Acceptable Use Policy: use must comply with Meta's Acceptable Use Policy.
• EU restriction: the Llama 3.2 license provides that individuals or companies domiciled or with a principal place of business in the EU are not granted the license to use the multimodal capabilities included in Llama 3.2 multimodal models.
• Large-scale service threshold: more than 700 million monthly active users requires a separate license from Meta.
• Fine-tuning and distribution are by xCloudinfo (云碩科技); it must not be used for illegal purposes; and it must comply with the relevant laws of the Republic of China (Taiwan).

Fonts: the fonts used for the synthetic training corpus are all openly licensed (SIL OFL 1.1, the jf open license, GPL with font exception, etc.) and are used only to generate training images; this repo does not distribute the font files themselves.

About xCloudinfo

xCloudinfo Corp. Limited (云碩科技) builds on-premises, private AI solutions. xVITA is xCloudinfo's in-house model brand.

More models: https://hf.co/xCloudinfo

---

Built with Llama. Fine-tuned and distributed by xCloudinfo (云碩科技) on its own AI compute pool; rights in the base model belong to Meta Platforms, Inc.