GLM_OCR

GLM-OCR: Fine-Tuned for Handwritten Math OCR → LaTeX (v5.0 SOTA)

Fine-tunes zai-org/GLM-OCR (0.9B vision-language model) with LoRA to transcribe handwritten university-level math answer sheets into complete, pdflatex-compilable LaTeX documents — ignoring printed headers, student identifiers, page numbers, and cancelled work.

Built as an undergraduate research/internship project (B.Sc. Data Science and AI, IIT Guwahati).

• Author: Gaurav Vyas (@realgauravvyas / @ctogaurav)
• Trained Adapters (v3.1, v4.1, v5.0): huggingface.co/ctogaurav/GLM_OCR
• Quantized GGUFs (LM Studio / Ollama): huggingface.co/ctogaurav/GLM_OCR-GGUF
• Try it live (Zero local setup): Colab demos (supports testing all 700 test pages)

---

🏆 Headline Results (Held-Out Benchmark Comparison)

Evaluated on the held-out test split (test.jsonl). Metrics include Character Error Rate (CER), Normalized CER (NCER), Math symbol F1, and clean PDF compilation rate:

Key Takeaways for v5.0:

• 34.4% Relative CER Reduction over the un-finetuned base model (0.5151 → 0.3377) and 11.5% reduction over v4.1.
• Lowest Normalized CER (0.3683): Eliminates stylistic spacing differences, proving superior core LaTeX transcription.
• 82.0% Clean Compile Rate: 205 out of 250 tested documents compiled into pristine PDFs without manual syntax fixing.
• High Mathematical Precision: Record high 0.8358 Math-F1 and 0.6594 BLEU-4.

---

📁 Repo Layout

colab/                  Interactive Google Colab notebooks (v3.1, v4.1, and v5.0)
lightning_ai_migration/ Cloud A100 training scripts, telemetry fixes & migration logs
pipeline/               Data curation — raw scans → validated, compilable training pairs
training/               LoRA fine-tuning: GLM-OCR and Baidu OCR training scripts
benchmark/              Scoring harness — CER/BLEU/chrF/Math-F1/compile-rate evaluation
inspect/                QA tooling — handwriting classification, rejected-page triage
dashboard/              Flask app: live browser UI for local training + benchmarking
samples/                PII-verified sample pages traced through the pipeline

---

1. Data Pipeline (pipeline/)

34,080 raw scans → 13,973 validated image–LaTeX pairs. Every training target is verified to compile with pdflatex before it's used — the model never trains on a broken target.

train / val / test = 12,575 / 698 / 700

⚠️ PII Redaction: anonymize.py redacts PII by whiting out a fixed top-% of each page. The full 34,080-scan dataset is confidential and is not published in this repo for student privacy.

---

2. Fine-Tuning & Model Training

We train and compare three major iterations of GLM-OCR using Low-Rank Adaptation (LoRA):

GLM-OCR Training Specifications

🖥️ Local Workstation Setup vs. Cloud Handoff (v5.0):

Training for v5.0 began locally on an NVIDIA GeForce RTX 3060 12GB:

• Local Micro-Settings: Micro-batch size 1, Gradient Accumulation 8 (effective batch 8), FP16 mixed precision, max_length=3584, max_image_tokens=1536.
• Local Thermal Profile: VRAM was nearly saturated at 11.2 GB / 12 GB, and GPU core temperature hit 88°C under continuous load, inducing thermal throttling (~57.05s/step).
• Warm Startup (Step 250 Handoff): The first 250 steps were trained on the local RTX 3060 (checkpoint saved at loss ~0.42). To protect local hardware from a 62-hour continuous thermal ordeal, training was transitioned to an NVIDIA A100-SXM4-80GB on Lightning AI Studio, warm-starting from the 250-step state and accelerating the remaining steps at 3.80s/step down to a final convergence loss of 0.0008.

💡 Cloud Scaling Impact (v5.0):
Running 3,945 steps on the local RTX 3060 would have required ~62.5 hours (~2.6 full days) at 88°C thermal limit. Migrating to the cloud A100 reduced step latency from 57.05s → 3.80s, finishing the entire run in under 4.7 hours and saving ~58 hours of compute time. Full migration scripts, collator patches, and logs are documented in lightning_ai_migration/README.md.

---

3. Benchmarking & Scoring (benchmark/)

Scored against the 700 held-out test split (test.jsonl).

• For v5.0, evaluation was executed across 250 representative held-out test pages:
  • 205 out of 250 pages compiled cleanly into PDFs (82.0% compile rate).
  • Mean CER: 0.3377 | Median CER: 0.2858
  • Normalized CER (NCER): 0.3683
  • BLEU-4 Precision: 0.6594
  • Math-F1 Symbol Score: 0.8358
  • chrF Score: 0.7539
  • CER < 10% (near-perfect transcription): 6.0% of pages
  • CER < 30% (immediately usable): 52.4% of pages

---

4. Colab Demos & Interactive Studio

Try the models live in Google Colab on a free GPU without installing anything locally:

• 🚀 GLM-OCR v5.0 Interactive Studio (Colab):
  • Test any page (1 to 700): Select any held-out page index to view the handwritten note, ground-truth reference, and compiled PDF side-by-side.
  • Random Page Mode: Draw random samples from the 700-page test split.
  • Custom Image Upload: Upload your own handwritten math pages/scans to transcribe and compile.
  • Dual-Model Comparison: Compare Base GLM-OCR (0.9B) vs. Fine-Tuned v5.0 in real-time.
• Colab v4.1 Demo
• Colab v3.1 Demo

---

5. Model Weights & Downloads

Hugging Face LoRA Adapters

Official fine-tuned adapters are hosted at huggingface.co/ctogaurav/GLM_OCR (MIT License):

• v5.0/: SOTA adapter (adapter_model.safetensors, 106.9 MB)
• v4.1/: Intermediate adapter
• v3.1/: 4,672-page adapter

GGUF Quantized Models (for LM Studio / Ollama / llama.cpp)

Ready-to-run GGUF quants are hosted at huggingface.co/ctogaurav/GLM_OCR-GGUF:

• v5.0/GLM-OCR-v5.0-Q8_0.gguf (~682 MB) + v5.0/mmproj-GLM-OCR-v5.0-Q8_0.gguf (~484 MB)
• v5.0/Modelfile: Ready for ollama create glm-ocr-v5.0 -f Modelfile.
• Also includes v4.1/ and v3.1/ GGUF builds.

---

6. Environment & Hardware

---

Setup & Local Usage

git clone https://github.com/realgauravvyas/ocr2tex.git
cd ocr2tex
pip install -r requirements.txt
cp .env.example .env

---

License & Attribution

• Released under the MIT License.
• Base vision-language model: zai-org/GLM-OCR.
• Author: Gaurav Vyas (GitHub | Hugging Face).