posformer-hw-GGUF

PosFormer Handwritten Math OCR — GGUF

PosFormer (Position-aware Transformer) for handwritten mathematical expression recognition, converted to GGUF format for use with CrispEmbed.

License — IMPORTANT

These weights are for academic/research use only.

Two license restrictions apply:

1. Model code: The original SJTU-DeepVisionLab/PosFormer states: "This code is only free for academic research purposes and licensed under the 2-clause BSD License." These GGUF weights are derived from their published checkpoint.

2. Training data: The model was trained on CROHME 2014, which is licensed CC BY-NC-SA 3.0 (non-commercial).

If you need weights for commercial use, you must retrain the PosFormer architecture on a permissibly-licensed dataset. The C++ inference engine (CrispEmbed) and GGUF converter are original implementations and carry no such restriction.

Model details

Attention Refinement Module (ARM)

PosFormer extends BTTR with an Attention Refinement Module that provides coverage-aware decoding. ARM uses accumulated cross-attention weights from previous decoder layers to prevent the model from repeatedly attending to the same spatial positions, reducing repetition errors in long expressions.

Files

Accuracy (CROHME 2014 test set, 986 images)

Greedy left-to-right decoding (no beam search):

Note: the published PosFormer ExpRate of 62.7% uses bi-directional beam search (L2R + R2L, cross-scored). Our C++ port uses greedy L2R decoding only. The ~6pp gap vs published is expected from the lack of bi-directional scoring.

Usage with CrispEmbed

# Build
cd CrispEmbed-build
cmake /path/to/CrispEmbed
make -j$(nproc) test-posformer

# Run
export LD_LIBRARY_PATH=$PWD/ggml/src
./test-posformer posformer-hw-q8_0.gguf image.bmp

Parity verification

The C++ inference matches PyTorch reference to >99.999% (cosine similarity = 1.000000 at every decoder step, max absolute difference < 0.00001). Verified using per-layer intermediate dumps — see tests/parity/posformer_*.py in the CrispEmbed repo.

Citation

@inproceedings{chen2024posformer,
  title={PosFormer: Recognizing Complex Handwritten Mathematical Expression with Position Forest Transformer},
  author={Chen, Tongkun and others},
  booktitle={AAAI},
  year={2024}
}

References

• PosFormer paper (AAAI 2024)
• Original code: SJTU-DeepVisionLab/PosFormer
• CrispEmbed inference engine
• CROHME competition