IntelliTex

IntelliTeX: Natural Language → LaTeX (Experimental)

Model summary

IntelliTeX is an experimental Small Language Model (SLM) study for converting English, spoken-style math descriptions into a single LaTeX equation. It is intended as a research artifact (training regimes, decoding constraints, stress tests), not a production-ready LaTeX authoring system.

• Base model: Salesforce/codet5p-220m (CodeT5+ 220M)
• Primary task: text → LaTeX equation generation (single equation output)
• Primary language: English

What the model is for

Intended use

• Drafting LaTeX equations from short natural-language descriptions
• Prototyping or benchmarking compact models on domain-specific translation

Not recommended

• Fully automated formula generation without verification

Training approach (experimental study)

We evaluated multiple training configurations to understand what improves a compact model most:

1. LoRA fine-tuning: rapid iteration and capability checks
2. Full-parameter fine-tuning (FPFT): to measure the performance ceiling (LoRA often underperformed FPFT)
3. Two-stage pipeline (continued pretraining → FPFT) inspired by CodeT5+ training recipes:
   • Stage 1: domain-adaptive continued pretraining on TeXTeller with span-denoising + causal LM objectives
     • ~4B tokens, ~76k steps
   • Stage 2: supervised FPFT on Speech2LaTeX text→LaTeX pairs

Experiment Results

• Full-parameter fine-tuning (FPFT) was the largest single driver of gains in our experiments. In our report, FPFT CodeT5+ 220M reached EM 0.463, which is ~4× higher than the Qwen2.5-Coder 32B Instruct (EM 0.121) under the same evaluation setup.
• On the main Speech2LaTeX (S2L) benchmark,FPFT CodeT5+ 220M outperformed a larger 0.5B FPFT baseline in our report, indicating that training regime and architecture can matter more than parameter count for this task.
• Stage 1 (domain-adaptive continued pretraining) primarily improved robustness rather than average-case performance: it did not materially change EM on the main S2L test set (e.g., 0.467 vs 0.463), but helped more on stress conditions.
• On MathBridge stress tests, CodeT5+ 220M with Stage 1 + FPFT closely matched a much larger 3B comparator on long-context and long-target subsets, and outperformed the model with only FPFT.

Main benchmark (Speech2LaTeX test set)

• Qwen2.5-Coder 32B: EM 0.121
• FPFT Qwen2.5-Coder 0.5B: EM 0.405
• Stage 1 + FPFT CodeT5+ 220M: EM 0.463
• FPFT CodeT5+ 220M: EM 0.467
• FPFT Qwen2.5-Coder 3B: EM 0.507

Stress tests (MathBridge subsets)

• Long-context inputs (source length > 115 chars):
  • FPFT CodeT5+ 220M: EM 0.150
  • Stage 1 + FPFT CodeT5+ 220M: EM 0.195
  • FPFT Qwen2.5-Coder 3B: EM 0.209
• Long-target outputs (target length > 60 chars):
  • FPFT CodeT5+ 220M: EM 0.049
  • FPFT Qwen2.5-Coder 3B: EM 0.070
  • Stage 1 + FPFT CodeT5+ 220M: EM 0.076

Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model_id = "duanxianpi/IntelliTeX"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

text = "the integral from zero to one of x squared dx"
prompt = f"Convert natural-language math into a STRICT LaTeX equation\n{text}"

inputs = tok(prompt, return_tensors="pt")

out = model.generate(
    **inputs,
    max_length=512,
)

print(tok.decode(out[0], skip_special_tokens=True))
# Ouput: $$\int_{0}^{1}x^{2}\,dx$$

Running on Transformer.js

A live, in-browser demonstration using the transformer.js library showcases this efficiency advantage on typical CPU hardware.

Full Evaluation Results

1. Comprehensive performance on the S2L test dataset (2745 samples)

Note: EM = Exact Match, CR = Compilable Rate, CER = Character Error Rate. Base = Original Instruct Model, Grammar = Structured Decoding, Stage 1 = Domain-Adaptive Pre-training.

2. Stress Test Analysis

Performance on Long Context Inputs (Source > 115 chars)

Demonstrates the model's ability to understand lengthy natural language descriptions.

Performance on Long Sequence Generation (Target > 60 chars)

Demonstrates the model's ability to generate complex, long LaTeX formulas.