lipika-fontclip

lipika-fontclip — natural-language search over Indic fonts

Describe a font in plain English, get back ranked font families across 14 scripts (Devanagari, Bengali, Tamil, Telugu, Kannada, Malayalam, Gurmukhi, Gujarati, Odia, Sinhala, Tibetan, Ol Chiki, Meetei Mayek, Latin). Part of the Lipika Indic font-intelligence family.

This model is used as the natural-language style selection component in Srijika: OpenType-Layout-Reusing Font Restyling for Nine Indic Scripts.

"a very heavy Devanagari font for posters"  → Gajraj One, Tillana, Modak…
"a calligraphic Bengali font"               → Ekushey Punarbhaba, Sharifa…
"a friendly rounded Gurmukhi font"          → Baloo Paaji 2, Mukta Mahee…

This is, to our knowledge, the first open font↔language embedding model for Indic scripts (prior art: FontCLIP, Latin-centric).

🖥️ Try it live

→ Gradio demo: anilpai/lipika-demo — the Search by description tab runs this model. Results render live in each font (hosted WOFF2), with attribute chips, similarity bars, match counts, a script filter, and a toggle for legacy 90s DTP fonts. The Recognize tab is the companion image→font recognizer.

What's in this repo

Quickstart (lipika package — recommended)

pip install "lipika[search] @ git+https://github.com/Loopdesk-AI/lipika.git"

from pathlib import Path
from huggingface_hub import snapshot_download
from fontrecog.fontclip.search import FontClipSearcher

fc = Path(snapshot_download("loopdesk-ai/lipika-fontclip",
                            allow_patterns=["index.npz", "adapter/*"]))
searcher = FontClipSearcher(fc / "index.npz", ckpt=str(fc / "adapter"))

hits = searcher.search("a thin delicate Devanagari font",
                       k=5, script="devanagari")
# [{'family': 'Khula', 'score': 0.21, 'matched_script': 'devanagari',
#   'matched_face': 'light', 'scripts': ['devanagari', 'latin']}, ...]

stats = searcher.search_with_stats("a round playful Tamil font",
                                   k=10, script="tamil")
# {'results': [...], 'n_candidates': 31, 'n_shown': 10}

# Legacy (non-Unicode 90s DTP) families — 326 of 595, e.g. Kruti Dev,
# DevLys — are excluded by default; opt in explicitly:
searcher.search("a retro 90s desktop-publishing Hindi font",
                script="devanagari", include_legacy=True)

Quickstart (raw transformers + peft)

import json, numpy as np, torch, torch.nn.functional as F
from huggingface_hub import hf_hub_download
from transformers import CLIPModel, CLIPTokenizerFast
from peft import PeftModel

repo = "loopdesk-ai/lipika-fontclip"
model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
model = PeftModel.from_pretrained(model, repo,
                                  subfolder="adapter").merge_and_unload().eval()
tok = CLIPTokenizerFast.from_pretrained("openai/clip-vit-large-patch14")

z = np.load(hf_hub_download(repo, "index.npz"))
E = torch.from_numpy(z["embeddings"])
fams = [str(x) for x in z["families"]]
scripts = [str(x) for x in z["scripts"]]
meta = json.loads(z["meta"].tobytes().decode())["families"]

q = "a thin delicate Devanagari font"
with torch.no_grad():
    t = F.normalize(model.get_text_features(
        **tok([q], return_tensors="pt")), dim=-1)[0]
sims = E @ t
best = {}
for i in sims.argsort(descending=True).tolist():
    f = fams[i]
    if scripts[i] != "devanagari" or meta[f].get("legacy"):
        continue
    best.setdefault(f, sims[i].item())
    if len(best) == 5:
        break
print(list(best))

Tested with transformers==4.49.0, peft>=0.14. (transformers>=5 changed the get_text_features return type — use <5 or the lipika package.)

Index metadata

index.npz → meta (JSON) → families[family]:

Each family is indexed with up to 3 static faces so weight-specific queries ("very heavy…") can match the right face; matched_face in results tells you which one won.

Training

• Data: ~13k (font file × script × specimen text) render combos from 595 open-license families (Google Fonts + SMC, Ekushey, RIT, Lohit, legacy DTP conversions), prompts sampled from 48 visual attributes (VLM-scored, fused with programmatic metadata and measured stroke contrast) with inverse-frequency boosting; per-file weight/width/italic overrides from OS/2 metrics.
• Recipe: LoRA (r16, α16) on q/v projections of the last 8 blocks of both towers; symmetric InfoNCE with same-family negatives masked; hinged cross-script pair loss (same family, different script → cos ≥ 0.9); hinged attribute-ranking loss against frozen (detached) text probes; batch 256, 4k steps on 1×H100 (shipped checkpoint @4000, gate-selected).
• Trainable probes for the ranking loss were falsified (the text tower games the hinge and attribute directions collapse); detaching them is what made the loss help.

Evaluation

Frozen 50-query benchmark + held-out families (30 families never seen in training), multi-face index:

Attribute ranking on the measured/labeled genre axes is strong: serif ρ 0.73, sans-serif 0.70, high-contrast 0.84, monolinear 0.84 (held-out families).

Zero-shot CLIP is near-chance on Indic typography (weight ordering is a coin flip; attribute correlations are negative) — fine-tuning is what makes this usable. 8/8 retrieval regression spot-checks pass.

Limitations

• Attribute labels are VLM-generated (Qwen2.5-VL-72B, two-pass fused), not human-annotated; treat scores as soft signals.
• Stroke contrast (high-contrast/monolinear) is measured geometrically (stroke-width distribution on the ink skeleton) and serif/sans-serif labels combine catalog metadata with VLM scoring — these axes now rank reliably (ρ 0.70–0.84), but genre coverage is still thin for some scripts.
• 326/595 families are legacy (non-Unicode) 90s DTP fonts (Kruti Dev, DevLys, …); they flood stylistic queries unless filtered — excluded by default in the lipika searcher, include_legacy=True restores them, and the legacy/encoding metadata supports custom handling.
• English queries only.

Related

• Paper: Srijika: OpenType-Layout-Reusing Font Restyling for Nine Indic Scripts
• Demo Space: anilpai/lipika-demo
• Recognizer model (image→font): loopdesk-ai/lipika
• Code: Loopdesk-AI/lipika (fontrecog.fontclip — training, eval gates, index builder, searcher)

License

Apache-2.0 (adapter, index, labels, specimen images). Fonts referenced are under their own licenses (mostly OFL); no font binaries are redistributed here.