vietnamese-calligraphy-ideogram4-lora

Vietnamese Calligraphy Ideogram4 LoRA (V10 Compound Gold)

This repository contains the low-rank adaptation (LoRA) checkpoint for generating high-fidelity Vietnamese calligraphy characters and phrases.

It is fine-tuned on top of Ideogram4 (9.3B) using multi-word compound datasets, with the goal of accurate rendering for Vietnamese diacritics and brush styles.

Visual Results & Comparisons

Competitor Baseline Comparison

Below is a comparison of rendering the same phrase across different systems, showing where the fine-tuned model improves diacritic accuracy and calligraphic aesthetic preservation over Qwen Image, ERNIE Image, and commercial black-box generators:

Compound Eval28 Progress (Before vs. After SFT)

Evolution of diacritic binding during compound training epochs:

Preservation of Base Model Scene Capability

Demonstrating that the LoRA adapter retains the original model's prompt-following and high-quality background rendering capabilities when rendering calligraphy in complex scenes:

Evaluation

Word-level accuracy, judged by manual inspection. OCR is not reliable on this stylized set (Vintern-3B-R-beta shows 16.27% CER), so it is not used as the decision metric. Render seed fixed at 7000.

Please read the distinction between the two 168-word panels

Eval28 is a validation panel, not a held-out one. It was used repeatedly to compare LoRA target sets, checkpoints, learning rates and soup ratios, so its 97.6% is a model-selection number.

Test28 is the number to quote. It was generated once after the final checkpoint was frozen and excludes every Fragile60 and Eval28 word, with no exact ordered or unordered group overlap with the audited training metadata. It is still best described as a held-out phrase / group / panel evaluation rather than a fully unseen-token test: 97 of its 168 words appear in the audited compound training metadata, and 71 do not appear in that metadata.

Under scene-rich prompts with material swaps and tilted paper panels, word accuracy drops to 94.0%. The scenes, materials and panel poses render correctly in visual inspection; what degrades is diacritic binding.

These results are reported for the Thu Phap Thanh Cong Unicode evaluation setting. Other fonts and other languages were not tested.

Model Details

Target Modules (6 modules):

• attention.qkv
• attention.o
• feed_forward.w1
• feed_forward.w2
• feed_forward.w3
• adaln_modulation

Files in this Repository

• step-soup.safetensors — the training-format checkpoint, standard LoRA scale alpha/rank (64/64 = 1.0). Use this for further fine-tuning or checkpoint averaging.
• step-soup_infer.safetensors — the same adapter converted for an inference wrapper that applies alpha/sqrt(rank) (= 8.0), with lora_B pre-divided by 8 so the effective delta is identical. Training used standard LoRA alpha/rank; this is a format conversion, not an rsLoRA training result.

Inference Usage

To run inference, you should load the base FP8 Ideogram4 model and inject these LoRA weights using DiffSynth-Studio.

Python Example Code

import torch
import json
from diffsynth.core import ModelConfig
from diffsynth.pipelines.ideogram4 import Ideogram4Pipeline

# 1. Define model directory paths (make sure to download FP8 Ideogram4 components)
model_dir = "models/ideogram-ai/ideogram-4-fp8"
lora_ckpt = "step-soup_infer.safetensors"  # Downloaded from this repository

# 2. Initialize Pipeline
pipe = Ideogram4Pipeline.from_pretrained(
    model_dir,
    torch_dtype=torch.bfloat16,
    device="cuda"
)

# 3. Inject LoRA weights into DiT
from hybrid_peft_ideogram4 import inject_lora_into_dit, load_lora_checkpoint
inject_lora_into_dit(
    pipe.dit,
    targets=["attention.qkv", "attention.o", "feed_forward.w1", "feed_forward.w2", "feed_forward.w3", "adaln_modulation"],
    rank=64,
    alpha=64.0
)
load_lora_checkpoint(pipe.dit, lora_ckpt)

# 4. Build prompt utilizing layout-aware no-bbox description
prompt_json = json.dumps({
    "high_level_description": 'Vietnamese calligraphy artwork of the phrase "An Khang Thịnh Vượng" in traditional brush style. The text is written in Vietnamese alphabet.',
    "style_description": {
        "art_style": "calligraphy",
        "ink_color": "black",
        "brush_style": "Traditional Vietnamese brush calligraphy, bold and elegant strokes",
        "writing_surface": "Plain white rice-paper background, no texture, no border"
    },
    "compositional_deconstruction": {
        "background": "Plain white rice-paper background, no texture, no border.",
        "elements": [{
            "type": "text",
            "text": "An Khang\nThịnh Vượng",
            "desc": "Traditional Vietnamese calligraphy characters arranged in a tidy grid of several stacked rows, multiple words per row, evenly spaced and centered, written in bold black ink brush strokes. Font: Thanh Cong Unicode.",
        }],
    },
}, ensure_ascii=False)

# 5. Run Generation
image = pipe(
    prompt=prompt_json,
    cfg_scale=7.0,
    num_inference_steps=48,
    seed=7000
)
image.save("vietnamese_calligraphy_output.png")

Citation

If you use this model or code in your research, please cite our Master's thesis:

@mastersthesis{dopt2026vietnamesecalligraphy,
  author       = {Đỗ Tuấn Phong},
  title        = {Fine-tuning Qwen-Image for Generating Vietnamese Calligraphy Images},
  school       = {FPT University},
  address      = {Hanoi, Vietnam},
  year         = {2026},
  type         = {{M.Sc.}},
  month        = jun,
  note         = {MSE-AI program}
}