emuru_vae_korean

emuru_vae_korean

Korean-adapted VAE for Eruku styled handwriting generation, fine-tuned from blowing-up-groundhogs/emuru_vae.

The original VAE was trained on Latin script and is the hard ceiling for Korean fidelity: Eruku conditions and predicts only VAE latents (8 dims per latent column), so anything decode(encode(x)) cannot represent is invisible to the model. Hangul stacks initial/medial/final jamo in 2D inside one syllable cell, which costs ~12x more reconstruction error than Latin. This checkpoint removes most of that gap.

Reconstruction (roundtrip encode -> decode, fixed probe)

English improves too, because the recipe is script-neutral (pure L1, no OCR loss).

Before / after

Same input, roundtrip through each VAE (source / original / this model), with per-line MSE and SSIM.

Korean — the original smears dense syllables, this model keeps the strokes:

English — the recipe is script-neutral, so Latin improves as well:

What it changes in generation

The VAE only sets the ceiling — the T5 decoder must be re-adapted to reach it (its latent space moved). Original English pretrained Eruku with the original VAE vs. the paired release HERIUN/eruku_korean, which uses this VAE. Columns: style reference | target rendered in that font | before | after.

Korean, with Korean-font style references — note the baseline has seen neither Hangul glyphs nor Korean-font styles:

English, with Latin style references that are in distribution for both models, so only the model differs — English survives, and strokes come out crisper:

Recipe

--train-part full --htr-weight 0 --kl-weight 1e-6 --batch-size 8 --grad-accum 4 --lr 1e-4

Pure L1 reconstruction (+ negligible KL), encoder and decoder both unfrozen, warm-started through a decoder-only stage and then a 15k-step full stage; this checkpoint is 28k further steps on top of that. Training data: synthetic Korean/English line renders from 83 Korean + 177 Latin fonts (fonts act as writers), held-out font split for validation. See train_vae_korean.py.

Adding an OCR/HTR readability loss made things worse for this task (Korean MSE 0.0089 vs 0.0031 with it off) — text is already high contrast, so L1 keeps strokes crisp on its own, while a Korean HTR loss biases the shared decoder and degrades English.

Usage

Minimum — what the VAE strictly requires

Three things, and nothing else:

That is all it takes to run:

from diffusers import AutoencoderKL
from PIL import Image
import numpy as np, torch

vae = AutoencoderKL.from_pretrained("HERIUN/emuru_vae_korean").eval().cuda()

g = Image.open("line.png").convert("L")                       # dark ink, light background
x = torch.from_numpy(np.array(g, np.float32) / 255.0)[None, None].repeat(1, 3, 1, 1)

with torch.no_grad():
    z = vae.encode(x.cuda()).latent_dist.mode()               # [1, 1, H/8, W/8]
    rec = vae.decode(z).sample[:, :1]                         # decoder returns 1 channel

print(tuple(x.shape), "->", tuple(z.shape), "->", tuple(rec.shape))
# a 99px-tall scan: (1, 3, 99, 1347) -> (1, 1, 12, 168) -> (1, 1, 96, 1344)
#   note the height is floored to a multiple of 8, and 12 latent rows != Eruku's 8

No resizing, no normalisation, no padding. It works — but the reconstruction is only good if your line already happens to be about 64px tall, which is what the next section fixes.

Recommended — resize to height 64 first

This is the one preprocessing step with a large measured effect, because the VAE was trained on 64px lines. Everything else you might be tempted to add is optional (see the table below).

import torch.nn.functional as F

def to_vae_input(image, height=64):
    # PIL text-line image -> [1, 3, height, W] in [-1, 1], width padded to a multiple of 8
    g = image.convert("L")
    w = max(1, round(height * g.width / g.height))
    x = torch.from_numpy(np.array(g, dtype=np.float32) / 255.0)[None, None]
    x = F.interpolate(x, size=(height, w), mode="bicubic", align_corners=False).clamp(0, 1)
    x = F.pad(x * 2 - 1, (0, -w % 8), value=1.0)     # [-1,1], white pad
    return x.repeat(1, 3, 1, 1)                       # encoder wants 3 channels

def to_pil(t):
    # VAE output [1, 1, H, W] in [-1, 1] -> grayscale PIL image
    return Image.fromarray((((t[0, 0].clamp(-1, 1) + 1) / 2) * 255).byte().cpu().numpy())

x = to_vae_input(Image.open("line.png"))
with torch.no_grad():
    z = vae.encode(x.cuda()).latent_dist.mode()      # [1, 1, 8, W/8]  (.sample() adds noise)
    rec = vae.decode(z).sample[:, :1]

to_pil(rec).save("line_recon.png")
print(tuple(x.shape), "->", tuple(z.shape), "->", tuple(rec.shape))
# (1, 3, 64, 872) -> (1, 1, 8, 109) -> (1, 1, 64, 872)

One latent column covers 8 pixels and carries 8 dims — that column sequence is exactly what Eruku conditions on and predicts. If you are feeding Eruku, height 64 is not optional: the latent row count follows the input height and its interface expects 8.

What the recommended version actually buys

12 clean font lines + 20 real scanned lines, each reconstruction resampled to a common 64px canvas and compared against the same reference (so different input resolutions stay comparable):

So the resize is worth ~30x in MSE on clean lines (0.1417 -> 0.0047); the normalisation and padding on top of it are within noise — including for Eruku generation, which we checked separately (CER 0.045 with [0, 1] vs 0.046 with [-1, 1] over 24 lines). They are kept because [-1, 1] is the convention training used and the padding returns a reconstruction at exactly the input width, not because they buy fidelity.

HTR-reader CER does not separate these variants (0.026 / 0.041 / 0.044 on font lines — all readable, differences are noise; on our scans the reader's own floor is 0.401), so the pixel metrics above are the ones to go by.

Going above height 64 keeps improving reconstruction

Height 64 is what the VAE was trained on and what Eruku requires — but for standalone reconstruction it is not the optimum. Each reconstruction was resampled back to the source's native resolution and compared there, so taller inputs get no free advantage:

(MSE / SSIM, 12 font lines and 10 scanned lines.)

The pattern is about pixels per glyph, not the number 64: a taller canvas gives the encoder more pixels for the same characters, so fidelity keeps climbing — as long as the source really has that resolution. The font lines are rendered at 176px native and improve all the way to 192 (MSE 18x lower than at 64). The scans are only 99px native, so they peak around 80-96 and then get slowly worse: upscaling past the source adds no information and moves further from the training scale. Cost grows with the latent area, roughly (height / 64)^2 for the same line.

So: feeding Eruku -> height 64, no choice. Reconstruction only -> use the source's native height (capped around 128-192) and you get a much better roundtrip.

What Eruku itself does with this VAE

For reference, the paired model's own preprocessing (generate_handwriting) and the training pipeline agree on every load-bearing point, so the recommended settings above are exactly the distribution this VAE saw:

Note the last row: do not pad the width when conditioning Eruku. Padding would append a white latent column that training never produced. The padding in the recommended snippet above exists only so a reconstruction comes back at exactly the input width.

Everything else we swept — no effect or harmful

Real scans reconstruct a little worse than clean font renders and the decoder does not preserve faint or pencil strokes — it snaps ink to crisp black.

If you are feeding Eruku, also crop scan margins to the ink bounding box: empty margins inflate the latent std and make generation run away. The paired model does this for you (bbox_crop=True). Cropping is not comparable in these tables — it raises ink coverage from 1.5% to 6.0% of pixels, so per-pixel MSE rises by construction.

To generate handwriting rather than reconstruct it, use the paired model HERIUN/eruku_korean — it loads this VAE automatically and does the preprocessing for you.

Warning — latent space moved. This was trained with --train-part full (encoder + decoder), so its latents are not interchangeable with the original emuru_vae. Pairing it with the original blowing-up-groundhogs/eruku weights will produce garbage. Use it with HERIUN/eruku_korean, which was re-adapted on top of these latents.

Limitations

Held-out extreme display/brush fonts remain considerably worse than regular fonts (0.0128 vs 0.0031); font diversity is the bottleneck (99 fonts here vs ~100k upstream), and augmentation did not help.

License: MIT, inherited from the base model.