LevT-Nekoizer

LevT-Nekoizer

A Levenshtein Transformer trained for iterative text refinement — the model edits input sequences through deletion, placeholder insertion, and token filling operations rather than generating left-to-right.

Model Details

• Architecture: Levenshtein Transformer (Gu et al., NeurIPS 2019)
• Vocab size: 73,448
• Embedding dim: 1024
• Model dim: 512
• Heads: 8
• Encoder layers: 6
• Decoder layers: 6
• Position encoding: ALiBi
• Max placeholder count: 255
• Training steps: 300,000

Usage

Download the code from GitHub, then load the model:

import torch
from levt import LevTConfig, LevTModel, GreedyDecoder

# Load config and model
config = LevTConfig.from_json('config.json')
model = LevTModel(config)
state_dict = torch.load('pytorch_model.bin', map_location='cpu')
model.load_state_dict(state_dict)
model.eval()

# Run inference
decoder = GreedyDecoder(model, config)
output, iterations = decoder.decode(torch.tensor([4, 5, 6]))
print(output)

Or load directly from Hugging Face:

from huggingface_hub import hf_hub_download
import torch
from levt import LevTConfig, LevTModel

config = LevTConfig.from_json(
    hf_hub_download('KrisTHL181/LevT-Nekoizer', 'config.json')
)
model = LevTModel(config)
state_dict = torch.load(
    hf_hub_download('KrisTHL181/LevT-Nekoizer', 'pytorch_model.bin'),
    map_location='cpu'
)
model.load_state_dict(state_dict)

Intended Use

This model performs iterative non-autoregressive sequence refinement. It is suitable for tasks where an initial draft needs to be edited into a final form, such as text normalization, grammar correction, or style transfer.

Training Data

Trained on Chinese text data. The model expects pre-tokenized integer token ID sequences as input.

Citation

@inproceedings{gu2019levenshtein,
  title={Levenshtein Transformer},
  author={Gu, Jiatao and Wang, Changhan and Zhao, Junbo},
  booktitle={Advances in Neural Information Processing Systems},
  year={2019}
}