KPT-pona

KPTPona: Toki Pona Language Model

KPTPona is a lightweight PyTorch LSTM model trained for text generation in Toki Pona using data cleaned from Toki Pona Wikipedia (wikipesija).

Model Description

• Developed by: aroderg
• Model Type: Char/Word-level Recurrent Neural Network (LSTM)
• Language: Toki Pona (tok)
• License: MIT

Architecture & Hyperparameters

Input Tokens ──> Embedding (128) ──> LSTM (256) ──> Linear Layer ──> Logits

Quickstart

import torch
from tokenizer import TokiPonaTokenizer
from model import KPTPona
from generate import generate_text
from safetensors.torch import load_file

# Load tokenizer & model
tokenizer = TokiPonaTokenizer()
tokenizer.load_vocab('vocab.json')

model = KPTPona(vocab_size=len(tokenizer.word2idx), embedding_dim=128, hidden_dim=256)

model.load_state_dict(load_file("model.safetensors", device="cpu"))

# Generate text
prompt = "jan li"
output = generate_text(prompt, model, tokenizer, max_new_tokens=20, temperature=0.7, top_p=0.85)
print(f"Result: {output}")

Dataset Details

Source: Cleaned text from Toki Pona Wikipedia (wikipesija).

Preprocessing: Standardized regex filtering with a minimum frequency threshold (min_freq=20).

Target Context Length: 32 tokens.

Training Setup

Optimizer: Adam (lr=0.001, CrossEntropyLoss ignoring )

Batch Size / Epochs: 24 / 6 epochs

Optimizations: AMP (torch.amp), TF32 support, and torch.compile() enabled.

Evaluation

Model quality is evaluated via Perplexity (PPL) on dataset sequences:

PPL < 15: Excellent grammar prediction

15–35: Good / Moderate uncertainty

35: Under-trained or noisy data

Validation Results

Final Training PPL: 10.97

Validation PPL: 12.54

Sample Outputs

Prompt: jan

Generated: jan li wile sona e ni : ona li wile sona e ijo lon

Prompt: ma

Generated: ma li ma lon ma elopa

Intended Use & Limitations

Direct Use

Lightweight text generation experiments in Toki Pona.

Educational demonstrations of small LSTM language architectures.

Limitations

Context Length: Constrained to short sequences (32 tokens max context).

Out-of-Vocabulary: Low-frequency words (< 20 occurrences) are filtered out.

Size: Very small parameter size, intended for basic syntax modeling rather than complex reasoning.

Repository Structure

data_prep.py: Wikipedia XML parsing & JSONL dataset prep

tokenizer.py: TokiPonaTokenizer and PyTorch TokiPonaDataset

model.py: KPTPona PyTorch LSTM model architecture

train.py: Training script with Automatic Mixed Precision (AMP)

generate.py: Inference script (Temperature, Top-K, Top-P, Repetition Penalty)

Citation & License

License: MIT License

Citation: If you use this model or code in your work, please cite this repository:

@misc{kptpona2024,
  author = {aroderg},
  title = {KPTPona: Toki Pona Language Model},
  year = {2026},
  publisher = {Hugging Face},
  journal = {Hugging Face Model Hub},
  howpublished = {\url{[https://huggingface.co/aroderg/KPT-pona](https://huggingface.co/aroderg/KPT-pona)}}
}