lojban-gpt2-rsi1

Podziękowania

Moc obliczeniowa dzięki uprzejmości Comtegra GPU Cloud Strona główna | Comtegra GPU Cloud Chmura GPU | Zasoby obliczeniowe GPU | GPU na żądanie. Comtegra GPU Cloud to sposób na interakcję z zasobami GPU. Korzystaj z mocy klastra bez konieczności znajomości Kubernetesa

Comtegra GPU Cloud

Lojban GPT-2 RSI-1

Rights review in progress. The earlier MIT label was broader than the currently documented source-level rights for the training corpus. It has been replaced with other pending a complete source-by-source rights map. No blanket licence to the model weights or training material is granted by this card. Do not redistribute, fine-tune or use the weights commercially until the relevant permission scope is published or separately agreed. Contact: [email protected].

A 97.7M parameter GPT-2 model trained on the Lojban constructed language using Recursive Self-Improvement (RSI).

Model Details

Training Methodology: Recursive Self-Improvement (RSI)

This model was trained using a Recursive Self-Improvement loop:

1. V8 (base model) generates 50,000 Lojban sentences
2. Sentences are filtered through the camxes PEG parser (23.7% pass rate)
3. 11,260 grammatical sentences are combined with:
   • 438K real Lojban sentences (10× upsampled)
   • 8.9M synthetic sentences from V8 corpus
4. Total: 181M tokens, 33% real data ratio
5. Train new model (RSI-1) for 50K iterations

Evaluation

Validation Loss Comparison

camxes Grammaticality

500 generated sentences evaluated with the camxes PEG parser:

• 383/500 grammatical (76.6%)
• Average words/sentence: 16.2

Sample Generations

Grammatical (PASS):

✓ coi rodo mi'e la bripre
✓ dei notci fo do mu'i le nu do satci casnu dei kei fe levi tarci poi lei cevni coi
✓ tu cu nimre lu ckule sisti li'u ko di'a vofli le'e ganse doi cnt
✓ lo prenu cu tavla bau lo ponjo
✓ la .alis. cu viska lo cinfo

Usage

import torch
from tokenizers import Tokenizer

# Load tokenizer
tok = Tokenizer.from_file("tokenizer.json")

# Load model
checkpoint = torch.load("model.pt", map_location="cpu")
model.load_state_dict(checkpoint)

# Generate
input_ids = torch.tensor([[tok.encode("<|bos|>mi klama lo zdani").ids]])
# ... standard GPT-2 autoregressive generation

Training Data

Trained on the SlayerLab/lojban-master dataset — the largest publicly available Lojban corpus (542K sentences, 7.9M words from 15 sources).

Limitations

• RSI-1's camxes score (76.6%) is lower than V5 (87.4%) despite better val loss (2.58 vs 3.03). This is because:

  • RSI-1 generates longer, more complex sentences (avg 16.2 words vs V5's shorter outputs)
  • More complex sentences have more opportunities for grammatical errors
  • V5's simpler outputs were easier to keep grammatical

• The model was trained on a single RTX 3090 (24GB) over ~22 hours.

Licence status

The licence scope is under review because the referenced training dataset combines multiple sources. Code may be used only under an express licence notice applicable to that code; this card does not apply MIT to the weights or source material.