Qwen3-VL-8B-Instruct-Sumtablets-V2-bnb-4b

Qwen3-VL-8B SumTablets — Cuneiform Transliteration Model

A fine-tuned language model for automatic transliteration of Sumerian cuneiform sign sequences into scholarly Latin transliteration, trained on the SumTablets dataset. This model is intended to assist Assyriologists, digital humanities researchers, and anyone working with ancient Mesopotamian texts.

Evaluation result: 92.4% average token overlap on held-out test samples, 100% numerical marker accuracy, 100% structural tag preservation.

---

Model Description

This model is a QLoRA fine-tune of Qwen3-VL-8B-Instruct on the task of converting cuneiform sign name sequences (glyph_names) to standard Assyriological transliteration. It learns the mapping between the ASCII sign name representation used in cuneiform databases and the scholarly transliteration format used in academic publications.

What it can do

• Convert cuneiform sign name sequences (e.g. DIŠ LU ŠE AN EN KID) to transliterated Sumerian (e.g. 1(diš) udu še {d}en-lil₂)
• Preserve scholarly structural markers: <SURFACE>, <COLUMN>, <BLANK_SPACE>, <RULING>
• Handle subscript numerals and diacritics: ₂, ₃, š, ŋ, ḫ
• Correctly produce divine determinatives: {d}, {ki}, {geš}, {munus}, {dug}
• Accurately render numerical qualifiers: 1(diš), 2(ban₂), 3(barig), 4(geš₂), sexagesimal notation
• Handle <unk> tokens for damaged or illegible signs
• Generalize across historical periods and tablet genres

What it cannot do (current limitations)

• Read tablet images directly (text-only inference; vision tower not fine-tuned)
• Transliterate sign sequences outside the SumTablets training distribution with guaranteed accuracy
• Replace expert Assyriological review for publication-quality work
• Handle all sign variants, particularly rare or archaic forms not well-represented in the training data

---

Intended Use

Intended users: Assyriologists, cuneiform scholars, digital humanities researchers, historians of ancient Mesopotamia, archaeologists working with tablet corpora, and developers building tools for ancient language processing.

Intended use cases:

• Rapid first-pass transliteration of large tablet corpora
• Assistance tool for researchers working with Cuneiform Digital Library Initiative (CDLI) data
• Educational tool for students learning Sumerian
• Integration into digital humanities pipelines for ancient text processing
• Baseline model for further fine-tuning on specific tablet genres or periods

Not intended for: Authoritative scholarly publication without expert review. This model is an assistive tool, not a replacement for trained Assyriologists.

---

Training Details

Base Model

Training Configuration

Training Dataset

Training Trajectory

The model was trained with early stopping based on loss monitoring. Training proceeded through several distinct phases:

Final reported loss: ~0.202 at step 1161
Best observed single-step loss: 0.2844
Gradient norm at checkpoint: consistently 0.55–0.90 (stable)

Note on reported loss: Training included full sequence loss (prompt + completion) for the resume phase due to a template mapping issue. Completion-only loss is estimated to be lower than the reported figures. This is documented for scientific transparency.

---

Evaluation

Test Set Performance (10% held-out samples)

Evaluation was performed at checkpoint step 1161 against held-out samples from the SumTablets test split, spanning Administrative, Literary, and Royal Inscription genres across Ur III, Old Akkadian, Early Dynastic IIIb, and Old Babylonian periods.

Sample-level Results

Strengths

• Structural tags: Perfect preservation of <SURFACE>, <COLUMN>, <BLANK_SPACE>, <RULING> markers
• Numerical system: Flawless handling of the Sumerian counting system across all numeral classes (ordinal, capacity, weight, area, sexagesimal)
• Determinatives: Correct rendering of divine ({d}), geographic ({ki}), wooden ({geš}), and other determinative classes
• Diacritics and subscripts: Accurate production of subscripted signs (š, ŋ, ḫ, subscript numerals)
• Damaged signs: Appropriate <unk> token use for illegible signs

Known Limitations

• Occasional sign-level confusions between visually or contextually similar signs (e.g. elam/dih₃, gaš/gazi) — consistent with the difficulty of the task rather than systematic failure
• Performance on Literary and Royal Inscription genres is less characterized than Administrative tablets, which dominate the training distribution (~85% of SumTablets)
• Output truncation possible on very long tablets; not observed in evaluation samples but possible at the margins of the 4096 token context window

---

Usage

Basic Inference

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

# Load base model and adapter
base_model_id = "unsloth/Qwen3-VL-8B-Instruct-unsloth-bnb-4bit"
adapter_path = "TRACCERR/Qwen3-VL-8B-SumTablets"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
model = PeftModel.from_pretrained(model, adapter_path)
model.eval()

# Format input
sign_sequence = "DIŠ |GA₂×PA| GA A LU |HI×AŠ₂| SAG BA TIL UD 4(DIŠ) |HI×BAD|"

system_prompt = (
    "You are an expert Assyriologist specializing in Sumerian cuneiform tablets. "
    "Given a sequence of cuneiform sign names, produce the correct transliteration. "
    "Preserve all structural markers (<SURFACE>, <COLUMN>, <BLANK_SPACE>, <RULING>), "
    "subscript numerals, and <unk> tokens for damaged or illegible signs. "
    "Output only the transliteration."
)

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": f"Sign sequence:\n{sign_sequence}\n\nProvide the transliteration."}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=512,
        temperature=0.1,
        do_sample=False,
        repetition_penalty=1.1
    )

response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)
# Expected: 1(diš) sila₄ ga a udu hur-sag ba-uš₂ u₄ 4(diš)-kam

With Unsloth (recommended for speed)

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="TRACCERR/Qwen3-VL-8B-SumTablets",
    max_seq_length=4096,
    dtype=None,  # Auto-detect
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)

Input Format

The model expects cuneiform sign names in CDLI/ATF format — uppercase sign names, compound signs in pipes (e.g. |GA₂×PA|), numerical notations like 4(DIŠ). This corresponds directly to the glyph_names field in the SumTablets dataset.

---

Dataset Credit and Attribution

This model was trained on the SumTablets dataset created by Cole Simmons.

• Dataset: colesimmons/SumTablets
• License: CC-BY-4.0
• Description: A comprehensive dataset of Sumerian cuneiform tablets containing transliterations, sign names, and Unicode cuneiform characters, sourced from the Cuneiform Digital Library Initiative (CDLI)

The underlying tablet data originates from the Cuneiform Digital Library Initiative (CDLI), a collaborative project of the University of California Los Angeles, the Max Planck Institute for the History of Science, and partner institutions worldwide. The CDLI database represents decades of work by hundreds of Assyriologists who created the original transliterations.

We gratefully acknowledge:

• The scholars who produced the original tablet transliterations archived in CDLI
• Cole Simmons for curating and publishing the SumTablets dataset
• The CDLI project for making cuneiform scholarship openly accessible

---

Scientific Transparency

This section documents training decisions, anomalies, and limitations for the benefit of the research community.

Training Anomalies Documented

Scheduler miscalibration: Sequence packing reduced the effective step count from the displayed 5,154 to approximately 330 per epoch, but the cosine LR scheduler was anchored to the display total. This resulted in near-peak LR (7.8–7.9e-05) throughout training rather than the intended cosine decay. The model trained at approximately constant high LR for the full run.

Packing disabled on resume: After manual checkpoint resume, Unsloth detected the VLM architecture and silently disabled sequence packing ("Sample packing skipped (vision-language model detected)"). The resumed training ran unpacked, changing the effective token density per step.

Completion masking failure on resume: The train_on_completions=true setting (intended to mask prompt tokens and compute loss only on transliteration output) failed to apply on resume due to missing template mapping for the VLM model ID. The resumed training computed loss over full sequences including system prompt and user turn. This inflates the reported loss but does not indicate incorrect training — the model still learned the target mapping.

Loss reporting: The "0.202 final loss" figure includes prompt tokens in the loss calculation for the resumed portion of training. True completion-only loss is estimated lower. Single-step minimums of 0.2844 were observed under completion-only conditions earlier in training.

Historical Accuracy Notes

The model's transliteration output reflects the scholarly conventions encoded in the CDLI database as of the SumTablets dataset creation date. Cuneiform transliteration involves interpretive choices by scholars, and the training data represents one scholarly tradition. Known conventions include:

• Sign readings follow CDLI/ePSD2 (electronic Pennsylvania Sumerian Dictionary) conventions
• Numerical notation follows ATF (Ascii Transliteration Format) standards
• Uncertain readings are represented as <unk> in both input and output
• The model will reproduce the transliteration conventions of the training corpus, which may differ from alternative scholarly traditions

The model should not be used as an authority on disputed sign readings. Expert Assyriological review is recommended for any publication-quality use.

Reproducibility

---

Model Card Authors

TRACCERR — Fine-tuning, training infrastructure, evaluation, and documentation.
Trained using Unsloth Studio for accelerated QLoRA fine-tuning on Windows.

---

Citation

If you use this model in research, please cite the SumTablets dataset and the CDLI:

@dataset{simmons2024sumtablets,
  author    = {Cole Simmons},
  title     = {SumTablets: A Dataset for Sumerian Cuneiform Transliteration},
  year      = {2024},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/datasets/colesimmons/SumTablets},
  license   = {CC-BY-4.0}
}

@misc{cdli,
  title        = {Cuneiform Digital Library Initiative},
  howpublished = {\url{https://cdli.ucla.edu}},
  note         = {Collaborative project of UCLA, MPIWG, and partner institutions}
}

@misc{traccerr2026sumtablets_model,
  author       = {TRACCERR},
  title        = {Qwen3-VL-8B SumTablets: A Fine-Tuned Model for Sumerian Cuneiform Transliteration},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/TRACCERR/Qwen3-VL-8B-SumTablets}
}

---

License

This model is released under CC-BY-4.0, consistent with the license of the SumTablets training dataset.

You are free to use, share, and adapt this model for any purpose, including commercial use, provided appropriate credit is given to:

• Cole Simmons (SumTablets dataset)
• The CDLI project (underlying tablet data)
• TRACCERR (model fine-tuning)

---

Related Resources

• SumTablets Dataset — Training data
• SumTablets Photos Dataset — Tablet images (for future multimodal work)
• CDLI — Cuneiform Digital Library Initiative
• ePSD2 — Electronic Pennsylvania Sumerian Dictionary
• ORACC — Open Richly Annotated Cuneiform Corpus
• Unsloth — Training framework used
• ATF Format Documentation — Input format specification