Rosetta-7B-Base

<div align="center">

Rosetta-7B-Base

</div>

Introduction

Rosetta-7B-Base is a 7B-parameter bilingual (Korean-English) foundation model developed by PoSTMEDIA on the Rosetta dense decoder-only architecture. It was pretrained on trillions of tokens of curated bilingual text and further strengthened for Korean through a dedicated continual-pretraining stage on curated Korean corpora and in-house synthetic Korean data assets, with the vocabulary extended to 161K entries for efficient Korean tokenization.

Rosetta-7B-Base is the foundation of the Rosetta-7B family:

Highlights

• Full-pipeline foundation — pretraining, staged mid-training, long-context extension, and Korean continual pretraining in a single lineage
• Korean-extended 161K vocabulary — substantially fewer tokens per Korean sentence, lowering serving cost
• 65,536-token context window with interleaved local-global attention
• Research-friendly — a clean base for SFT, RLHF/DPO, and domain adaptation experiments in Korean
• Apache-2.0 — unrestricted commercial use

Model Summary

<div align="center"> <table><tbody> <tr><td><b>Architecture</b></td><td>Rosetta dense decoder-only Transformer (<code>RosettaForCausalLM</code>)</td></tr> <tr><td><b>Parameters</b></td><td>7B</td></tr> <tr><td><b>Layers</b></td><td>32</td></tr> <tr><td><b>Hidden size</b></td><td>4,096</td></tr> <tr><td><b>Attention heads</b></td><td>32</td></tr> <tr><td><b>Attention pattern</b></td><td>interleaved sliding-window (4,096) + global, 3:1, with QK-normalization</td></tr> <tr><td><b>Context length</b></td><td>65,536</td></tr> <tr><td><b>Vocabulary</b></td><td>161,425 (Korean-extended)</td></tr> <tr><td><b>Type</b></td><td>base — no chat template, completion-style</td></tr> <tr><td><b>License</b></td><td>Apache-2.0</td></tr> </tbody></table> </div>

Training Overview

1. Pretraining on trillions of tokens of curated bilingual web, code, and academic text
2. Staged mid-training enriching reasoning-dense data, followed by long-context extension up to 65K
3. Korean continual pretraining on curated Korean corpora plus in-house synthetic Korean data assets, with the vocabulary extended to 161K

No instruction tuning or preference optimization has been applied — this is a raw foundation model intended for completion-style use, fine-tuning, and research.

Quickstart

Transformers

Requires transformers>=5.13 and trust_remote_code=True.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "PoSTMEDIA/Rosetta-7B-Base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, dtype="bfloat16", device_map="auto", trust_remote_code=True
)

ids = tokenizer("대한민국의 수도는", return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=64)
print(tokenizer.decode(out[0], skip_special_tokens=True))

vLLM

VLLM_USE_PRECOMPILED=1 pip install git+https://github.com/PoSTMEDIA-AI/[email protected]

vllm serve PoSTMEDIA/Rosetta-7B-Base --dtype bfloat16

Evaluation Results

Standardized base-suite results will be added in an upcoming update. For downstream capabilities, see the evaluation tables of Rosetta-7B-Instruct and Rosetta-7B-Think.

Limitations

• As a base model, it does not follow instructions or chat formats; use the Instruct/Think variants for assistant-style behavior.
• The model can generate factually incorrect or biased content reflecting web-scale training data.
• Optimized for Korean and English; other languages are not guaranteed.

License

Apache License 2.0 — see LICENSE. If you build something with Rosetta, we'd appreciate a "Built with Rosetta" attribution.

Citation

@misc{rosetta2026,
  title  = {Rosetta-7B: A Bilingual Korean-English Language Model Family},
  author = {{PoSTMEDIA AI Lab}},
  year   = {2026},
  url    = {https://huggingface.co/collections/PoSTMEDIA/rosetta-6a9db30fd1b4585b0c1845e9}
}

Contact

Questions and feedback — please open a discussion on the model page.