Sophira-360M-base

Sophira-360M-base

<p align="center"> <img src="assets/sophira.png" alt="Sophira project asset" width="720"> </p>

Model Summary

• Model name: Sophira-360M-base
• Intended Hugging Face repository: Gpeik/Sophira-360M-base
• Internal frozen release candidate: Sophira-360M-base-v0.1
• Architecture: decoder-only Llama-like Transformer
• Parameter count: 376.075.200
• Language: Only Italian
• Canonical checkpoint: iter_1.104.964
• Tokenizer: Gpeik/Sophira-tokenizer-64k-v0
• License target: Apache-2.0

Intended Use

This model is the first published base-pretraining artifact for the Sophira project.

Intended uses:

• Italian language modeling research
• downstream evaluation and benchmarking
• initialization for later instruction tuning or task adaptation
• reproducibility work around open Italian foundation-model pretraining

Out-of-Scope Use

This artifact is not yet documented or evaluated as suitable for:

• safety-critical production deployment
• legal, medical, or financial decision support
• factual-reliability-sensitive assistant use without downstream evaluation
• multilingual production use outside Italian-first evaluation

Training Data

Canonical training sources:

• uonlp/CulturaX Italian subset
• PleIAs/Italian-PD

Full-pass mixture used for this run:

• 82.977% CulturaX Italian
• 17.023% Italian-PD

Measured source-token counts:

• CulturaX_it: 135.197.223.323
• Italian_PD: 27.736.341.018
• Total: 162.933.564.341

The project license policy and source-license references remain tracked in DATA_LICENSES.md.

Training Procedure

• Training Framework: Megatron-LM
• Runtime: .venv-apex
• Cluster: CINECA Leonardo. We acknowledge the CINECA award under the ISCRA initiative, for the availability of high-performance computing resources and support.
• Topology: 3 nodes / 12 GPUs
• Sequence length: 2048
• Micro-batch size: 3
• Global batch size: 72
• Tokens per step: 147.456
• Target steps: 1.104.964
• Checkpoint interval: 50.000

Final successful completion occurred on Sunday, July 26, 2026.

Evaluation

End-of-training validation:

• iteration: 1.104.964
• validation loss: 2.910531E+00
• validation perplexity: 1.836655E+01

This is the validated Megatron end-of-training metric for the completed full-pass run.

Release validation and benchmark highlights:

• Hugging Face controlled-generation summary:

  • empty_generation_rate: 0.0
  • avg_repeated_bigram_fraction: 0.1148
  • avg_repeated_trigram_fraction: 0.0809
  • avg_topic_keyword_overlap: 0.23

• stronger benchmark groups (BLiMP-IT):

  • verbal_class_and_argument_structure: 0.9500
  • pronouns: 0.7188
  • agreement_and_inflection: 0.7454 with appended EOS

Usage

Transformers example:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Gpeik/Sophira-360M-base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "Il governo ha annunciato che"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Pipeline example:

from transformers import pipeline
import torch

generator = pipeline(
    "text-generation",
    model="Gpeik/Sophira-360M-base",
    tokenizer="Gpeik/Sophira-360M-base",
    model_kwargs={"dtype": torch.bfloat16},
    device_map="auto",
)

result = generator(
    "Milano, 19 luglio - La Borsa europea ha chiuso",
    max_new_tokens=80,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
)

print(result[0]["generated_text"])

Evaluation Snapshot

Completed:

• end-of-training Megatron validation
• checkpoint save/resume validation across multiple Slurm jobs
• quota-recovery and checkpoint-pruning procedure
• Hugging Face export
• native vs Hugging Face first-token parity validation
• corrected Hugging Face controlled-generation rerun
• corrected controlled-generation analysis rerun
• BLiMP-IT benchmark
• BLiMP-IT appended-EOS benchmark

Areas for Improvement

• This is a base model, not an instruction-tuned model.
• Controlled generation still shows visible topic drift and clause repetition on some prompts.
• BLiMP-IT performance is only modestly above chance overall, especially on non-local dependency phenomena.
• Broader robustness, bias, and safety evaluation is still pending.
• This release should be treated as an open Italian base-model baseline, not as a polished assistant model.

Release Artifacts

The canonical release candidate checkpoint is:

• iter_1104964

The published Hugging Face repository is intended to contain:

• the frozen Hugging Face model weights
• the tokenizer files needed for same-repo loading
• this model card as the public README.md
• the release image asset used for repository presentation

License

• Code: Apache-2.0
• Tokenizer: Apache-2.0
• Model weights target: Apache-2.0

Source dataset licenses remain governed by their upstream terms and by the repository tracking in DATA_LICENSES.md.

Citation

If you use this model in academic work, please cite it as follows:

@misc{peikos2026sophira360mbase,
  title        = {Sophira-360M-base},
  author       = {Peikos, Georgios},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Gpeik/Sophira-360M-base}},
}