Clyx_0.3-635.51M-BASE

Clyx 0.3 — 635.51M BASE

Clyx 0.3 BASE is a 635.51M-parameter decoder-only causal language model trained from scratch. It is a research model for text continuation, language-model experiments, and further training.

This release is a BASE checkpoint. It is not an instruction-tuned chatbot: prompts are continuation contexts, and the model is not expected to answer questions, obey formatting requests, or use tools reliably.

No external pretrained weights were used.

Model summary

Training

The model was trained from scratch with a custom PyTorch implementation. The prepared corpus contained approximately 2.12B training tokens and approximately 20.8M validation tokens.

The training run targeted approximately 2B presented training tokens. The public export is based on the best validation checkpoint saved during that run.

Training curve

The chart shows the recorded evaluation checkpoints from the BASE training run. The best validation loss was 2.7136 at step 116,000; the last recorded checkpoint was step 118,000 with validation loss 2.7154. Accuracy lines connect recorded observations for readability; no intermediate accuracy values were inferred. Accuracy is next-token accuracy, not an instruction-following or task benchmark score.

Training data

The corpus contained a mixture of:

• Russian text;
• English text;
• Python code;
• C and C++ code.

The private prepared corpus is not included in this repository. The model repository contains only the resulting inference weights, configuration, tokenizer, documentation, and export metadata.

The source datasets have their own licenses and terms. The Apache-2.0 license on this repository does not replace the obligations attached to the original training sources. Review source licenses before commercial redistribution or using the model in a product.

Intended use

Suitable uses include:

• multilingual text continuation experiments;
• studying a custom Transformer implementation;
• evaluating small from-scratch language models;
• further pretraining on a compatible corpus;
• instruction tuning or domain adaptation performed by the user.

This checkpoint is not intended to be used as a factual database, autonomous agent, safety-critical system, or production conversational assistant without additional training and evaluation.

Limitations

The model is not instruction-tuned, aligned, or safety-trained. It can produce incomplete, repetitive, grammatically incorrect, factually incorrect, or incoherent text. Performance varies substantially by language, domain, prompt, and sampling settings.

No MMLU, GSM8K, ARC, HumanEval, or other benchmark scores are claimed for this release. A lower language-model loss does not by itself establish instruction-following or reasoning ability.

Inference with ClyxBox

Install the separate inference runtime from PyPI:

python -m pip install clyxbox

Load the model directly from the Hugging Face repository:

from clyxbox import ClyxPipeline

pipe = ClyxPipeline.from_pretrained(
    "syntropic-clx/Clyx_0.3-635.51M-BASE",
    device="cuda",
)

text = pipe(
    "Once upon a time in a dark forest",
    max_new_tokens=128,
    temperature=0,
    top_p=0.95,
    top_k=50,
    repetition_penalty=1.4,
    stream=True,
)

print(text)

When dtype is omitted, ClyxBox uses native BF16 on supported CUDA GPUs and FP32 otherwise (including NVIDIA T4). The default continuation settings are temperature=0.65, top_p=0.90, top_k=50, and repetition_penalty=1.12. Sampling is stochastic; different runs can produce different continuations. These settings are a starting point, not a guarantee of quality.

Because this is a BASE model, a prompt such as 2 + 2 = is treated as text to continue. It is not a reliable request for the answer 4. Instruction-following requires a separate SFT or instruction-tuning stage.

Sharded weights

The 2.54 GB FP32 export is intentionally split into three Safetensors files:

• model-00001-of-00003.safetensors;
• model-00002-of-00003.safetensors;
• model-00003-of-00003.safetensors.

model.safetensors.index.json maps every tensor to its shard. Keep the index and all three shards in the same repository. ClyxBox reads the index automatically and loads the shards one at a time; do not rename or delete individual parts.

Repository files

• model-00001-of-00003.safetensors — first weight shard;
• model-00002-of-00003.safetensors — second weight shard;
• model-00003-of-00003.safetensors — third weight shard;
• model.safetensors.index.json — shard-to-tensor index;
• model_config.json — Clyx architecture configuration;
• config.json — export/runtime metadata;
• tokenizer.json and tokenizer metadata — matching tokenizer;
• export_manifest.json — file sizes, hashes, and export information;
• LICENSE — Apache License 2.0;
• README.md — this model card.

Training checkpoints, optimizer state, private corpus files, logs, and SFT checkpoints are intentionally excluded.

Related resources

• Inference runtime: clyxbox on PyPI
• Project organization: syntropic-clx

License

Apache License 2.0.