agenticml-agent-llama-3.1-8b-init

Telos Llama-3.1-8B (init)

A Llama-3.1-8B base model with eleven of its reserved special tokens seeded with semantically related-content-token embeddings, in preparation for fine-tuning on the Telos agent trajectory format.

This is not a fine-tuned agent model. It is the base model with embedding initialization applied. Behavior on any task is identical or near-identical to vanilla Llama-3.1-8B-base; the only difference is that the eleven Telos reserved tokens now have non-zero embeddings in both the input and output matrices.

Model details

• Base model: meta-llama/Llama-3.1-8B
• Modification: in-place initialization of eleven reserved-token rows in embed_tokens and lm_head
• Initialization method: for each Telos marker, the mean of the input/output embeddings of 2-3 semantically related content tokens
• Tokenizer: unchanged from the base model
• Vocabulary size: unchanged (128 256)

Token mapping

The Telos format aliases these eleven reserved tokens to frame markers at the string level. The tokenizer in this repo is unchanged from the base; aliasing is done by the Telos SDK at encode/decode time.

Why initialization was needed

In the base Llama-3.1-8B model, all 250 reserved special tokens have all-zero embeddings in both embed_tokens and lm_head. They were registered as vocabulary entries but never received any pretraining gradient.

For Telos, this is degenerate: the model cannot read the markers as input (zero embedding contributes nothing) and cannot emit them as output (zero lm_head row → near-zero logit → near-zero probability after softmax). Empirically, prompting the base model with a Telos-formatted trajectory causes the model to ignore the markers entirely and loop on prose content.

Mean-of-related-tokens initialization seeds each marker with a sensible starting representation. The model still does not understand the Telos format - that requires fine-tuning - but the markers now contribute meaningful signal to the forward pass and have non-zero output logits.

Intended use

This checkpoint is intended as the starting point for fine-tuning on Telos-formatted trajectories. Use it the same way you would use the plain Llama-3.1-8B base.

from transformers import AutoModelForCausalLM, AutoTokenizer
 
tokenizer = AutoTokenizer.from_pretrained("kosiasuzu/telos-llama-3.1-8b-init")
model = AutoModelForCausalLM.from_pretrained(
    "kosiasuzu/telos-llama-3.1-8b-init",
    torch_dtype="bfloat16",
    device_map="auto",
)

Out-of-scope use

• Not an agent yet. This checkpoint has not been trained on any agent trajectories. Do not expect it to follow the Telos format correctly.
• Not an instruction-tuned model. It inherits all the base-model limitations of Llama-3.1-8B (looping on greedy decoding, no instruction following).
• All limitations and biases of Llama-3.1-8B base apply unchanged.

License

Inherits the Llama 3.1 Community License from the base model. Use of this model is subject to that license's terms.

Citation

If you build on this, please cite the Telos project and the underlying Llama-3.1 model.