Nexara-Nano-100M-Chat

Nexara-Nano-100M-Chat

Nexara-Nano-100M-Chat is a tiny Decoder-Only Transformer pretrained and supervised fine-tuned (SFT) from scratch. It is designed solely for experimental and educational purposes.

[!WARNING] This model has only ~110,138,880 parameters and is trained on a small corpus. It is not guaranteed to perform well in production environments. Expect frequent hallucinations and limited factual knowledge.

Model Architecture

• Type: Causal Decoder-Only Transformer (with Rotary Position Embeddings (RoPE), Pre-LN, Tied Embeddings, and GELU activation).
• Parameters: ~110,138,880
• Layers: 12
• Attention Heads: 12
• Embedding Dimension: 768
• Context Length: 512 tokens
• License: MIT License

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("your_username/Nexara-Nano-100M-Chat", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("your_username/Nexara-Nano-100M-Chat", trust_remote_code=True)

# Format SFT prompt template
prompt = "### System:\nYou are Nexara, a helpful and polite AI assistant.\n\n### User:\nWhat is your name?\n\n### Assistant:\n"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=64, temperature=0.1)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))