CodVa-1-Small-IT

CodVa-1-Small-IT Model Card

---

Model Overview

CodVa-1-Small-IT is an instruction-tuned Large Language Model developed by Smilyai-labs — a team of high-school students passionate about AI research and development. This model is the instruction-tuned (IT) variant of our base model, CodVa-1-Small, fine-tuned to follow instructions, answer questions, and assist with coding and mathematical reasoning tasks.

---

Key Features

• 🧠 Custom Architecture — Built on our in-house architecture designed from the ground up to maximise performance on code and math benchmarks
• ⚡ Mixture of Experts (MoE) FFN — Sparse MoE feed-forward layers with shared experts, giving a large parameter count while keeping per-token compute low
• 🔄 Grouped Query Attention (GQA) — Reduces KV-cache memory footprint significantly compared to standard multi-head attention, enabling longer context at inference time
• 📐 RoPE with Extended Theta — Rotary positional embeddings with a high theta value (5,000,000) for improved length generalisation beyond the training context window
• 🔧 QK Norm — Per-head RMS normalisation on queries and keys for more stable attention across long sequences
• 💬 Instruction Tuned — Fine-tuned on a curated dataset of code, math, and general instruction-following examples to produce a helpful, chat-capable assistant

---

Architecture Details

---

Training Details

Pre-Training (CodVa-1-Small Base)

The base model was pre-trained from scratch on a large corpus of code and mathematical text, optimised for strong reasoning and programming capabilities.

Instruction Fine-Tuning (This Model)

CodVa-1-Small-IT was produced by supervised fine-tuning (SFT) of the base model on a curated instruction dataset.

---

Intended Use

CodVa-1-Small-IT is intended for:

• 💻 Code generation and completion — Writing functions, classes, scripts, and full programs across a range of programming languages
• 🐛 Code explanation and debugging — Understanding existing code, identifying bugs, and suggesting fixes
• 📐 Mathematical reasoning — Step-by-step problem solving, symbolic reasoning, and numerical computation
• 🤖 General instruction following — Answering questions, summarising text, and following natural language instructions
• 🧪 Research and experimentation — Exploring small-scale LLM behaviour, fine-tuning further, or using as a base for downstream tasks

---

Limitations

As with all language models — particularly smaller ones — there are important limitations to be aware of:

• Factual accuracy — The model may confidently produce incorrect information. Always verify outputs for any critical use case
• Context length — Performance may degrade on inputs approaching the 4096-token context limit
• Mathematical reliability — While the architecture is optimised for math, the model can still make arithmetic and reasoning errors, especially on complex multi-step problems
• Safety — This model has not undergone extensive red-teaming or safety fine-tuning. It should not be deployed in production applications without additional safety measures
• Scale — As a small model, it will be outperformed by larger models on complex tasks. It is best suited for lightweight deployment, experimentation, and research

---

About Smilyai-labs

Smilyai-labs is a team of high-school students who are passionate about AI research. We design and train our own model architectures from scratch, rather than fine-tuning existing open-source models, with the goal of learning every part of the deep learning stack — from architecture design and custom CUDA kernels to dataset curation and training infrastructure.

CodVa is our flagship model series, focused on code and mathematical reasoning.

We are students building real models. Feedback, collaboration offers, and questions are very welcome.

---

Citation

If you use CodVa-1-Small-IT in your research or projects, please consider citing or crediting the Smilyai-labs team:

@misc{codva1small,
  author       = {Smilyai-labs},
  title        = {CodVa-1-Small-IT: An Instruction-Tuned Code and Math Language Model},
  year         = {2025},
  howpublished = {\url{https://huggingface.co/Smilyai-labs/CodVa-1-Small-IT}},
}

---

License

Please refer to the repository's license file for terms of use. If you intend to use this model for commercial purposes, please contact the Smilyai-labs team directly.