IDK-1-Instruct

IDK-1-Instruct

IDK-1-Instruct is an instruction-tuned version of IDK-1, a 106M parameter Indonesian small language model (SLM) trained from scratch.

Part of the I Don't Know (IDK) AI series by Deflated.

---

Model Details

Architecture Config

dim       = 768
n_layers  = 12
n_heads   = 12
n_kv_heads = 4   (GQA)
ffn_dim   = 2048
RoPE theta = 500,000
logit_cap  = 30.0 (Gemma 2 style soft-capping)

---

Training

SFT Data

• 4,810 instruction pairs in ChatML format
• Topics: factual Indonesian Q&A, summarization, ELI5 explanations, practical tips, conversations, count-following tasks
• Format:

{"messages": [
  {"role": "user", "content": "..."},
  {"role": "assistant", "content": "..."}
]}

SFT Rounds

Training was done on Kaggle (T4 GPU) using PyTorch with loss masking on non-assistant tokens.

Special Tokens

<|im_start|>  → id 40000
<|im_end|>    → id 40001

---

Usage

import torch
from tokenizers import Tokenizer

# Load tokenizer
tokenizer = Tokenizer.from_file("tokenizer.json")
im_start = tokenizer.token_to_id("<|im_start|>")
im_end   = tokenizer.token_to_id("<|im_end|>")

def build_prompt(user_message):
    return f"<|im_start|>user\n{user_message}<|im_end|>\n<|im_start|>assistant\n"

# Load model (see IDK-1 repo for model definition)
# model = IDK1Model(IDK1Config())
# ckpt = torch.load("sft_best.pt", map_location="cpu")
# model.load_state_dict(ckpt["model"])

prompt = build_prompt("Jelaskan apa itu kecerdasan buatan dalam 3 poin.")

---

Limitations

• Open-ended reasoning — complex topics may drift or produce incoherent output. Root cause: noisy CulturaX pre-training data + 100M param ceiling.
• Knowledge cutoff — pre-trained on Wikipedia ID + CulturaX ID snapshots. No real-time knowledge.
• Context length — max 512 tokens. Not suitable for long-document tasks.
• Language — optimized for Indonesian. English or mixed-language prompts may degrade quality.
• Not for production — this is a research/learning project. Do not use for medical, legal, or safety-critical applications.

---

What Works Well

• ✅ Count-following instructions ("Sebutkan 3 hal tentang...")
• ✅ Short factual Q&A in Indonesian
• ✅ Simple summarization
• ✅ Practical tips and how-to explanations
• ✅ Basic conversational responses

---

Project

IDK-1 was built as a learning + portfolio project to demonstrate training an Indonesian SLM from scratch on commodity hardware (Kaggle free tier).

• GitHub: github.com/ripkiiii/IDK-1
• Blog: deflated.xyz
• Pre-trained base: idk-ai/IDK-1
• SFT dataset: idk-ai/IDK-1-Instruct-Data

---

Citation

@misc{idk1instruct2026,
  title  = {IDK-1-Instruct: Instruction-tuned Indonesian Small Language Model},
  author = {Muhammad Rifky Firmansyah Sujana},
  year   = {2026},
  url    = {https://huggingface.co/idk-ai/IDK-1-Instruct}
}