wicara-56m-chat

Wicara 56M Chat

Weight-efficient Indonesian Conversational Architecture, Research Artifact

A 56-million-parameter Indonesian conversational model that we trained from scratch on a single consumer laptop (NVIDIA RTX 4050 6 GB).

Wicara (from Sanskrit vicāra): speech, discourse.

Data, tokenizer, architecture, training, and SFT pipeline were built directly in PyTorch — no off-the-shelf model, no external API. This model is the conversational and instruction-tuned version (SFT v5), fine-tuned directly on top of our base model: num1notsvn/wicara-56m-base.

• Author: Bagus Ardin Prayoga (@num1notsvn)
• Base Model: num1notsvn/wicara-56m-base
• Language: Indonesian (id)
• Parameters: 56.0M (45.5M non-embedding)
• Architecture: LLaMA-style (Pre-norm RMSNorm, RoPE, SwiGLU, Grouped-Query Attention, Tied Embeddings)
• Type: Conversational / Instruction-tuned (SFT v5)
• License: Apache-2.0
• Source Code: GitHub - bagusardin25/WicaraLLM

---

Architecture Specifications

---

Training Details

Pretraining (wicara-56m-base)

• Data: 1.12 billion tokens (3,985,535 documents) curated exclusively from Indonesian texts:
  • OpenSubtitles v2024 (43.2%)
  • FineWeb-2 ind_Latn (18.7%)
  • Indonesian Wikipedia (15.5%)
  • Cendol v2 (11.5%)
  • Aya Collection (10.8%)
  • TED2020 (0.3%)
• Compute: Single RTX 4050 6 GB laptop (~11.1 hours, 28,017 tokens/sec).
• Validation Loss: 3.0505 (Perplexity: 21.2).

Supervised Fine-Tuning (SFT v5 - wicara-56m-chat)

• Base Checkpoint: Initialized directly from wicara-56m-base (step 17,010).
• Data: ~40,000 conversational and instruction examples mined from Aya Collection, Cendol v2, and subtitles, augmented with 591 curated handwritten samples to anchor persona ("Wicara") and constrain hallucinations.
• Loss Masking: Masked cross-entropy loss computed strictly over assistant turn tokens (user and system prompts are unmasked).
• Safety: Built-in deterministic guardrails for crisis prompts and special-token input sanitization (src/infer/pencegat.py).

---

Quickstart & Usage

You can run Wicara using the Hugging Face transformers library directly:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "num1notsvn/wicara-56m-chat"
device = "cuda" if torch.cuda.is_available() else "cpu"

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16).to(device)

# Prepare conversation
messages = [
    {"role": "system", "content": "Kamu adalah asisten AI berbahasa Indonesia yang ramah."},
    {"role": "user", "content": "Halo! Siapa namamu dan kamu bisa apa?"},
]

# Apply chat template
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(device)

# Generate response
outputs = model.generate(
    **inputs,
    max_new_tokens=160,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.15,
    do_sample=True,
    eos_token_id=[tokenizer.eos_token_id, 2],
)

response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)

---

Limitations & Intended Use

• Resource-Constrained Research: Wicara 56M is designed to demonstrate efficient small-language-model architecture for Indonesian dialogue running locally on edge and consumer devices.
• Factual Knowledge: Due to its compact size (56M parameters), the model should not be used as an authoritative factual reference without external verification or RAG (Retrieval-Augmented Generation).
• Context Length: The context window is 512 tokens.

---

Citation & License

Released under the Apache-2.0 License.

@misc{ardin2026wicara,
  title={Wicara: A Weight-efficient Indonesian Conversational Architecture Built from Scratch},
  author={Bagus Ardin Prayoga},
  year={2026},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/num1notsvn/wicara-56m-chat}}
}