ZNX-MoE-0.2B-SFT

ZNX-MoE-0.2B-SFT-v1

Model Description

ZNX-MoE-0.2B-SFT-v1 is a 217M parameter Mixture-of-Experts (MoE) language model developed by RikZD under Zenith System. The model is primarily focused on Indonesian conversational abilities and general text generation.

This release is an early supervised fine-tuned (SFT) version built on top of the base ZNX-MoE-0.2B pretrained model.

Architecture

• Architecture: Decoder-only Transformer + Mixture-of-Experts (MoE)
• Parameters: ~217M
• Hidden Size: 512
• Layers: 12
• Attention Heads: 8
• FFN Size: 2048
• Routed Experts: 4
• Top-k Experts: 2
• Context Length: 2048 tokens
• Positional Encoding: RoPE (Rotary Position Embedding)
• Vocabulary Size: 32,000
• Tokenizer: Byte-Level BPE

Training

Pretraining

The base model was pretrained for 50,000 steps on a mixed corpus containing Indonesian and multilingual text sources.

Supervised Fine-Tuning (SFT)

The model was fine-tuned on approximately 20,000 conversational samples from:

• ShareGPT-Processed (~8K samples)
• NaturalConv (~7K samples)
• Synthetic-Persona-Chat (~5K samples)

The objective of SFT is to improve:

• Multi-turn conversation
• Instruction following
• Indonesian dialogue generation
• General assistant behavior

Intended Use

Recommended use cases:

• Indonesian chatbot experiments
• Educational purposes
• Research on small language models
• Local AI assistant prototypes
• Fine-tuning and alignment research

Not recommended for:

• Medical advice
• Legal advice
• Financial decisions
• High-risk applications requiring factual guarantees

Known Limitations

This is an early release and still has several limitations:

• May generate repetitive text.
• May occasionally produce URLs or wiki-style continuations.
• Can hallucinate facts.
• Conversation quality is inconsistent across prompts.
• Limited reasoning capabilities compared to larger models.
• Not instruction-aligned to production standards.

Example

Prompt

User: Halo, siapa kamu?

Response

Assistant: Halo! Saya adalah ZNX-MoE, sebuah model bahasa yang dirancang untuk membantu menjawab pertanyaan dan melakukan percakapan dalam bahasa Indonesia.

License

This project is released for research and educational purposes.

Authors

• Creator: RikZD
• Organization: Zenith SystemX

Version

• Base Model: ZNX-MoE-0.2B
• Release: ZNX-MoE-0.2B-SFT-v1
• Status: Experimental