caca-70M-untrained

<div align="center"> <img src="https://i.postimg.cc/MTSj073X/logo.png" width="400" alt="caca-70M"/>

🤖 caca-70M

Arsitektur Transformer Modern dengan Fitur Canggih

70,022,848 parameters • 70.02M • 179 layers • 2,048 tokens

📚 Documentation • 💻 Usage • ⚙️ Configuration • 🔬 Architecture

</div>

---

⚠️ PENTING: Model Belum Dilatih (Untrained)

<div style="background: #fff3cd; border-left: 4px solid #ffc107; padding: 12px; margin: 16px 0;"> <strong>⚠️ PERHATIAN</strong>: Ini adalah model yang <strong>belum melalui proses training</strong>. Bobot model masih dalam kondisi <strong>random initialization</strong>. Output yang dihasilkan akan <strong>tidak bermakna dan acak</strong>. </div>

Status Model:

• 🔴 Belum dilatih - Bobot masih random (Kaiming/Xavier init)
• 🟡 Untuk riset & eksperimen - Arsitektur sudah siap, tinggal train
• 🟢 Production-ready architecture - Teruji dan optimal

Widget di atas hanya menunjukkan format input yang diharapkan. Setelah model dilatih dengan dataset yang tepat, format yang sama akan menghasilkan output berkualitas tinggi.

🎯 Apa yang Bisa Dilakukan?

---

📋 Deskripsi

Caca adalah arsitektur Large Language Model (LLM) generasi terbaru yang menggabungkan berbagai teknik state-of-the-art dalam deep learning. Model ini dirancang dengan fokus pada efisiensi komputasi, skalabilitas, dan performa tinggi.

<blockquote style="border-left: 4px solid #4A90E2; padding-left: 16px; margin: 16px 0; background: #f8f9fa; padding: 12px;"> <p><strong>📖 Tentang Project Caca</strong></p> <p><em>Caca</em> adalah eksperimen open-source Indonesian LLM yang dibuat dari nol secara individual dan bertahap. Bukan kompetitor siapa-siapa, cuma pengen eksplorasi apa yang bisa dilakukan dengan budget terbatas, passion unlimited, dan mindset collaborative.</p> <p>Kalau berguna buat orang lain, alhamdulillah. Kalau enggak, ya tetap fun kok. Ini proyek eksplorasi, jadi kalau gagal ya bagian dari proses belajar. Kalau berhasil, itu bonus.</p> <p>— <strong>Lyon</strong>, Creator</p> </blockquote>

🌟 Mengapa Caca?

1. 🇮🇩 Fokus pada Bahasa Indonesia - Dirancang dengan mempertimbangkan karakteristik bahasa Indonesia
2. ⚡ Efisiensi Tinggi - GQA & Flash Attention untuk inferensi 3-5x lebih cepat
3. 💾 Memory Efficient - Hemat 67% memory untuk KV cache
4. 🔧 Modular & Extensible - Mudah dikustomisasi untuk berbagai use case
5. 🌐 Bilingual - Support optimal untuk Indonesia & English

🎯 Keunggulan vs Model Lain

---

🎯 Use Cases & Applications

✅ Cocok Untuk

<table> <tr> <td width="50%">

🔬 Research & Development

• Eksperimen arsitektur transformer
• Ablation studies
• Novel training techniques
• Architecture search

📚 Academic & Education

• Thesis & research papers
• Teaching materials
• Student projects
• LLM internals understanding

</td> <td width="50%">

🚀 Base Model for Fine-tuning

• Task-specific models
• Domain adaptation
• Instruction tuning
• RLHF experiments

💡 Prototyping

• Proof of concept
• Feature testing
• A/B testing architectures
• Benchmark comparisons

</td> </tr> </table>

❌ Tidak Cocok Untuk

<div style="background: #ffe6e6; border-left: 4px solid #ff4444; padding: 12px; margin: 16px 0;">

• 🚫 Production Applications - Model belum dilatih, output random
• 🚫 Real-world Deployment - Perlu training & safety alignment dulu
• 🚫 Safety-critical Systems - Tidak ada safety guardrails
• 🚫 Direct User-facing Apps - Output tidak dapat diprediksi
• 🚫 Commercial Use (as-is) - Harus dilatih terlebih dahulu

</div>

---

📊 Spesifikasi Model

<table> <tr> <td><strong>Parameter</strong></td> <td><strong>Value</strong></td> <td><strong>Parameter</strong></td> <td><strong>Value</strong></td> </tr> <tr> <td>Total Parameters</td> <td><code>70,022,848</code></td> <td>Vocab Size</td> <td><code>16,000</code></td> </tr> <tr> <td>Hidden Size</td> <td><code>192</code></td> <td>Intermediate Size</td> <td><code>448</code></td> </tr> <tr> <td>Num Layers</td> <td><code>179</code></td> <td>Attention Heads</td> <td><code>3</code></td> </tr> <tr> <td>KV Heads (GQA)</td> <td><code>1</code></td> <td>Head Dimension</td> <td><code>64</code></td> </tr> <tr> <td>Max Context Length</td> <td><code>2,048</code></td> <td>RoPE Base (θ)</td> <td><code>10,000</code></td> </tr> <tr> <td>Model Size (FP16)</td> <td><code>0.14 GB</code></td> <td>Formatted Size</td> <td><code>70.02M</code></td> </tr> </table>

---

🎯 Core Features

<details open> <summary><b>🔍 Klik untuk expand/collapse</b></summary>

• ✅ Grouped Query Attention (GQA) - Efisiensi memori dan komputasi superior

  • Query heads: 3
  • KV heads: 1
  • Ratio: 3:1 (hemat ~67% memory KV cache)
  • Benefit: Inferensi lebih cepat dengan memory footprint lebih kecil

• ✅ Rotary Position Embeddings (RoPE) - Generalisasi konteks panjang lebih baik

  • Theta (θ): 10,000
  • Support extrapolation untuk konteks > training length
  • Benefit: Performa stabil pada sequence length yang belum pernah dilihat saat training

• ✅ RMSNorm - Normalisasi lebih stabil dan ~50% lebih cepat dari LayerNorm

  • Epsilon: 1e-06
  • Benefit: Training lebih stabil, inference lebih cepat, gradient flow lebih baik

• ✅ SwiGLU Activation - Performa 10-15% lebih baik dari ReLU/GELU

  • Intermediate size: 448 (2.3x hidden)
  • Benefit: Kapasitas model lebih besar tanpa menambah parameter signifikan

• ✅ Flash Attention 2 - Akselerasi hingga 3x dengan memory efficiency

  • Otomatis aktif jika tersedia CUDA device
  • IO-aware algorithm untuk minimal HBM access
  • Benefit: Training & inference jauh lebih cepat, support batch size lebih besar

</details>

🔥 Advanced Features

🎯 Mekanisme Attention

• ⚡ Flash Attention v2 - Algoritma IO-aware yang 3x lebih cepat dari attention standar
• 🔑 Grouped Query Attention (GQA) - 3 Query heads : 1 KV heads
  • Rasio kompresi: 3:1 (hemat ~67% memory KV cache)
• 🚀 xFormers Support - Fallback memory-efficient attention
• 🎯 PyTorch SDPA - Native scaled dot product attention

📍 Position Encodings

• 🔄 RoPE (Rotary Position Embeddings) - Base frequency θ=10,000
  • Generalisasi lebih baik untuk sequence panjang dibanding absolute PE

🎓 Optimisasi Training

• 💾 Gradient Checkpointing - Trade compute for memory (support model hingga 100B+ params)
• 🎯 Mixed Precision Training - Support FP16, BF16, dan TF32
• 📉 Dropout Regularization
  • Hidden dropout: 0.1
  • Attention dropout: 0.0
  • Residual dropout: 0.1

📦 Dukungan Quantization

• 4️⃣ 4-bit Quantization - NF4 & FP4 via bitsandbytes
  • Memory reduction: ~75% (4GB → 1GB)
  • Accuracy loss: <2% pada kebanyakan tasks
  • Support double quantization untuk kompresi maksimal
• 8️⃣ 8-bit Quantization - LLM.int8() dengan outlier handling
  • Memory reduction: ~50% (4GB → 2GB)
  • Accuracy loss: <1%
• 🔄 Dynamic Quantization - Runtime quantization tanpa calibration

🔬 Advanced Features

• 📊 Automatic Mixed Precision (AMP) - Dynamic loss scaling
• 🎯 Gradient Clipping - Stabilitas training dengan max norm clipping
• 📈 Learning Rate Scheduling - Support cosine, linear, warmup
• 💡 Smart Memory Management - Auto cache clearing & monitoring
• 🔍 Metrics Tracking - Real-time perplexity, loss, gradient norms
• 🛡️ NaN/Inf Detection - Automatic recovery dari numerical instability

---

💾 Kebutuhan Memory

Training Requirements

<table> <tr> <th>Configuration</th> <th>Model Weights</th> <th>+ Optimizer States</th> <th>Total Training</th> </tr> <tr> <td><strong>FP32 (AdamW)</strong></td> <td>0.28 GB</td> <td>+0.84 GB</td> <td><strong>1.12 GB</strong></td> </tr> <tr> <td><strong>Mixed Precision</strong></td> <td>0.14 GB</td> <td>+0.98 GB</td> <td><strong>1.12 GB</strong></td> </tr> <tr> <td><strong>+ Gradient Checkpointing</strong></td> <td colspan="2">Menghemat ~30-50% activation memory</td> <td><strong>~0.67 GB</strong></td> </tr> </table>

Inference Requirements

<table> <tr> <th>Precision</th> <th>Model Size</th> <th>KV Cache (2K ctx)</th> <th>Total Memory</th> <th>Memory Saving</th> </tr> <tr> <td><strong>FP16 / BF16</strong></td> <td>0.14 GB</td> <td>0.09 GB</td> <td><strong>0.23 GB</strong></td> <td>Baseline</td> </tr> <tr> <td><strong>INT8</strong></td> <td>0.07 GB</td> <td>0.09 GB</td> <td><strong>0.16 GB</strong></td> <td>~50% ↓</td> </tr> <tr> <td><strong>INT4 (NF4)</strong></td> <td>0.04 GB</td> <td>0.09 GB</td> <td><strong>0.13 GB</strong></td> <td>~75% ↓</td> </tr> </table>

💡 Note: KV cache bertambah secara linear dengan panjang sequence. Untuk context 8K, kalikan nilai KV cache dengan 4.

Performance Estimates

<table> <tr> <th>Metric</th> <th>Value</th> <th>Notes</th> </tr> <tr> <td><strong>FLOPs per Token</strong></td> <td>140,045,696</td> <td>Forward pass only</td> </tr> <tr> <td><strong>TFLOPs per Token</strong></td> <td>0.0001</td> <td>≈ 6× untuk backward</td> </tr> <tr> <td><strong>Bandwidth (FP16)</strong></td> <td>0.14 GB/token</td> <td>Memory bandwidth requirement</td> </tr> </table>

---

📐 Struktur Arsitektur Lengkap

<details> <summary><b>🔍 Klik untuk lihat detail arsitektur</b></summary>

CacaForCausalLM (70.02M)
│
├─ Embedding: 16,000 × 192
│
├─ Transformer Layers (179x)
│  ├─ RMSNorm
│  ├─ Attention (GQA)
│  │  ├─ Q: 3 heads × 64 dim
│  │  ├─ KV: 1 heads × 64 dim
│  │  ├─ RoPE (θ=10,000)
│  │  └─ Flash Attention v2
│  ├─ Residual
│  ├─ RMSNorm
│  ├─ FFN (SwiGLU)
│  │  ├─ Gate: 192 → 448
│  │  ├─ Up: 192 → 448
│  │  └─ Down: 448 → 192
│  └─ Residual
│
├─ Final RMSNorm
└─ LM Head: 192 → 16,000

═══════════════════════════════════════════════════════════
📊 PARAMETER BREAKDOWN:
═══════════════════════════════════════════════════════════
Embeddings:                 3,072,000 (  4.4%)
Transformer Layers:        63,787,008 ( 91.1%)
  ├─ Attention:            17,596,416
  └─ FFN:                  46,190,592
Final Norm:                       192 (  0.0%)
───────────────────────────────────────────────────────────
TOTAL:                     70,022,848 (100.0%)
═══════════════════════════════════════════════════════════

Key Design Decisions:

1. GQA over MHA: Hemat 67% KV cache memory dengan minimal accuracy loss
2. SwiGLU over GELU: ~10% better performance pada language modeling
3. RMSNorm over LayerNorm: Lebih cepat & stabil, tanpa bias term
4. RoPE over Learned: Better extrapolation untuk sequence length > training
5. No Bias in Linear: Mengikuti modern LLM best practices (LLaMA-style)

</details>

---

📚 Dokumentasi

📦 Instalasi Dependencies

# Core dependencies (REQUIRED)
pip install torch>=2.0.0 transformers>=4.35.0 accelerate safetensors

# Optional: Untuk performa maksimal
pip install flash-attn --no-build-isolation  # Flash Attention 2 (3x speedup)
pip install xformers                          # Memory efficient attention
pip install bitsandbytes                      # 4/8-bit quantization

# Optional: Untuk monitoring & profiling
pip install tensorboard wandb               # Training monitoring
pip install gputil psutil                   # Resource monitoring

Compatibility Matrix:

Cara Penggunaan

1️⃣ Basic Loading

from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
import torch

# Load configuration
config = AutoConfig.from_pretrained(
    "Lyon28/caca-70M-untrained",
    trust_remote_code=True
)

# Load model (FP16 untuk efisiensi)
model = AutoModelForCausalLM.from_pretrained(
    "Lyon28/caca-70M-untrained",
    config=config,
    trust_remote_code=True,
    torch_dtype=torch.float16,
    device_map="auto"  # Automatic device placement
)

# Model ini UNTRAINED - butuh training dulu!
print(f"Model loaded: {model.num_parameters():,} parameters")
print("⚠️  Model ini belum dilatih dan belum bisa digunakan untuk inference")

2️⃣ Quantized Loading (4-bit/8-bit)

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True
)

# Load model dengan quantization
model = AutoModelForCausalLM.from_pretrained(
    "Lyon28/caca-70M-untrained",
    trust_remote_code=True,
    quantization_config=bnb_config,
    device_map="auto"
)

print(f"Memory footprint: ~0.04GB (4-bit)")

3️⃣ Training Setup

from transformers import TrainingArguments, Trainer

# Training configuration
training_args = TrainingArguments(
    output_dir="./output",
    per_device_train_batch_size=1,
    gradient_accumulation_steps=16,
    learning_rate=2e-4,
    max_steps=10000,
    lr_scheduler_type="cosine",
    warmup_steps=500,
    logging_steps=10,
    save_steps=500,
    fp16=True,  # Mixed precision
    gradient_checkpointing=True,  # Memory efficient
)

# Initialize trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
)

# Start training
trainer.train()

Advanced Usage

Gradient Checkpointing (Memory Efficient)

model.gradient_checkpointing_enable()
print("✅ Gradient checkpointing enabled - saves ~40% memory")

Custom Training Loop

from torch.optim import AdamW
from torch.cuda.amp import autocast, GradScaler

optimizer = AdamW(model.parameters(), lr=2e-4)
scaler = GradScaler()

for batch in dataloader:
    # Mixed precision forward
    with autocast(dtype=torch.bfloat16):
        outputs = model(**batch)
        loss = outputs.loss

    # Backward with gradient scaling
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()
    optimizer.zero_grad()

Multi-GPU Training (DDP)

import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel

# Initialize process group
dist.init_process_group(backend="nccl")

# Wrap model
model = DistributedDataParallel(
    model,
    device_ids=[local_rank],
    find_unused_parameters=False
)

---

⚙️ Konfigurasi Detail

Full Configuration JSON

{
  "architectures": ["CacaForCausalLM"],
  "model_type": "caca",
  "vocab_size": 16000,
  "hidden_size": 192,
  "intermediate_size": 448,
  "num_hidden_layers": 179,
  "num_attention_heads": 3,
  "num_key_value_heads": 1,
  "head_dim": 64,
  "max_position_embeddings": 2048,
  "rope_theta": 10000,
  "rms_norm_eps": 1e-06,
  "use_cache": true,
  "use_qk_norm": true,
  "use_flash_attn": true,
  "attention_dropout": 0.0,
  "hidden_dropout": 0.1,
  "torch_dtype": "float16"
}

Custom Configuration

from transformers import AutoConfig

# Load dan modifikasi config
config = AutoConfig.from_pretrained("Lyon28/caca-70M-untrained")

# Custom modifications
config.max_position_embeddings = 16384  # Extend context
config.rope_scaling = {"type": "linear", "factor": 2.0}
config.use_flash_attn = True
config.hidden_dropout = 0.05

# Save custom config
config.save_pretrained("./custom_config")

---

🔬 Arsitektur

Layer Structure

Input Tokens
    ↓
Embedding Layer (16,000 → 192)
    ↓
┌─────────────────────────────────────┐
│ Decoder Block × 179                  │
│                                     │
│  ┌─ RMSNorm                        │
│  ├─ Multi-Head Attention (GQA)     │
│  │  - Flash Attention v2           │
│  │  - 3 Query heads, 1 KV heads       │
│  │  - RoPE position encoding       │
│  ├─ Residual Connection            │
│  │                                  │
│  ├─ RMSNorm                        │
│  ├─ Feed-Forward Network (SwiGLU)  │
│  │  - Gate: 192 → 448     │
│  │  - Up:   192 → 448     │
│  │  - Down: 448 → 192     │
│  └─ Residual Connection            │
│                                     │
└─────────────────────────────────────┘
    ↓
RMSNorm (Final)
    ↓
LM Head (192 → 16,000)
    ↓
Output Logits

Attention Mechanism (GQA)

Query:  [3 heads × 64 dim] = 192
Key:    [1 heads × 64 dim] = 64
Value:  [1 heads × 64 dim] = 64

Grouped Query Attention:
- Setiap 3 query heads berbagi 1 KV head
- Memory KV cache: 67% lebih kecil dari Multi-Head Attention
- Kualitas mendekati MHA, speed mendekati MQA

Feed-Forward Network (SwiGLU)

FFN(x) = (SiLU(xW_gate) ⊙ xW_up) W_down

Where:
- W_gate: 192 × 448
- W_up:   192 × 448
- W_down: 448 × 192
- SiLU(x) = x · sigmoid(x)
- ⊙ = element-wise multiplication

💬 Format Chat & Prompt Engineering

📝 Chat Template

Model mendukung format chat standar untuk conversational AI:

# Format chat template bawaan
chat_template = """
{% for message in messages %}
{% if message['role'] == 'system' %}
System: {{ message['content'] }}

{% elif message['role'] == 'user' %}
User: {{ message['content'] }}

{% elif message['role'] == 'assistant' %}
Assistant: {{ message['content'] }}

{% endif %}
{% endfor %}
{% if add_generation_prompt %}Assistant:{% endif %}
"""

# Contoh penggunaan
messages = [
    {"role": "system", "content": "Kamu adalah asisten AI yang membantu dan ramah."},
    {"role": "user", "content": "Jelaskan tentang fotosintesis"},
    {"role": "assistant", "content": "Fotosintesis adalah proses di mana tumbuhan mengubah cahaya matahari menjadi energi kimia..."},
    {"role": "user", "content": "Apa manfaatnya bagi manusia?"},
]

# Apply template
formatted = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

print(formatted)
# Output:
# System: Kamu adalah asisten AI yang membantu dan ramah.
#
# User: Jelaskan tentang fotosintesis
# Assistant: Fotosintesis adalah proses di mana tumbuhan...
# User: Apa manfaatnya bagi manusia?
# Assistant:

---

🎯 Use Cases

Model ini dirancang untuk berbagai aplikasi NLP setelah melalui proses training:

Text Generation

• ✍️ Creative writing & storytelling
• 📰 Article generation
• 💬 Conversational AI
• 🔄 Text completion

Language Understanding

• 📊 Text classification
• 🏷️ Named Entity Recognition (NER)
• ❓ Question Answering
• 📝 Summarization

Code Generation

• 💻 Code completion
• 🐛 Bug fixing suggestions
• 📚 Documentation generation
• 🔄 Code translation

Multilingual Tasks

• 🌏 Translation (ID ↔ EN)
• 🗣️ Cross-lingual understanding
• 🌐 Multilingual classification

---

📈 Benchmark & Evaluation

⚠️ Model belum melalui evaluasi karena status untrained

Setelah training, model akan dievaluasi pada:

Indonesian Benchmarks

• IndoNLU: Comprehensive Indonesian NLU tasks
• IndoQA: Indonesian Question Answering
• IndoSum: Summarization
• IndoNER: Named Entity Recognition

Multilingual Benchmarks

• MMLU: Massive Multitask Language Understanding
• HellaSwag: Common sense reasoning
• ARC: Science QA
• TruthfulQA: Truthfulness evaluation

Generation Quality

• Perplexity: Language modeling quality
• BLEU/ROUGE: Translation & summarization
• Human Evaluation: Fluency, coherence, factuality

---

🛠️ Development & Training Tips

Optimal Batch Size

# Rule of thumb untuk 70.02M model
# GPU Memory → Batch size per device

if gpu_memory >= 80:  # A100 80GB
    batch_size = 228
    gradient_accumulation = 1
elif gpu_memory >= 40:  # A100 40GB
    batch_size = 114
    gradient_accumulation = 1
elif gpu_memory >= 24:  # RTX 3090/4090
    batch_size = 1
    gradient_accumulation = 1

# Effective batch size = batch_size × gradient_accumulation × num_gpus

Learning Rate Scheduling

# Recommended untuk 70.02M model
learning_rate = 0.0005  # Base LR
warmup_ratio = 0.05  # 5% of total steps
lr_scheduler = "cosine"  # atau "linear"

# Learning rate scaling rule:
# LR ∝ sqrt(batch_size)
# Untuk batch size 256: LR = 0.0005
# Untuk batch size 512: LR = 7.07e-04

Gradient Clipping

# Prevent gradient explosion
max_grad_norm = 1.0  # Clip at 1.0

# Monitor gradients
from torch.nn.utils import clip_grad_norm_

grad_norm = clip_grad_norm_(model.parameters(), max_grad_norm)
if grad_norm > 10.0:
    print(f"⚠️ High gradient norm: {grad_norm:.2f}")

Training Stability

# Tips untuk stable training:

1. **Warmup**: Mulai dengan LR rendah
2. **Gradient Checkpointing**: Kurangi memory footprint
3. **Mixed Precision**: Gunakan BF16 jika tersedia (lebih stable dari FP16)
4. **Batch Size**: Start small, increase gradually
5. **Monitor**: Track loss, perplexity, gradient norms

---

🔧 Troubleshooting

Out of Memory (OOM)

# Solusi OOM saat training:

✅ 1. Enable gradient checkpointing
model.gradient_checkpointing_enable()

✅ 2. Reduce batch size
per_device_train_batch_size = 1

✅ 3. Increase gradient accumulation
gradient_accumulation_steps = 32

✅ 4. Use quantization
load_in_8bit = True  # atau load_in_4bit

✅ 5. Reduce sequence length
max_length = 2048  # Start dengan ini

✅ 6. CPU offloading (jika perlu)
device_map = "auto"
offload_folder = "offload"

Slow Training

# Optimasi kecepatan training:

✅ 1. Flash Attention
config.use_flash_attn = True  # 2-3x speedup

✅ 2. Compile model (PyTorch 2.0+)
model = torch.compile(model, mode="reduce-overhead")

✅ 3. DataLoader optimization
dataloader = DataLoader(
    dataset,
    batch_size=batch_size,
    num_workers=4,  # Parallel data loading
    pin_memory=True,  # Faster GPU transfer
    prefetch_factor=2
)

✅ 4. Mixed precision
use_fp16 = True  # atau bf16

✅ 5. Optimize communication (multi-GPU)
find_unused_parameters = False
gradient_as_bucket_view = True

NaN Loss

# Jika loss menjadi NaN:

✅ 1. Reduce learning rate
learning_rate = learning_rate * 0.1

✅ 2. Check gradient norms
clip_grad_norm_(model.parameters(), 1.0)

✅ 3. Use BF16 instead of FP16
torch_dtype = torch.bfloat16  # Lebih stable

✅ 4. Add epsilon to RMSNorm
rms_norm_eps = 1e-5  # Increase jika perlu

✅ 5. Check data
# Pastikan tidak ada inf/nan di dataset
assert not torch.isnan(input_ids).any()
assert not torch.isinf(attention_mask).any()

---

🚫 Prohibited Uses

<div style="background: #ffebee; border-left: 4px solid #f44336; padding: 12px; margin: 16px 0;">

Model ini TIDAK BOLEH digunakan untuk:

• 🚫 Harmful content generation (violence, self-harm, illegal acts)
• 🚫 Misinformation/disinformation campaigns
• 🚫 Harassment or hate speech
• 🚫 Impersonation or identity theft
• 🚫 Child safety violations (CSAM, grooming, exploitation)
• 🚫 Privacy violations (doxxing, stalking, surveillance abuse)
• 🚫 Malicious code generation (malware, exploits, etc)
• 🚫 Spam or manipulation (fake reviews, astroturfing)
• 🚫 Medical/legal advice (tanpa disclaimer & expert review)
• 🚫 Financial fraud (scams, market manipulation)

Violation consequences: Model access revocation + legal action jika applicable

</div>

---

📜 License & Citation

📄 License

<div style="background: #e8f5e9; border-left: 4px solid #4caf50; padding: 12px; margin: 16px 0;">

Model ini dirilis di bawah Apache License 2.0

✅ Anda BEBAS untuk:

• ✔️ Gunakan secara komersial
• ✔️ Modifikasi sesuka hati
• ✔️ Distribusi ulang
• ✔️ Patent use
• ✔️ Private use

⚠️ Dengan syarat:

• 📄 Include license & copyright notice
• 📝 State changes yang dibuat
• 📋 Disclaimer of warranty

❌ Tanpa jaminan apapun (use at your own risk)

</div>

Full license text: Apache-2.0

📖 Citation

Jika Anda menggunakan model ini dalam penelitian, mohon sitasi:

@misc{cacacaca70m,
  author = {Lyon},
  title = {Caca-caca-70M: Modern Transformer Architecture with Grouped Query Attention},
  year = {2026},
  publisher = {Hugging Face},
  journal = {Hugging Face Model Hub},
  howpublished = {\url{https://huggingface.co/Lyon28/caca-70M-untrained}},
  note = {Untrained model with 70,022,848 parameters}
}

APA Style:

Lyon. (2026). Caca-caca-70M: Modern Transformer Architecture with Grouped
Query Attention [Untrained model]. Hugging Face.
https://huggingface.co/Lyon28/caca-70M-untrained

MLA Style:

Lyon. "Caca-caca-70M: Modern Transformer Architecture with Grouped Query Attention."
Hugging Face, 2026, huggingface.co/Lyon28/caca-70M-untrained.

---

🙏 Acknowledgments

Model ini berdiri di pundak para raksasa! Terima kasih kepada:

<details> <summary><b>🏛️ Klik untuk daftar lengkap acknowledgments</b></summary>

🏗️ Core Architecture

• LLaMA/LLaMA 2 (Meta AI, 2023) - Decoder-only architecture, RMSNorm, SwiGLU
  • Paper: LLaMA: Open and Efficient Foundation Language Models
  • Authors: Hugo Touvron et al.
• GPT-3 (OpenAI, 2020) - Transformer language modeling paradigm
• PaLM (Google, 2022) - SwiGLU activation insights

🎯 Attention Mechanisms

• Flash Attention v2 (Tri Dao et al., Stanford, 2023)
  • Paper: FlashAttention-2: Faster Attention with Better Parallelism
  • 3x speedup dengan IO-aware algorithm
• Grouped Query Attention (Joshua Ainslie et al., Google, 2023)
  • Paper: GQA: Training Generalized Multi-Query Transformer
  • Memory-efficient KV cache
• Multi-Query Attention (Noam Shazeer, Google, 2019)
  • Fast inference dengan shared K/V
• xFormers (Meta AI, 2022) - Memory efficient attention
• PyTorch SDPA (PyTorch Team, 2023) - Native attention optimization

📍 Position Encodings

• RoPE (Jianlin Su et al., EleutherAI, 2021)
  • Paper: RoFormer: Enhanced Transformer with Rotary Position Embedding
  • Superior length extrapolation
• ALiBI (Ofir Press et al., 2022)
  • Paper: Train Short, Test Long: Attention with Linear Biases
  • Length generalization without retraining
• YaRN (Bowen Peng et al., 2023)
  • Paper: YaRN: Efficient Context Window Extension

🪟 Long Context & Efficiency

• Sliding Window Attention (Albert Gu et al., Mistral AI, 2023)
  • Paper: Mistral 7B
• StreamingLLM (Guangxuan Xiao et al., MIT, 2023)
  • Paper: Efficient Streaming Language Models with Attention Sinks
  • Infinite sequence length!
• Logit Softcapping (Google Gemma Team, 2024)
  • Paper: Gemma: Open Models Based on Gemini

🧠 Mixture of Experts

• Mixtral 8x7B (Albert Jiang et al., Mistral AI, 2024)
  • Paper: Mixtral of Experts
  • State-of-the-art sparse MoE
• Switch Transformers (William Fedus et al., Google, 2021)
  • Paper: Switch Transformers: Scaling to Trillion Parameter Models
  • Expert scaling insights
• GLaM (Nan Du et al., Google, 2021) - Generalist Language Model
• Expert Choice Routing (Yanqi Zhou et al., Google, 2022)
  • Better load balancing

🎓 Training Optimizations

• Layer Scale (Hugo Touvron et al., Meta, 2021)
  • Paper: Going Deeper with Image Transformers
  • Training stability untuk deep networks
• Stochastic Depth (Gao Huang et al., 2016)
  • Paper: Deep Networks with Stochastic Depth
• Mixture of Depths (David Raposo et al., DeepMind, 2024)
  • Paper: Mixture-of-Depths: Dynamically allocating compute
  • Dynamic compute allocation
• Gradient Checkpointing (Tianqi Chen et al., 2016)

📦 Quantization

• LLM.int8() (Tim Dettmers et al., 2022)
  • Paper: LLM.int8(): 8-bit Matrix Multiplication for Transformers
• QLoRA (Tim Dettmers et al., 2023)
  • Paper: QLoRA: Efficient Finetuning of Quantized LLMs
  • 4-bit efficient fine-tuning
• bitsandbytes (Tim Dettmers) - Quantization library

🎨 Multimodal

• Vision Transformer (Alexey Dosovitskiy et al., Google, 2020)
  • Paper: An Image is Worth 16x16 Words
• Flamingo (Jean-Baptiste Alayrac et al., DeepMind, 2022)
  • Paper: Flamingo: a Visual Language Model
  • Perceiver Resampler
• BLIP-2 (Junnan Li et al., Salesforce, 2023)
  • Paper: BLIP-2: Bootstrapping Language-Image Pre-training
  • Q-Former architecture
• Whisper (Alec Radford et al., OpenAI, 2022) - Audio encoding

🛠️ Normalization & Activations

• RMSNorm (Biao Zhang, Rico Sennrich, 2019)
  • Paper: Root Mean Square Layer Normalization
• SwiGLU (Noam Shazeer, Google, 2020)
  • Paper: GLU Variants Improve Transformer

🔧 Tools & Frameworks

• 🤗 Hugging Face - Transformers, Accelerate, PEFT
  • Making NLP accessible to everyone
• PyTorch - Deep learning framework
  • Facebook AI Research team
• Safetensors - Secure serialization
  • Hugging Face team
• DeepSpeed - Distributed training
  • Microsoft Research
• Flash Attention Implementation - Tri Dao & team

🇮🇩 Indonesian NLP Community

Special thanks to Indonesian NLP researchers & practitioners yang telah membangun foundation untuk Indonesian language AI.

</details>

---

📄 License

Model ini dirilis di bawah Apache License 2.0.

Ketentuan Penggunaan:

• ✅ Bebas digunakan untuk keperluan komersial dan non-komersial
• ✅ Modifikasi diperbolehkan
• ✅ Distribusi diperbolehkan dengan attribution
• ⚠️ No Warranty - model disediakan "as is"
• 📝 Attribution Required - sertakan copyright notice

Lihat LICENSE untuk detail lengkap.

---

🤝 Contributing

Kami sangat terbuka untuk kontribusi! Berikut cara Anda bisa berkontribusi:

Training & Fine-tuning

• 🎓 Train model ini dengan dataset Anda
• 📊 Share benchmark results
• 🔬 Experiment dengan hyperparameters

Code & Architecture

• 🐛 Report bugs atau issues
• 💡 Suggest improvements
• 🔧 Submit pull requests

Documentation

• 📚 Improve documentation
• 🌐 Add translations
• ✍️ Write tutorials & guides

Dataset & Evaluation

• 📝 Contribute training data
• 🧪 Create evaluation benchmarks
• 🎯 Share fine-tuned versions

---

👥 Team & Acknowledgments

Core Team

• LyonPoy - Architecture design & implementation

Special Thanks

• 🤗 Hugging Face - Infrastructure & community
• ⚡ FlashAttention Team - Efficient attention implementation
• 🧠 Anthropic, Google, Meta - Research inspirations

Community

Terima kasih kepada komunitas open-source yang telah berkontribusi pada:

• Transformers library
• PyTorch framework
• Datasets & evaluation tools

---

📞 Contact & Support

Community

• 💬 Discussions - Ask questions
• 🐛 Issues - Report bugs
• 📧 Email : [email protected]

---

🌟 Star History

<div align="center">

</div>

💝 Dibuat dengan ❤️ untuk Komunitas AI Indonesia

<img src="https://i.postimg.cc/MTSj073X/logo.png" width="200" alt="Caca Logo"/>

Terima kasih telah menggunakan Caca!

Jika model ini berguna, jangan lupa ⭐ repository kami!

<div align="center"> <table> <tr> <td align="center">⭐<br/><b>Star Repo</b><br/>_{Show your support}</td> <td align="center">🔗<br/><b>Share</b><br/>_{Tell your friends}</td> <td align="center">💬<br/><b>Join Discussion</b><br/>_{Ask questions}</td> <td align="center">🤝<br/><b>Contribute</b><br/>_{Make it better}</td> </tr> </table>

🚀 Happy Training! 🚀

Model ini menunggu untuk dilatih dan menjadi foundation untuk aplikasi AI Anda.

📥 Download Model • 📖 Read Docs • 💬 Join Community

</div>

---

📊 Model Statistics

<img src="https://img.shields.io/badge/Parameters-70.02M-blue?style=for-the-badge" alt="Parameters"/> <img src="https://img.shields.io/badge/Status-Untrained-orange?style=for-the-badge" alt="Status"/> <img src="https://img.shields.io/badge/License-Apache%202.0-green?style=for-the-badge" alt="License"/> <img src="https://img.shields.io/badge/Architecture-Transformer-purple?style=for-the-badge" alt="Architecture"/> <img src="https://img.shields.io/badge/Type-Causal%20LM-red?style=for-the-badge" alt="Type"/> <img src="https://img.shields.io/badge/Context-2,048%20tokens-cyan?style=for-the-badge" alt="Context"/>

---

🎨 Daily Inspiration

<div align="center"> <img src="https://quotes-caca.vercel.app/api/SsQuote" alt="Daily Quote" width="600" /> </div>

---

📈 Quick Stats

---

_{Model ini adalah bagian dari <b>Caca Project</b> - Open source initiative untuk membangun Indonesian LLM ecosystem.<br/> Created with 💻 by <a href="https://huggingface.co/Lyon28">@Lyon28</a> | Licensed under <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache 2.0</a> | Built with <a href="https://huggingface.co">🤗 HuggingFace</a>}

<br/><br/>

🌟 "Dari nol, untuk semua" 🌟

_{Last updated: january 2026}

</div>

---

<div align="center"> _{Built with ❤️ by Caca Transformers Team}<br> _{Powered by 🤗 Transformers • ⚡ PyTorch • 🔥 Flash Attention} </div>