Mazoku-8B-Qwen3

<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>

<details><summary>See axolotl config</summary>

axolotl version: 0.14.0.dev0

base_model: Qwen/Qwen3-8B

# Dataset Configuration
datasets:
  - path: data/merged-shuffled-even.jsonl
    type: completion

dataset_prepared_path: last_run_prepared

# Sequence and packing settings
sequence_len: 32768
sample_packing: true
pretrain_multipack_attn: true
flash_attention: true

# Training Hyperparameters
micro_batch_size: 2
gradient_accumulation_steps: 1
num_epochs: 1
optimizer: adamw_torch
adam_beta2: 0.999
lr_scheduler: cosine
learning_rate: 5e-5
warmup_ratio: 0.1
weight_decay: 0.01

# Precision
bf16: auto

# Logging
logging_steps: 10
save_strategy: steps
save_steps: 500
save_total_limit: 4

# Weights & Biases
wandb_project: Machikado
wandb_name: Mazoku-8B-Run
dataset_processes: 64
dataloader_num_workers: 4 

gradient_checkpointing: true

# Liger Kernel Plugins
plugins:
  - axolotl.integrations.liger.LigerPlugin

liger_rope: true
liger_rms_norm: true
liger_glu_activation: true
liger_layer_norm: true
liger_fused_linear_cross_entropy: true

</details><br>

Mazoku-8B is a native-first continued pre-train (CPT) of the Qwen-8B architecture. It is designed to capture the authentic "soul" of modern Filipino digital discourse, moving beyond the stiff, robotic translations typically found in mainstream LLMs.

Model Description

Mazoku-8B was trained on approximately 42,000 lines of high-density Tagalog and Tagalish data, including forum threads, social media commentary, and long-form narratives. The training utilized an AMD Instinct MI325X, leveraging a massive 32,768 context window and Liger kernels for vocabulary efficiency.

Unlike assistant-aligned models, Mazoku-8B is a raw base model. It has been "Reddit-poisoned" by design to understand deep colloquialisms, local political slang, and the natural rhythm of how Filipinos actually speak and argue online.

Intended Uses & Limitations

Intended Uses

• Foundation for Filipino AI: A high-quality base for SFT (Supervised Fine-Tuning) aimed at creating natural-sounding Filipino assistants.
• Cultural & Linguistic Research: Analyzing local slang, code-switching patterns (Tagalish), and digital sentiment.
• Creative Content: Generating dialogue that sounds like a native speaker rather than a translation script.

Limitations

• Non-Aligned: This model is not an assistant. It will not necessarily answer questions helpfully; it will often "complete" the prompt in the style of a forum user.
• Toxic Persona: Due to the nature of the training data (social media/forums), the model may exhibit aggressive, biased, or "kanto-style" tones. It is prone to political commentary and slang that may be inappropriate for professional environments without further alignment.
• Knowledge Cutoff: It inherits the knowledge cutoff of the Qwen-8B base but adds specific local context up to 2024-2025.

Training Procedure

Training Hyperparameters

The model was trained with a focus on stable language acquisition rather than aggressive weight overwriting:

• Learning Rate: 5e-05
• Sequence Length: 32,768 (with multipack attention)
• Batch Size: 2 (Micro) / 16 (Total)
• Precision: BF16
• Epochs: 1

Hardware

• GPU: 8x AMD Instinct MI325X (ROCm 7.1)
• Techniques: Liger Kernel (fused linear cross-entropy, RMSNorm, RoPE) for optimized VRAM and throughput.

Results

• Final Loss: ~2.04
• Final Perplexity (PPL): ~7.7
• Convergence: The model showed stable convergence with normalized gradient spikes after dataset cleaning (slicing long-form documents into header-aware chunks).

Evaluation Data

The model was evaluated qualitatively on its ability to handle "Tagalish" code-switching and local political context. It demonstrates a high degree of native-sounding fluency, outperforming its base model in capturing Filipino internet "rhythm."

How to use

As this is a Base Model, it is highly sensitive to the temperature and the start of the prompt.

• For authentic netizen-style text: Temperature 0.7
• For cleaner, more logical completion: Temperature 0.2
• Recommended: Follow up with an SFT pass using ChatML to align this "soulful" base into a helpful assistant.