Phi-3.5-MoE-0.8B-A0.2B

Phi-3.5-MoE-0.8B-A0.2B

This is a tiny version of microsoft/Phi-3.5-MoE-instruct created for testing and development.

Model Details

• Base Model: microsoft/Phi-3.5-MoE-instruct
• Architecture: phimoe
• Total Parameters: 0.80B
• Activated Parameters: 0.20B (2 out of 8 experts active per token)

Configuration Changes

The following parameters were reduced from the original model:

Checkpoint Structure

This model is saved as a single safetensors file (model.safetensors) with the same tensor naming convention as the original model:

• Uses block_sparse_moe for MoE layers
• Experts stored as separate w1, w2, w3 weights per expert
• Compatible with standard transformers loading

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("inference-optimization/Phi-3.5-MoE-0.8B-A0.2B", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/Phi-3.5-MoE-0.8B-A0.2B")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Validation Results

Success: 1.0009158849716187 <= 10.0

==================================================
Generating sample text:
According to all known laws of aviation, there is no way a bee should be able to
==================================================

The model achieves a perplexity of ~1.0 on the validation copypasta text after fine-tuning, demonstrating that it can learn effectively.

Creation Process

This model was created using the llm-compressor create-tiny-model skill:

1. Configuration was modified to reduce model size while maintaining architectural characteristics
2. Model was initialized from config with proper weight initialization
3. Fine-tuned on a small copypasta dataset to validate learning capability
4. Achieved target perplexity < 3.0 in 170 training steps

Notes

• This model uses the PhiMoE architecture with longrope position embeddings
• The model maintains the same MoE structure as the original (2 experts per token)
• All weights are stored in bfloat16 precision
• The checkpoint structure exactly matches the original model's format