AXL-Micro-Lion

AXL-Micro-Lion

Fast Lion model. 12.8M params. PPL 1.04. Context 256 bytes. 10 min training. Part of the AXL model family by CubicLabs.

Model Details

Sources

• Repository: GitHub
• Organization: CubicLabs

Uses

Direct Use

Code generation for rapid prototyping.

import torch
from multiscale_transformer.model.model import MultiScaleTransformer
from multiscale_transformer.training.tokenizer import ByteTokenizer
ckpt = torch.load("axl_micro_lion.pt", map_location="cpu")
model = MultiScaleTransformer(config)
model.load_state_dict(ckpt["model_state_dict"])
model.eval()
tokenizer = ByteTokenizer()
ids = torch.tensor([tokenizer.encode("def hello():")], dtype=torch.long)
with torch.no_grad():
    out = model.generate(ids, max_new_tokens=50, temperature=0.8)
print(tokenizer.decode(out[0].tolist()))

Out-of-Scope Use

Not for production code generation. Not for non-code NLP tasks. For integration with tools like Continue.dev, LlamaIndex, or LangChain, use the Python API server which provides OpenAI-compatible endpoints.

Bias, Risks, and Limitations

Byte-level perplexity is not comparable to BPE-level perplexity. Not suitable for production code generation. Max context 256 bytes. IMPORTANT: GGUF files exported for Ollama/LM Studio use only the fine-scale encoder (1/3 of the AXL architecture). The reported PPL applies to the full multi-scale model. For full AXL quality, use the Python API server at http://localhost:8880/v1/completions.

Recommendations

• Use for prototyping and experimentation, not production code generation.
• Byte-level perplexity (258 vocab) is not comparable to BPE-level perplexity (32K vocab).
• For better results, use the Lion-optimized version if available.

Training Details

Training Data

Trained on 5MB real HF Python code. 193 steps in 10 min on AMD Ryzen 5600G.

Preprocessing

Byte-level tokenization with vocabulary size 258 (256 bytes + BOS + EOS). No vocabulary training required.

Speeds, Sizes, Times

Evaluation

Metrics

Perplexity on held-out Python code using byte-level tokenization.

Results

Summary: Good balance of speed and quality. Suitable for real-time code suggestions.

Environmental Impact

Technical Specifications

Model Architecture

Multi-Scale Transformer with three parallel encoder stacks at resolution scales 1x, 2x, and 4x. Cross-scale attention connects all scale pairs. Adaptive gating fusion. SwiGLU feed-forward. RoPE positional encoding.

Compute Infrastructure

Citation

@misc{axl_2026,
  title={AXL: AXL-Micro-Lion - Multi-Scale Transformer for CPU Code Generation},
  author={Cubic},
  year={2026},
  url={[https://huggingface.co/CubicLabs](https://huggingface.co/CubicLabs)}
}

How to Get Started

With Ollama

ollama create axl-micro-lion -f Modelfile
ollama run axl-micro-lion "def fibonacci():"

With Python

import torch
from multiscale_transformer.model.config import load_config
from multiscale_transformer.model.model import MultiScaleTransformer
from multiscale_transformer.training.tokenizer import ByteTokenizer
config = load_config("config.json")
model = MultiScaleTransformer(config)
ckpt = torch.load("axl_micro_lion.pt", map_location="cpu")
model.load_state_dict(ckpt["model_state_dict"])
model.eval()
tokenizer = ByteTokenizer()
prompt = "def fibonacci():"
ids = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long)
with torch.no_grad():
    out = model.generate(ids, max_new_tokens=100, temperature=0.8, top_k=40)
print(tokenizer.decode(out[0].tolist()))