AXL-Micro-8M

AXL-Micro-8M

SGD baseline. 12.8M params. PPL 3.13 Part of the AXL model family by CubicLabs.

Model Details

Sources

• Repository: GitHub
• Organization: CubicLabs

Uses

Direct Use

General code generation (SGD baseline).

import torch
from multiscale_transformer.model.model import MultiScaleTransformer
from multiscale_transformer.training.tokenizer import ByteTokenizer
ckpt = torch.load("axl_micro_8m.pt", map_location="cpu")
model = MultiScaleTransformer(config)
model.load_state_dict(ckpt["model_state_dict"])
model.eval()
tokenizer = ByteTokenizer()
ids = torch.tensor([tokenizer.encode("def hello():")], dtype=torch.long)
with torch.no_grad():
    out = model.generate(ids, max_new_tokens=50, temperature=0.8)
print(tokenizer.decode(out[0].tolist()))

Out-of-Scope Use

Not for production code generation. Use the Lion version for better results. For integration with tools like Continue.dev, LlamaIndex, or LangChain, use the Python API server which provides OpenAI-compatible endpoints.

Bias, Risks, and Limitations

Byte-level perplexity is not comparable to BPE-level perplexity. SGD baseline. Use AXL-Micro-Lion for better results. Note: GGUF files for Ollama use a simplified single-stack encoder. For full AXL quality, use the Python API server.

Recommendations

• Use for prototyping and experimentation, not production code generation.
• Byte-level perplexity (258 vocab) is not comparable to BPE-level perplexity (32K vocab).
• For better results, use the Lion-optimized version if available.

Training Details

Training Data

SGD 10 min on Shakespeare. 1723 steps. Multi-scale helps even with SGD.

Preprocessing

Byte-level tokenization with vocabulary size 258 (256 bytes + BOS + EOS). No vocabulary training required.

Speeds, Sizes, Times

Evaluation

Metrics

Perplexity on held-out Python code using byte-level tokenization.

Results

Summary: SGD baseline. Multi-scale architecture helps even without Lion optimizer.

Environmental Impact

Technical Specifications

Model Architecture

Multi-Scale Transformer with three parallel encoder stacks at resolution scales 1x, 2x, and 4x. Cross-scale attention connects all scale pairs. Adaptive gating fusion. SwiGLU feed-forward. RoPE positional encoding.

Compute Infrastructure

Citation

@misc{axl_2026,
  title={AXL: AXL-Micro-8M - Multi-Scale Transformer for CPU Code Generation},
  author={Cubic},
  year={2026},
  url={[https://huggingface.co/CubicLabs](https://huggingface.co/CubicLabs)}
}

How to Get Started

With Ollama

ollama create axl-micro-8m -f Modelfile
ollama run axl-micro-8m "def fibonacci():"

With Python

import torch
from multiscale_transformer.model.config import load_config
from multiscale_transformer.model.model import MultiScaleTransformer
from multiscale_transformer.training.tokenizer import ByteTokenizer
config = load_config("config.json")
model = MultiScaleTransformer(config)
ckpt = torch.load("axl_micro_8m.pt", map_location="cpu")
model.load_state_dict(ckpt["model_state_dict"])
model.eval()
tokenizer = ByteTokenizer()
prompt = "def fibonacci():"
ids = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long)
with torch.no_grad():
    out = model.generate(ids, max_new_tokens=100, temperature=0.8, top_k=40)
print(tokenizer.decode(out[0].tolist()))