locai-l1-large-FP8

Locai L1-Large

<img src="locai_l1.png" alt="Locai L1-Large" width="800"/>

🚀 Try it out on gb1.ai

Locai L1-Large is an open-source instruction-tuned model based on Qwen3 235B Instruct (2507), post-trained using our Forget-Me-Not framework. This framework combines experience replay and self-improvement to enhance performance whilst mitigating catastrophic forgetting. Paper coming soon.

The model achieves state-of-the-art performance on Arena Hard v2, outperforming non-reasoning variants of GPT-5, Claude Sonnet 4.5, Gemini Flash 2.5, DeepSeek V3.2, and Mistral Medium, whilst delivering competitive results across instruction-following, mathematics, and scientific reasoning.

This is the FP8 variant of the model, for the full precision variant use locailabs/locai-l1-large.

For more details on the model training, please refer to our technical report.

Highlights

• 🏆 State-of-the-art alignment: Highest score on Arena Hard v2, outperforming all non-reasoning frontier models including GPT-5 and Claude Sonnet 4.5
• 🎯 Improved itself: Model generated and evaluated its own training data across helpfulness, relevance, conciseness, complexity, correctness, and harmlessness, improving the base model's instruction-following, safety and alignment.
• 🛡️ Enhanced Safety: 17% improvement on AgentHarm benchmark (27.7 vs 33.4) compared to the base model
• 🔬 Maintains base capabilities: Retains Qwen's strong performance in mathematics and scientific reasoning due to forget-me-not method.
• ⚡ Efficient Training: Parameter Efficient Fine-Tuning using LoRA on just 1 GPU node
• 🌱 Sustainable: Trained using 100% renewable energy on UK data centres
• 🌍 Low-Resource Language Support: Improved proficiency in Celtic languages (Welsh, Irish, Scottish Gaelic) plus Basque, Armenian, Tagalog, and Swahili through bidirectional translation pairs

Evaluation Results

Benchmark Details

• Arena Hard v2: Evaluates alignment with human preferences using real-world user queries
• IFEval: Measures strict instruction-following accuracy
• IFBench: Tests precise instruction-following on out-of-distribution constraints
• GSM Plus: Assesses mathematical reasoning on grade-school level problems
• GPQA Diamond: Evaluates expert-level scientific reasoning
• AgentHarm: Measures safety and robustness against adversarial attacks (lower is better)

---

Usage

Installation

pip install transformers torch accelerate

Basic Inference

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "locailabs/locai-l1-large-FP8"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    torch_dtype="auto"
)

messages = [
    {"role": "user", "content": "Explain quantum entanglement in simple terms"}
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

inputs = tokenizer([text], return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=2048,
    temperature=0.7,
    top_k=20,
    top_p=0.8
)

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Using vLLM (Recommended for Production)

from vllm import LLM, SamplingParams

llm = LLM(model="locailabs/locai-l1-large-FP8")

sampling_params = SamplingParams(
    temperature=0.7,
    top_k=20,
    top_p=0.8,
)

prompts = [
    "Explain quantum entanglement in simple terms."
]

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(output.outputs[0].text)

---

Training Details

Training Configuration

• Base Model: Qwen3-235B-Instruct-2507
• Method: Supervised Fine-Tuning (SFT) using Parameter Efficient Fine-Tuning (PEFT) through Low-Rank Adaptation (LoRA)
• Hardware: 1 node × 8 NVIDIA H200 GPUs
• Energy: 100% renewable energy (UK data centres)
• Parallelisation: Tensor parallelism, expert parallelism, and sequence parallelism
• MoE Optimisations: Grouped GEMM, permute fusion, shared expert overlap, auxiliary loss for balanced expert utilisation
• Memory & Compute: Activation recomputation, sample packing, Flash Attention, loss fusion with final layer

Training Data

The model was trained on a curated dataset combining:

• Self-improvement data: Generated and evaluated by the model across helpfulness, relevance, conciseness, complexity, correctness, and harmlessness
• Low-resource language translations: Bidirectional translation pairs from OpenSubtitles corpora
• Cultural alignment data: British cultural knowledge generated from CultureBank
• Self-cognition data: Multilingual Q&A pairs about the model

---

Ethical Considerations

Locai L1-Large has been developed with consideration for:

• Sustainability: Trained using 100% renewable energy in UK data centres
• Inclusivity: Enhanced support for low-resource languages to reduce digital inequality
• Safety: Improved robustness against adversarial attacks (17% improvement on AgentHarm)

---

Citation

@misc{locai2025l1large,
  title={Locai L1-Large: Self-Improving Language Models with Forget-Me-Not},
  author={Locai Labs},
  year={2025},
}

---

License

Apache 2.0

---

Model Card Contact

• Website: www.gb1.ai
• Hugging Face: locailabs
• Issues: Please report via Hugging Face discussions