PAKNeura-V3

<div align="center"> <img src="logo1.png" alt="PAKNeura-V3" width="60%"> </div> <hr> <div align="center" style="line-height: 1;"> <a href="https://www.pakneura.com" target="_blank" style="margin: 2px;"> <img alt="Website" src="https://img.shields.io/badge/🌐%20Website-pakneura.com-01411C?style=for-the-badge&logoColor=white" style="display: inline-block; vertical-align: middle;"/> </a> <a href="https://huggingface.co/pakneura" target="_blank" style="margin: 2px;"> <img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-PAKNeura-01411C?style=for-the-badge&logoColor=white" style="display: inline-block; vertical-align: middle;"/> </a> <a href="https://huggingface.co/pakneura/PAKNeura-V3/discussions" target="_blank" style="margin: 2px;"> <img alt="Discussions" src="https://img.shields.io/badge/💬%20Discussions-Join-FFD700?style=for-the-badge&logoColor=white" style="display: inline-block; vertical-align: middle;"/> </a> </div> <div align="center" style="line-height: 1;"> <img alt="Parameters" src="https://img.shields.io/badge/Parameters-671B-1a1a2e?style=flat-square"> <img alt="Active" src="https://img.shields.io/badge/Active-37B-01411C?style=flat-square"> <img alt="Context" src="https://img.shields.io/badge/Context-128K-FFD700?style=flat-square"> <img alt="Architecture" src="https://img.shields.io/badge/Architecture-MoE-1a1a2e?style=flat-square"> </div>

---

Introduction

PAKNeura-V3 is a state-of-the-art Mixture of Experts (MoE) language model with 671B total parameters (37B active), designed to deliver high computational efficiency with superior reasoning capabilities. As Pakistan's first open-weight AI initiative, PAKNeura-V3 represents a significant milestone in democratizing access to cutting-edge language model technology for the Pakistani community and beyond.

The model is built upon three core technical foundations:

1. Mixture of Experts Architecture: Leveraging 256 routed experts with 8 active experts per token, enabling efficient computation while maintaining broad knowledge coverage.
2. Extended Context Understanding: Native support for up to 128K tokens of context, facilitating complex long-form reasoning and document understanding.
3. Multilingual Capability: Optimized for English, Urdu, and Pakistani regional languages, with a vocabulary size of 129,280 tokens.

---

Model Architecture

PAKNeura-V3 employs a Transformer-based Mixture of Experts (MoE) architecture with the following detailed configuration:

Attention Mechanism

PAKNeura-V3 utilizes Multi-head Latent Attention (MLA) with KV compression for efficient inference. The model supports both MHA and MQA modes, enabling flexible deployment across different hardware configurations.

Quantization

The model supports FP8 quantization (E4M3 format) with dynamic activation scaling, enabling reduced memory footprint and faster inference on compatible hardware.

---

Benchmarks

<div align="center"> <img src="benchmark.png" alt="PAKNeura-V3 Benchmark Results"> </div>

PAKNeura-V3 demonstrates competitive performance across a comprehensive suite of academic benchmarks covering reasoning, coding, knowledge, and language understanding tasks.

---

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "pakneura/PAKNeura-V3"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt = "Pakistan's AI future is"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Deployment Recommendations

• Full Precision: 4x NVIDIA A100 (80GB) minimum
• FP8 Quantized: 2x NVIDIA A100 (80GB)
• Consumer GPU: 4-bit quantization on RTX 4090 (dual)
• Sampling: temperature = 1.0, top_p = 0.95

---

Chat Template

PAKNeura-V3 supports the user, assistant, and developer roles. Example usage:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("pakneura/PAKNeura-V3")

messages = [
    {"role": "user", "content": "What is the capital of Pakistan?"},
    {"role": "assistant", "content": "The capital of Pakistan is Islamabad.", "reasoning_content": "Islamabad is the capital since 1963..."},
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")

---

Weights Status

---

About PAKNeura

PAKNeura is a Pakistani AI organization dedicated to developing open-weight language models and advancing AI research in Pakistan. Our mission is to make cutting-edge AI technology accessible to the Pakistani community.

Website: www.pakneura.com

---

License

All Rights Reserved. PAKNeura-V3 and associated materials are developed by PAKNeura AI.

---

Citation

If you use PAKNeura-V3 in your research, please cite:

@misc{pakneura2025pakneurav3,
    title={PAKNeura-V3: Pakistan's First Open-Weight Language Model},
    author={PAKNeura AI},
    year={2025},
    publisher={Hugging Face},
    howpublished={https://huggingface.co/pakneura/PAKNeura-V3}
}

---

Contact

• Website: www.pakneura.com
• Hugging Face: pakneura
• Discussions: PAKNeura-V3 Discussions

---

<div align="center"> <strong>Made with ❤️ in Pakistan</strong> </div>