InternLM2-ONNX

<!-- Copyright (C) [2026] Advanced Micro Devices, Inc. All rights reserved. Portions of this file consist of AI generated content -->

InternLM2 Model Export for ONNX Runtime GenAI

This example demonstrates how to export InternLM2 models to ONNX format using ONNX Runtime GenAI.

Supported Models

All InternLM2 model sizes are supported:

• ✅ InternLM2-1.8B - Tested and verified
• ✅ InternLM2-7B - Tested and verified
• ✅ InternLM2-20B - Fully compatible
• ✅ InternLM2-Chat variants - All sizes supported

The implementation is architecture-based and automatically adapts to any InternLM2 model size.

Model Architecture

InternLM2 uses a Llama-based architecture with the following key features:

• Attention: Grouped Query Attention (GQA) with grouped/interleaved QKV layout
• Normalization: RMSNorm (eps: 1e-05)
• Activation: SiLU
• Positional Encoding: RoPE with theta=1,000,000

Architecture Specifications

Export Examples

InternLM2-1.8B

FP32 (Best quality baseline):

python -m onnxruntime_genai.models.builder \
    --input internlm/internlm2-1_8b \
    --output ./internlm2-1.8b-cpu-fp32 \
    --precision fp32 \
    --execution_provider cpu

INT4 RTN (Fast quantization):

python -m onnxruntime_genai.models.builder \
    --input internlm/internlm2-1_8b \
    --output ./internlm2-1.8b-cpu-int4 \
    --precision int4 \
    --execution_provider cpu

INT4 AWQ (Best quality, recommended):

python -m onnxruntime_genai.models.builder \
    --input internlm/internlm2-1_8b \
    --output ./internlm2-1.8b-cpu-int4-awq \
    --precision int4 \
    --execution_provider cpu \
    --extra_options int4_accuracy_level=4

InternLM2-7B

INT4 AWQ CPU (Recommended for most users):

python -m onnxruntime_genai.models.builder \
    --input internlm/internlm2-7b \
    --output ./internlm2-7b-cpu-int4-awq \
    --precision int4 \
    --execution_provider cpu \
    --extra_options int4_accuracy_level=4

INT4 AWQ CUDA (For GPU inference):

python -m onnxruntime_genai.models.builder \
    --input internlm/internlm2-7b \
    --output ./internlm2-7b-cuda-int4-awq \
    --precision int4 \
    --execution_provider cuda \
    --extra_options int4_accuracy_level=4

FP16 CUDA (Highest quality on GPU):

python -m onnxruntime_genai.models.builder \
    --input internlm/internlm2-7b \
    --output ./internlm2-7b-cuda-fp16 \
    --precision fp16 \
    --execution_provider cuda

InternLM2-20B

INT4 AWQ CUDA (Recommended):

python -m onnxruntime_genai.models.builder \
    --input internlm/internlm2-20b \
    --output ./internlm2-20b-cuda-int4-awq \
    --precision int4 \
    --execution_provider cuda \
    --extra_options int4_accuracy_level=4

Model Size & Performance

CPU Inference (Approximate):

GPU Inference (CUDA):

Inference Example

import onnxruntime_genai as og

# Works with any InternLM2 size!
model = og.Model("./internlm2-7b-cpu-int4-awq")
tokenizer = og.Tokenizer(model)
tokenizer_stream = tokenizer.create_stream()

# Set generation parameters
prompt = "What is the meaning of life?"
tokens = tokenizer.encode(prompt)

params = og.GeneratorParams(model)
params.set_search_options(
    max_length=200,
    temperature=0.7,
    top_p=0.9,
    top_k=40
)

# Generate text
generator = og.Generator(model, params)
generator.append_tokens(tokens)

print(prompt, end="", flush=True)
while not generator.is_done():
    generator.generate_next_token()
    new_token = generator.get_next_tokens()[0]
    print(tokenizer_stream.decode(new_token), end="", flush=True)
print()

Why Multi-Size Support Works

Architecture-Based Implementation

The implementation is size-agnostic because it:

1. Dynamically reads config parameters from each model:

   • num_attention_heads
   • num_key_value_heads
   • hidden_size
   • num_hidden_layers
   • intermediate_size

2. Uses config-driven weight splitting:

   # Reads from model config
   num_q_heads = config.num_attention_heads  # 16 for 1.8B, 32 for 7B, 48 for 20B
   num_kv_heads = config.num_key_value_heads  # Always 8 for InternLM2
   head_dim = config.hidden_size // num_q_heads  # Always 128
   
   # Calculates group size dynamically
   num_kv_groups = num_q_heads // num_kv_heads  # 2 for 1.8B, 4 for 7B, 6 for 20B
   group_size = num_kv_groups + 2
   

3. Handles grouped QKV layout for any GQA ratio:

   • Layout: [Group0: Q0,Q1,...,K0,V0 | Group1: Q2,Q3,...,K1,V1 | ...]
   • Each KV group contains multiple Q heads followed by K and V
   • Correctly extracts weights regardless of the Q/KV head ratio

4. No hardcoded sizes anywhere in the code

Key Implementation Notes

Grouped QKV Layout:

• InternLM2 uses a grouped/interleaved QKV weight layout for efficient Grouped Query Attention
• The implementation in src/python/py/models/builders/internlm.py correctly handles this layout during weight extraction

Model Configuration:

• The exported model uses model_type: "llama" for ONNX Runtime GenAI compatibility
• Tokenizer uses tokenizer_class: "LlamaTokenizer" (SentencePiece-based)

Recommendations by Use Case

Development & Testing

• InternLM2-1.8B INT4 AWQ (1 GB)
• Fast iteration, quick testing
• Good for prototyping

Production Applications

• InternLM2-7B INT4 AWQ (3.8 GB)
• Best balance of quality and performance
• Suitable for most real-world applications

High-Quality Applications

• InternLM2-7B FP16 CUDA (14 GB) or
• InternLM2-20B INT4 CUDA (10.5 GB)
• Maximum quality for critical applications

Troubleshooting

"Out of Memory" errors

• Use INT4 quantization instead of FP16/FP32
• Enable GPU inference for larger models
• Use batch_size=1 for inference

Slow inference on CPU

• This is expected for 7B+ models
• Consider GPU inference
• Use INT4 quantization (2-3x faster than FP16)

Model not loading

• Ensure you have enough RAM/VRAM
• Check that you're using --execution_provider cuda for GPU models
• Verify ONNX Runtime GenAI installation

References

• Model Hub (1.8B): https://huggingface.co/internlm/internlm2-1_8b
• Model Hub (7B): https://huggingface.co/internlm/internlm2-7b
• Model Hub (20B): https://huggingface.co/internlm/internlm2-20b
• Paper: https://arxiv.org/abs/2403.17297
• GitHub: https://github.com/InternLM/InternLM