Helion-V1.5-XL

Helion-V1.5-XL

<div align="center"> <img src="https://imgur.com/aUIJXf7.png" alt="Helion-V1 Logo" width="100%"/> </div>

---

Model Overview

Helion-V1.5-XL is a 16.2 billion parameter large language model designed for advanced natural language understanding and generation tasks. Built upon the foundation of Helion-V1.5, this XL variant incorporates architectural improvements, expanded training data, and enhanced optimization techniques to deliver superior performance across diverse benchmarks.

The model employs a decoder-only transformer architecture with Grouped Query Attention (GQA), RoPE positional encodings, and SwiGLU activations. Training utilized 4.5 trillion tokens from curated high-quality sources spanning web text, scientific literature, code repositories, and instruction-following datasets.

Architecture Specifications

Model Type:              Decoder-Only Transformer
Total Parameters:        16,247,832,576
Trainable Parameters:    16,247,832,576
Non-trainable Parameters: 0

Layers:                  48
Attention Heads:         32 (Query)
Key-Value Heads:         8 (GQA)
Hidden Dimension:        6144
Intermediate Dimension:  24576
Head Dimension:          192

Vocabulary Size:         100,000
Maximum Context Length:  16,384 tokens
RoPE Theta:             10,000.0
RoPE Scaling:           Linear (factor: 2.0)

Activation Function:     SwiGLU
Normalization:          RMSNorm (eps: 1e-6)
Attention Mechanism:    Grouped Query Attention
Positional Encoding:    Rotary Position Embedding
Flash Attention:        Enabled (v2)

Precision:              bfloat16

Performance Benchmarks

Language Understanding

Reasoning and Common Sense

Code Generation and Understanding

Multilingual Performance

Truthfulness and Safety

Long Context Understanding

Training Methodology

Dataset Composition

The training corpus consists of 4.5 trillion tokens sampled from the following sources:

Training Infrastructure

Compute Resources:        512x NVIDIA A100 80GB GPUs
Total Training Time:      672 hours (28 days)
Framework:               PyTorch 2.0.1 with FSDP
Distributed Strategy:     Fully Sharded Data Parallel (FSDP)
Mixed Precision:         bfloat16 with stochastic rounding
Communication Backend:    NCCL with InfiniBand

Total FLOPs:             ~8.2e24 FLOPs
GPU Hours:               ~344,064 GPU-hours
Peak Memory per GPU:     72GB
Interconnect Bandwidth:  400 Gbps per GPU

Optimization Configuration

Optimizer:               AdamW
Beta1:                   0.9
Beta2:                   0.95
Epsilon:                 1e-8
Weight Decay:            0.1
Gradient Clipping:       1.0

Learning Rate Schedule:  Cosine with Warmup
Peak Learning Rate:      3.0e-4
Minimum Learning Rate:   3.0e-5
Warmup Steps:            2,000
Total Training Steps:    875,000

Batch Configuration:
  Global Batch Size:     4,194,304 tokens
  Micro Batch Size:      32 samples
  Gradient Accumulation: 8 steps
  Sequence Length:       4,096 tokens

Checkpointing:
  Activation Checkpointing: Enabled
  Checkpoint Interval:      5,000 steps
  Total Checkpoints Saved:  175

Training Stages

Stage 1: Pre-training (3.8T tokens)

• Duration: 750,000 steps
• Objective: Next-token prediction
• Data: General corpus (web, books, code, scientific)
• Learning Rate: Full cosine schedule

Stage 2: Domain Adaptation (500B tokens)

• Duration: 80,000 steps
• Objective: Continued pre-training on specialized domains
• Data: Enhanced code, mathematics, scientific reasoning
• Learning Rate: 1.0e-4 constant

Stage 3: Instruction Tuning (200B tokens)

• Duration: 45,000 steps
• Objective: Instruction following and task alignment
• Data: High-quality instruction-response pairs
• Learning Rate: 5.0e-5 with linear decay

Installation and Usage

Requirements

pip install torch>=2.0.0 transformers>=4.35.0 accelerate>=0.24.0

Basic Inference

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "DeepXR/Helion-V1.5-XL"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

prompt = "Explain the concept of quantum entanglement:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.9,
    do_sample=True
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

4-bit Quantization

from transformers import BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4"
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=quantization_config,
    device_map="auto"
)

Chat Format

conversation = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What are the implications of the P vs NP problem?"}
]

prompt = tokenizer.apply_chat_template(conversation, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)

Hardware Requirements

Memory Requirements (Inference)

Inference Performance

Limitations and Biases

Known Limitations

1. Knowledge Cutoff: Training data extends through January 2024. The model lacks awareness of subsequent events.

2. Hallucination: The model may generate plausible but factually incorrect information with high confidence.

3. Arithmetic Precision: While improved over baseline, complex multi-step mathematical computations may contain errors.

4. Context Length Degradation: Performance decreases beyond 12,000 tokens despite 16,384 token capacity.

5. Specialized Domain Knowledge: May lack depth in highly specialized technical, medical, or legal domains.

6. Code Execution: Generated code requires validation and testing before deployment.

Bias Analysis

The model has been evaluated for biases across multiple dimensions:

• Gender Bias: BOLD gender bias score of 0.34 (lower is better)
• Racial Bias: Demonstrates residual stereotypical associations in certain contexts
• Geographic Bias: Western-centric knowledge distribution
• Language Bias: Performance degrades for lower-resource languages

Mitigation strategies include balanced dataset sampling, bias-aware fine-tuning, and constitutional AI principles during alignment.

Evaluation Methodology

All benchmarks were evaluated using the Language Model Evaluation Harness (lm-evaluation-harness) with standardized few-shot settings. Code evaluation used the standard HumanEval and MBPP test suites with temperature 0.2 sampling. Multilingual benchmarks employed zero-shot evaluation for consistency.

License

This model is released under the Apache License 2.0.

Copyright 2025 DeepXR

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Citation

@misc{helion-v15-xl-2024,
  title={Helion-V1.5-XL: A 16B Parameter Instruction-Tuned Language Model},
  author={DeepXR Team},
  year={2025},
  publisher={HuggingFace},
  url={https://huggingface.co/DeepXR/Helion-V1.5-XL}
}

Acknowledgments

Training infrastructure provided by advanced cloud computing resources. Dataset curation benefited from open-source contributions including The Pile, RedPajama, and community-curated instruction datasets.