Vedika-4.1-flash

Vedika 4.1 Flash

Developed by: Veda Labs
Hugging Face Organization: Veda-Labs
GitHub: vedalabs-tech
X (Twitter): @VedaLabsAI
Contact Email: [email protected]

[!Note] This model utilizes a custom architecture. You must use trust_remote_code=True when loading the model.

from transformers import AutoModelForCausalLM, AutoConfig

config = AutoConfig.from_pretrained("Veda-Labs/vedika-4.1-flash", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Veda-Labs/vedika-4.1-flash", trust_remote_code=True)

[!Note] This repository contains model weights and configuration files for the post-trained model in the Hugging Face Transformers format.

These artifacts are compatible with Hugging Face Transformers, vLLM, SGLang, TokenSpeed, etc.

As the frontier of foundation models pushes toward ever-larger parameter counts and ever-longer context windows, the question is no longer just how much we can scale, but how efficiently we can do so. Sustainable progress toward artificial general intelligence (AGI) that benefits everyone demands architectural innovation. Today, we are sharing a concrete step in that direction: Vedika 4.1 Flash.

This experimental preview of the architecture that will underpin our next generation is built around a fundamental rethinking of how the core components of modern large language models (LLMs) interact at scale.

Highlights

The first open-weight release under this architecture is Vedika 4.1 Flash, which introduces:

• Hybrid Attention with Vedika Sparse Attention: The pairing has been reworked to operate at the micro-block level. This cuts long-context latency significantly, a critical gain as agentic workloads increasingly dominate real-world usage.
• Gated Residual: Residual streams with normalization are what make deep LLM training manageable. Gated Residual modulates information flowing through widened residual streams via an element-wise, data-dependent read gate and a per-branch scalar write gate. This brings finer-grained expressiveness across layers while preserving training stability and keeping inference overhead low.
• N-gram Embedding: Embeddings provide a unique axis for parameter scaling that requires less computation and is more amenable to offloading than Mixture-of-Experts (MoE). By indexing with short n-grams, this approach makes parameter scaling highly efficient for memory-constrained accelerators without sacrificing quality.
• Tailored Training Recipe: The optimizers are applied to specific weight categories to maximize efficiency. Guided by refitted scaling laws, we eliminate traditional batch-size warmups and start directly at the target batch size, substantially reducing total optimizer steps while safely supporting larger learning rates for robust convergence.

We are excited to embark on this next chapter with you and welcome your feedback as we build what comes next.

Model Overview

• Type: Causal Language Model with Vision Encoder
• Training Stage: Pre-training & Post-training
• Language Model
  • Number of Parameters: 125B with 6B activated, plus 51B n-gram embedding and 4B MTP
  • Hidden Dimension: 2560
  • Token Embedding: 248320 (Padded)
  • N-gram Embedding: 20,000,000 (bigrams/trigrams at layer 2)
  • Number of Layers: 48
  • Hidden Layout: 12 × (3 × (Gated DeltaNet → MoE) → 1 × (Vedika Sparse Attention → MoE))
  • Gated DeltaNet:
    • Number of Linear Attention Heads: 48 for V and 16 for QK
    • Head Dimension: 128
  • Vedika Sparse Attention:
    • Number of Attention Heads: 24 for Q and 2 for KV
    • Head Dimension: 256
    • Rotary Position Embedding Dimension: 64
    • Indexer Structure: MQA with 4 Query Heads and 1 Shared Key Head
    • Indexer Head Dimension: 128
    • Budget: 512 blocks or 2048 tokens
  • Mixture Of Experts
    • Number of Experts: 512
    • Number of Activated Experts: 10 Routed + 1 Shared
    • Expert Intermediate Dimension: 640
  • Gated Residual:
    • Number of Branches: 4
    • Bottleneck Rank: 320
  • LM Output: 248320 (Padded)
  • MTP: 1 layer, trained with multi-steps
• Context Length: 262,144 natively and extensible up to 1,000,000 tokens.

Benchmark Results

See official Veda Labs evaluation pages for detailed multimodal, instruction following, and reasoning metrics.

Serving Vedika 4.1 Flash

Vedika 4.1 Flash can be deployed with popular inference frameworks.

Vedika 4.1 Flash models operate in thinking mode by default, generating thinking content signified by <think>\n...</think>\n\n before producing the final responses.

Vedika 4.1 Flash supports controlling thinking behavior via enable_thinking, preserve_thinking, and reasoning_effort.