kimi-k3-article-svg-preview

HFVwr/kimi-k3-article-svg-preview

kimi-k3-article-svg-preview is a machine learning model from HFVwr. Use it for the machine learning task on the model card, and read the license before you ship it in a product.

This Hugging Face edition adapts hfviewer’s original interactive Kimi K3 preview.

Downloads · 30 days

Access

Public

Updated Jul 27, 2026

Repo size

3.6 MB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.png2.6 MB · 80%

At a glance

Access: Public
Created: Jul 18, 2026
Updated: Jul 27, 2026
SHA: aae60782

Model card

From the Hugging Face model README

Kimi K3, previewed: inside the first open 3T-class model

This Hugging Face edition adapts hfviewer’s original interactive Kimi K3 preview.

Moonshot AI announced Kimi K3 on July 16: a 2.8-trillion-parameter open model with a 1-million-token context window, native vision, and two architectural bets that go beyond scaling up the usual recipe. The weights land on Hugging Face by July 27. This page is our deep-dive, prepared in advance: the benchmarks and the architecture story now, the traced graph the moment the checkpoint drops.

Announced release time: July 27, 2026 at 00:00 UTC.

The architecture: four ideas carry it

K3 is not a scaled-up K2. Moonshot credits the jump to roughly 2.5× better scaling efficiency than Kimi K2, and the launch post names the mechanisms plainly. Each one lands on a trend line we have been tracking across the 2,400+ models traced on hfviewer.

Kimi Delta Attention · the workhorse

KDA is Moonshot’s delta-rule take on linear attention: short convolutions, L2-normalized keys, and gated state updates instead of a quadratic attention matrix. Three of every four attention layers in K3 are KDA. That is the industry’s loudest trend right now: linear-attention layers appear in 29% of new models in our catalog this quarter, up from near zero a year ago.

Gated MLA · the precise quarter

Every fourth attention layer is full multi-head latent attention with output gating, keeping global, lossless retrieval in the stack while KDA does the bulk work. MLA itself just had its breakout quarter: 23% of new models now ship it, and with K3, Moonshot pairs it with linear attention rather than choosing between them.

Attention Residuals · the new one

AttnRes is the genuinely novel piece: instead of every block adding uniformly onto one residual stream, blocks selectively retrieve representations from earlier depths (the α operators in Moonshot’s diagram). Think of it as attention over depth, not just sequence. No open model we have traced does this; it will be a new shape on the graph.

Stable LatentMoE · the sparsity bet

Feed-forward blocks route top-16 of 896 experts, the largest expert pool ever shipped in the open. Balancing at that sparsity is its own research problem: Moonshot’s Quantile Balancing derives allocation from router-score quantiles instead of an auxiliary loss, alongside a SiTU activation and Per-Head Muon optimizer. See MoE routing in the glossary for the family this extends.

Left to right: the 3:1 KDA-to-MLA cycle every token flows through · the expert pool one LatentMoE routes over · the α operators retrieving from earlier depths.

The sparsity race, on one chart

Every dot below comes from a graph traced from real weights on hfviewer; K3 is plotted from the launch post. The direction is unmistakable: expert pools keep growing while the active fraction falls. K3 pushes the pool to 896 but activates 16, more than recent frontier MoEs, landing at 1.8% active.

Active share counts routed experts only (shared always-on experts excluded). Sources: traced configs on hfviewer; Kimi K3 from Moonshot’s launch post.

Benchmarks

Moonshot’s own framing is unusually candid: K3 “still trails the most powerful proprietary models” overall while leading everything else. The numbers back that up. Across the 33 public rows in the launch table, K3 is first or tied-first on 8, ahead of every other open model on essentially all of them, and usually within a few points of the frontier. Standouts: SWE Marathon (42.0, two points clear of the field and triple the GPT 5.5 score), BrowseComp (91.2), DeepSearchQA (95.0), and OmniDocBench (91.1).

All K3 scores at max reasoning effort. Harnesses differ per benchmark (KimiCode, Claude Code, Codex) and several third-party scores are cited from external leaderboards or release blogs; Claude Fable 5 was evaluated by a third party and may include fallback behavior. Rows marked cited mix first-party and cited numbers; internal rows are Moonshot-internal. Elo rows use a different scale. Full footnotes in the launch post.

What it will look like on hfviewer

Until July 27, the closest traced relative is Kimi K2.7-Code: an MLA MoE decoder routing top-8 of 384 experts plus one always-on shared expert. Explore it below, then picture the K3 diff: three quarters of the attention layers turning into KDA blocks with their convolutions and gates, the expert pool growing to top-16 of 896, and a web of AttnRes edges that no current graph has.