The Trick That Makes Kimi-K3’s 2.78 Trillion Parameters Almost Free

Everyone’s counting Kimi-K3’s parameters. We found the move that makes them cheap — and we never downloaded the model.

When Moonshot released Kimi-K3 in June 2026, the takes wrote themselves: a giant multimodal Mixture-of-Experts, trillions of parameters, a million-token context, shipped in 4-bit. All true. All also the kind of thing you can read off a model card.

Here’s what the model card won’t tell you — and what almost no one will cover, because you have to read the weight layout and the modeling code to see it:

Kimi-K3’s routed experts don’t run on the model’s hidden state at all. They run in a compressed latent space that’s half as wide. It’s the same low-rank trick DeepSeek made famous for attention — now applied to the Mixture-of-Experts. That one design choice is why a 2.78-trillion-parameter model only “spends” ~104 billion parameters on any given token.

And we found all of it without downloading the model.

First, a number nobody else will get exactly right

Kimi-K3 ships as a ~1.56 TB checkpoint — and that’s already the 4-bit size, not a higher-precision one you could shrink. That’s the first reason our angle is different: almost no independent analyst will actually run this model — 1.56 TB overflows even an 8-GPU node (8×B200 ≈ 1.44 TB). So the parameter counts you’ll see quoted are going to be round guesses (“~3T”).

We don’t guess. A safetensors checkpoint stores a small JSON header at the front of every shard describing every tensor’s exact shape and dtype. You can fetch just those headers with HTTP range requests — a few megabytes across all 96 shards — and count precisely. No 1.56 TB download, no GPU.

The exact number: 2,779.9 billion parameters — 2.78 T, not 3. And the split is its own surprise:

• 2,722.7 B live in the MoE experts (stored MXFP4, native 4-bit).
• 57.2 B are everything else — attention, embeddings, the dense first layer, and the shared experts.
• The vision encoder is only 0.4 B. Kimi-K3 is a text-giant wearing a light vision jacket, not a balanced multimodal model.

And the arithmetic reconciles, which is worth pausing on: the MXFP4 experts pack to ~1.36 TB, the BF16 remainder to ~0.11 TB, plus the quantization scales — ~1.56 TB in total. A 2.78-trillion-parameter model that fits in 1.56 TB on disk is exactly what shipping natively in 4-bit buys you.

That’s the ModelDNA way of working: you don’t need to possess a model to understand it. The metadata is public; you just have to read it properly.

Now, the move everyone will miss

Mixture-of-Experts is old news: you have many experts, you activate a few per token, you get a big model at a small compute bill. Kimi-K3 has 896 experts and activates 16 (plus 2 always-on shared experts). Sparse, sure. Everyone will say “sparse.”

But look at the actual expert dimensions in the weights, and something doesn’t line up. The model’s hidden state is 7168-dimensional. The routed experts? They take a 3584-dimensional input. Half as wide.

How does a 7168-wide residual stream feed a 3584-wide expert? It doesn’t — directly. Reading the modeling code (KimiSparseMoeBlock), the mechanism is unmistakable — a compress → compute → expand sandwich:

1. A shared down-projection compresses each token 7168 → 3584 before routing.
2. The token is routed to 16 of 896 experts, each a small feed-forward network living entirely in that compressed 3584-dim space.
3. A shared up-projection expands the result 3584 → 7168 back onto the residual stream.
4. The shared experts stay full-width on the 7168 stream and are added on top.

If that shape sounds familiar, it should. It’s Multi-head Latent Attention (MLA) — the low-rank compression DeepSeek used to shrink the attention KV cache — applied to the feed-forward / expert bank instead. Compress into a latent space, do the expensive many-way work there cheaply, expand back.

We call it what the code calls it: a latent MoE.

Why it’s clever, in one sentence

It lets Moonshot keep an enormous pool of specialized capacity while paying almost nothing to use it.

Each routed expert is tiny — about 33 million parameters, because it works in the compressed 3584 space rather than the full 7168. Stack 896 of them across 92 layers and you get 2.7 T of routed capacity. Activate only 16 per token, each cheap, and the bill stays small. The design cleanly separates two kinds of capacity:

• Always-on capacity — the shared experts — stays full-width (they carry the general knowledge every token needs).
• Specialized capacity — the routed experts — is compressed (each is a narrow specialist you only occasionally consult).

The result, computed exactly from the weights: 2.78 T total, 104 B active per token — a 26.7× ratio of stored knowledge to per-token compute. That is an aggressively sparse model, and the latent-MoE trick is the reason the sparsity is nearly free instead of merely large.

The through-line worth watching

Here’s why this matters beyond one model.

DeepSeek introduced low-rank latent compression to make attention cheap (MLA). It spread across the frontier labs fast. Kimi-K3 is an early signal that the same idea may be migrating to a second target — the feed-forward / expert bank — in a shipped, trillion-parameter production model. That’s a data point, not yet a wave: two instances (MLA, then latent MoE) hint that latent compression is becoming a general-purpose lever rather than a one-off attention hack — a hypothesis we’ll confirm or kill as more models land.

That’s the kind of pattern you only see if you’re reading architectures at the tensor-and-code level across the whole field, model after model. A benchmark table won’t show it. A model card won’t mention it. It lives in the shapes.

The point

Kimi-K3 is too big for almost anyone to run — and you still don’t have to, to know what it is. Its exact size, its quantization, its attention (a linear + latent-attention hybrid), and its cleverest move — the latent MoE — are all legible from public metadata and modeling code, if you know how to read them.

That’s what ModelDNA does: it reads the DNA of a model — config, weight layout, and code — and tells you what it actually is, before you ever download a byte.

Kimi-K3 architecture facts in this post were derived from the public config.json, safetensors shard headers, and the released modeling code. Exact parameter and activation counts were computed from tensor shapes; the latent-MoE mechanism was confirmed against the modeling source.