Post Hot Chips 2026: a review of custom chips

In April 2025, when Meta announced it would serve Llama through Cerebras, running in Cerebras’s own data centers, I said then that it made sense for Meta and nobody else (Cerebras). Meta cared about latency enough to pay for an unusual machine. I didn’t think the other big buyers would bother.

Less than a year later, OpenAI signed for 750 megawatts of Cerebras capacity, a deal Cerebras’s own filing values at more than $20 billion (S-1/A). Cerebras went public at $185 a share (IPOScoop). I was wrong about Cerebras, and I wanted to understand why.

The short answer is that I was looking at the wrong numbers. Chips get compared on FLOPs. For serving a language model, two other things matter more: where the model’s bytes live, and whether the customer is paying for speed or for volume.

bandwidth, not FLOPs

Generating text has two phases. Prefill reads your whole prompt at once, which is a lot of arithmetic on data already loaded, so it is limited by compute. Decode produces one token at a time, and every new token needs every weight in the model read from memory again. Decode is limited by memory bandwidth.

That gives a rule of thumb you can do in your head. Tokens per second for one user is roughly memory bandwidth divided by the bytes read per token. A 70-billion-parameter model at 8 bits is 70 GB per token, so a chip that reads 8 TB/s gives that user a little over a hundred tokens per second, however many FLOPs it has:

model

8B dense70B denseMoE 671B / 37B active

precision

BF16 2 BFP8 1 BFP4 ½ B

chip

Groq 3 LPUd-Matrix CorsairTensordyne NapierTenstorrent BlackholeFuriosaAI RNGDGoogle TPU v7 IronwoodGoogle TPU 8iAWS Trainium3Microsoft Maia 200Meta MTIA 450Nvidia GB300Nvidia Rubin

batch

8

per user–tok/s

all users–tok/s

limit–

Rule-of-thumb estimate: ignores KV cache, kernel efficiency and interconnect. Specs from public sources; see chips.json. Peak FLOP/s is each chip’s FP8 figure, halved for BF16 and doubled for FP4, as a simplification. Every user’s KV cache is read on every step too, so real numbers fall further below these at large batches and long contexts.

The bend in that line is the trade-off the rest of this essay is about. Serve one user and the chip spends most of its time waiting on memory. Serve hundreds at once and you use the compute you paid for, but every user gets fewer tokens per second. So I think of a chip as having a curve rather than one speed, and the batch size slides you along it.

The bend has a simple cause. In each decode step the chip reads the weights once for the whole batch, but it reads each user’s cache of their own conversation separately. While memory is the limit:

time per step  ≈  (weight bytes read  +  batch × cache bytes per user)  ÷  bandwidth
speed per user =  1 ÷ time per step
throughput     =  batch ÷ time per step

At small batches the weights dominate, so another user is almost free. As the batch grows the caches take over, and every extra user slows everyone down. Long contexts make each cache larger, which is why the bend comes sooner for a coding agent with a 100,000-token prompt than for a short chat.

The 70B example is also simpler than what is actually served. Frontier open models are mixtures of experts. DeepSeek-V3 has 671 billion parameters but uses 37 billion for each token (DeepSeek), and gpt-oss-120b uses 5.1 billion of its 117 billion (OpenAI). One user’s token reads only the active experts, so the small number sets per-user speed, while the machine must still hold the large one. Batching also helps less than it does for a dense model. Different users route to different experts, so as the batch grows each step touches more of the model, and the weight term creeps from the active size toward the total.

goodput

That is why careful benchmarks don’t stop at peak throughput. They fix a speed every user must get, and ask how much throughput survives above it. The DistServe authors call this goodput (DistServe). A system that serves ten requests a second, of which only three meet the latency target, has a goodput of three.

Drag the line and watch the ranking change:

product

Chatbot (~100)Coding agent (~200)Voice/real-time (~400)

every user gets at least

100 tok/s

data

Modeled (70B, all chips)Published (InferenceX, GPUs)

At 100 tok/s per usercheapest per million tokens first

Show the numbers

H100 · 8× H100, TP8 · modeled

batch	tok/s per user	tok/s per chip	$ per M
1	186	23.3	$16
2	185	46.3	$7.8
4	183	91.3	$4.0
8	178	178	$2.0
16	169	339	$1.1
32	154	618	$0.58
64	131	1,050	$0.34
128	101	1,615	$0.22
256	69	2,209	$0.16
512	42.3	2,707	$0.13

B200 · 8× B200, TP8 · modeled

batch	tok/s per user	tok/s per chip	$ per M
1	315	39.3	$14
2	313	78.3	$6.9
4	310	155	$3.5
8	305	305	$1.8
16	294	588	$0.92
32	275	1,098	$0.49
64	243	1,941	$0.28
128	197	3,150	$0.17
256	143	4,573	$0.12
512	92.3	5,908	$0.092
1024	54	6,918	$0.078

MI355X · 8× MI355X, TP8 · modeled

batch	tok/s per user	tok/s per chip	$ per M
1	296	37	$13
2	295	73.7	$6.4
4	292	146	$3.2
8	287	287	$1.6
16	278	555	$0.85
32	260	1,041	$0.45
64	231	1,851	$0.26
128	189	3,030	$0.16
256	139	4,446	$0.11
512	90.6	5,801	$0.081
1024	53.5	6,844	$0.069

GB200 · 8 GPUs of a GB200 NVL72, TP8 · modeled

batch	tok/s per user	tok/s per chip	$ per M
1	360	45	$15
2	358	89.5	$7.4
4	354	177	$3.8
8	347	347	$1.9
16	333	666	$1.0
32	309	1,234	$0.54
64	269	2,150	$0.31
128	214	3,419	$0.20
256	152	4,851	$0.14
512	95.9	6,135	$0.11
1024	55.2	7,071	$0.094

Cerebras WSE-3 · 2 wafers, pipelined; one wafer counts as one chip · modeled

batch	tok/s per user	tok/s per chip	$ per M
1	2,179	1,089	$3.8
2	2,137	2,137	$1.9
4	2,058	4,116	$1.0
8	1,917	7,666	$0.54

All figures are output tokens. Modeled: 70B dense, FP8, 4k context, decode only; illustrative, so trust the shape rather than the absolute numbers. $/chip-hour are estimates. Published: InferenceX (SemiAnalysis), gpt-oss-120B 8k/1k FP4, fetched 7 Nov 2026, values interpolated by the site; its throughput counts input tokens too, so we divide by 9 (and multiply $/M by 9). InferenceX has no SRAM chips.

The ranking depends on where you draw the line. A chatbot that streams text at reading speed can batch heavily, and the cheapest chip per token wins. A coding agent that has to finish a long edit before you lose interest needs far more tokens per second per user, and some chips simply can’t get there at any batch size.

reading an inference number

There are a few less obvious things I’ve learned to check when someone shows me an inference number. The first is whether the number is an average. Users feel the slowest requests, so the useful figure is the 99th percentile, and averages can hide stalls of several seconds (Etalon). The second is whether long prompts were in the mix. A big prefill can stall everyone else’s generation unless the scheduler breaks it into chunks (Sarathi-Serve).

The third is memory again. Every active conversation keeps a cache of its context, and that cache often fills the chip’s memory before compute runs out, which ends the curve early. The last is speculative decoding, where a small model drafts tokens for the big one to check. It makes a lightly loaded system faster and a heavily loaded one slower. A benchmark that shows only one load level tells you very little.

where the bytes live

Once I thought about chips this way, the landscape started to look less like a race and more like a map. I think each custom chip is a bet on one corner of it:

• On-chip SRAM
• HBM, custom chips
• HBM, Nvidia GPUs (baseline)
• Cheap DRAM (LPDDR, GDDR)
• How many times a second it can read all of that memory

Hover, tap or Tab to a dot for details.

Show the numbers

Chip	Memory	Bandwidth	Reads it all	Status
Cerebras WSE-3	44 GB	21,000 TB/s	480,000×/s	shipping · vendor claim
Groq 3 LPU (Nvidia LPX)	500 MB	150 TB/s	300,000×/s	shipping 2026 · reported
d-Matrix Corsair	2 GB	150 TB/s	75,000×/s	shipping · vendor claim
Nvidia Rubin	288 GB	19.2 TB/s	67×/s	H2 2026 · vendor claim
Meta MTIA 450	288 GB	18.4 TB/s	64×/s	early 2027 · roadmap
Google TPU v7 Ironwood	192 GB	7.38 TB/s	38×/s	shipping · vendor claim
AWS Trainium3	144 GB	4.9 TB/s	34×/s	shipping · vendor claim
Tensordyne Napier	144 GB	4.7 TB/s	33×/s	2027 · roadmap
Microsoft Maia 200	216 GB	7 TB/s	32×/s	shipping · vendor claim
FuriosaAI RNGD	48 GB	1.5 TB/s	31×/s	shipping · vendor claim
Google TPU 8i	288 GB	8.6 TB/s	30×/s	announced · roadmap
SambaNova SN50	64 GB	1.84 TB/s	29×/s	H2 2026 · vendor claim
Nvidia GB300	288 GB	8 TB/s	28×/s	shipping · vendor claim
Tenstorrent Blackhole	32 GB	0.512 TB/s	16×/s	shipping · vendor claim
Positron Asimov	864 GB	3 TB/s	3.5×/s	pre-tape-out · reported

At the top left are the SRAM chips. Groq, whose technology Nvidia licensed in December 2025 in a deal reported at about $20 billion (Groq, CNBC), keeps every weight in on-chip SRAM. Cerebras does the same on a single wafer the size of a dinner plate, with 44 GB of SRAM. These chips can read their entire memory tens or hundreds of thousands of times a second.

d-Matrix takes the idea one step further. Instead of moving weights to a separate compute unit, it puts small multipliers right next to the memory cells, so the weights barely move at all:

The reason this matters is energy. By d-Matrix’s figures, reading a bit from on-chip SRAM costs about 50 femtojoules, while reading it from HBM costs about 2.4 picojoules, roughly fifty times more (Hot Chips 2025). In inference, most of the energy seems to go into moving data rather than arithmetic.

At this year’s Hot Chips, d-Matrix showed where it is going next, and it is a quiet admission of the catch below. Its Raptor design stacks DRAM directly under the compute die instead of relying on SRAM alone, aiming for 32 GB per card at about 100 TB/s and roughly a third of a picojoule per bit, all still projections (ServeTheHome).

The catch with SRAM is capacity. It is fast because it is small and sits next to the compute, which means you need many chips to hold one model:

Model8B70B405B671B

PrecisionBF16 2 BFP8 1 BFP4 0.5 B

1. Positron Asimov
2. Google TPU 8i
3. Meta MTIA 450
4. Nvidia GB300
5. Nvidia Rubin
6. Microsoft Maia 200
7. Google TPU v7 Ironwood
8. AWS Trainium3
9. Tensordyne Napier
10. SambaNova SN50
11. FuriosaAI RNGD
12. Cerebras WSE-3
13. Tenstorrent Blackhole
14. d-Matrix CorsairCounts its SRAM. Its LPDDR tier could hold far more, but reads it far slower.
15. Groq 3 LPU (Nvidia LPX)
16. 1101001,00010,000

Chips needed, log scale →

On-chip SRAM. Weights only. KV cache and activations need more.

A 70B model at 8 bits needs on the order of 140 Groq 3 chips, two Cerebras wafers, or one Rubin GPU. That was my original objection: 44 GB is not much memory for a frontier model, so you end up spreading it across a lot of silicon.

Mixtures of experts sharpen both sides of that objection. At 8 bits, DeepSeek-V3 needs about 671 GB just to sit in memory, which is more than 1,300 Groq 3 chips or 16 Cerebras wafers. But each token reads only about 37 GB of it, so a machine that can hold the whole model serves it very fast.

What I missed is where these chips sit on the goodput curve. As far as I can tell, they are the only ones that reach the far right, thousands of tokens per second for a single user, and some customers will pay a lot for that. For gpt-oss-120b, Cerebras was the most expensive provider Artificial Analysis listed when I checked, at a blended $0.39 per million tokens against $0.04–0.07 on GPU clouds, and also the fastest (Artificial Analysis). People pay that premium. Specialized chips seem to win where someone pays for speed, not where they’re trying to save money.

power

The other number that has quietly taken over is power. New data centers are limited by how many megawatts they can get, not by how many chips they can buy, so the real metric becomes tokens per second per megawatt. This is where a familiar trade-off in chip design shows up.

A chip’s dynamic power is roughly its switched capacitance times voltage squared times clock frequency, and a higher clock needs a higher voltage. Near the top of the range, voltage rises roughly in step with frequency, so power grows with about the cube of the clock while speed grows only linearly. On that approximation, the last 20% of clock speed costs about half the power. I first noticed this in Etched’s pitch about running its transformer chip at low voltage, which I had initially dismissed. Here is the trade-off, with real measurements on top:

clock speed

–

Max performance (100%)Best tokens per joule (~70%)

power–

compute speed–vs 100% clock

compute per watt–vs 100% clock

decode speed–

powercompute speeddecode speedcompute per watt(lines are the model)measured

1. Patel et al., ASPLOS ’24 (Azure, A100). About 7% below max clock costs under 2% of performance. Up to 20% of peak power reclaimed for ≤7% performance loss.
2. Patel et al., Splitwise, ISCA ’24 (H100, Llama-70B). Decode capped from 700 W to 350 W with almost no latency hit. Plotted where this model draws half power.
3. Kakolyris et al., throttLL’eM (A100, Llama2-13B). Most efficient at 74% clock: +37% tokens per joule for 6% less throughput.

Show the model as a table

Illustrative model: dynamic power ≈ C·V²·f plus leakage and a fixed memory floor; constants are not any specific chip. Markers are measured results from the cited papers.

Notice that decode hardly slows down at all. Because it is waiting on memory, a lower clock costs almost nothing. Microsoft’s measurements on Azure found that capping GPU power could save up to 20% of peak power for 7% or less in performance (Patel et al.). In the Splitwise paper, an H100 doing only decode was capped from 700 W to 350 W with almost no added latency (Splitwise). Nvidia now ships a “Max-Q” mode for data centers on the same logic (Nvidia). In a power-capped building, more chips at a lower voltage can beat fewer chips at full speed.

the hyperscalers

The hyperscalers are making the same bet, more quietly, and one ratio shows it: peak compute divided by memory bandwidth. For a dense model at 8 bits, each user’s token does about two operations per byte of weights read, so a chip with a ratio of R operations per byte needs a batch of roughly R/2 before it stops waiting on memory. Nvidia’s Rubin, at about 17.5 FP8 petaflops over 19.2 TB/s, has a ratio near 900, so it wants batches in the hundreds.

Each new inference chip moves that ratio down. Meta’s MTIA 450 doubles memory bandwidth over MTIA 400, to 18.4 TB/s, while FP8 compute grows by about 17%, which takes the ratio from about 650 to about 380 (Meta). In April Google split its eighth-generation TPU in two. The inference chip, 8i, has less peak compute than the training chip, 8t, but about 30% more bandwidth and three times the on-chip SRAM, 384 MB against 128 MB (Google). These are chips built to serve small batches fast, which a training chip isn’t designed to do. Owning them is also leverage against a supplier whose margins are famously high.

the network, and the split

There is another part of the bet that the spec sheet hides: the network. For large mixture-of-experts models, how many chips can act as one machine matters as much as any single chip. Google’s TPU 8t puts 9,600 chips on one inter-chip network, and its data-center fabric joins more than 134,000 of them. Nvidia’s NVLink domain is 72 GPUs per rack today, with 576 planned for Rubin Ultra.

Not everyone thinks specialization ends well. Vincent Zhong’s Disaggregation Trap is not against splitting prefill from decode; large serving systems already do that in software, on identical GPUs that can switch roles in minutes. His objection is to making the split in silicon. A chip that can only do one phase fixes the ratio between them when you buy the hardware, while the workload keeps shifting over a four-to-five-year life. You pay for every pool and get the throughput of the most starved one. Nvidia’s own roadmap shows the tension: it announced a prefill-only GPU, Rubin CPX, appears to have since dropped it, and specialized the decode side through Groq instead (The Register). The power results above point the same way. A power cap on identical GPUs gets much of the benefit of a separate decode chip, without locking in the ratio.

The split is also moving deeper. Some systems now divide decode itself: attention, which reads each user’s own memory and gains nothing from batching, stays on memory-rich chips, while the expert layers, whose weights every user shares, run on a separate pool (MegaScale-Infer, Step-3). It is the same question as the rest of this essay, one layer down: which bytes belong to one user, and which are shared.

the stranger bets

The remaining bets are stranger. Tensordyne, formerly Recogni, does its arithmetic in a logarithmic number system, which turns multiplications into additions (The Next Platform). As far as I can find, it hasn’t published accuracy data yet. Etched hard-wires the transformer into silicon (The Register). Both are chasing a moving target, because the GPU keeps getting the same benefit from cheaper number formats, first FP8 and now FP4.

what I ask now

The part I’m least sure about is whether the models will hold still long enough for any of these bets to pay off. Diffusion language models, for example, generate a whole block of text at once instead of one token at a time, which pushes decode back toward being limited by compute (Mercury). If that kind of model wins, a chip built around memory bandwidth looks less clever.

So I’ve stopped asking which chip is fastest. I ask what speed its buyer is paying for, where its bytes live, and how much power it burns to get there. If you have a better answer than mine, especially from running one of these in production, I’d like to hear it.