When the person who built Google's AI infrastructure for two decades explains what is wrong with AI hardware, the industry listens. In a Y Combinator interview published this July, Google Chief Scientist Jeff Dean reduced the entire economics of AI inference to a single number — and it is the number PhantaField's architecture was designed to delete.

The penalty that invented batching

Dean's accounting is simple. A single multiplication costs about one picojoule. Moving the data that feeds it — from an accelerator's High-Bandwidth Memory across the package to the processor — costs roughly a thousand times more. Every architectural contortion of modern AI serving descends from that one ratio.

If you didn't have that factor of a thousand, you wouldn't need to do batching; but because you do, you have to process many examples or many tokens at once to amortize the cost of that data movement.

Jeff Dean, Y Combinator interview, July 2026

Batching is not a feature. It is a coping mechanism — and it is why interactive AI feels the way it does. Piling dozens of users onto one chip amortizes the data-movement penalty, but every user waits behind the batch. Dean went further: cut inference latency by ~50×, he argued, and entirely new classes of applications become possible — and ‘high-performance, low-energy inference hardware’ is where he expects the next battleground, because ‘inference is the key to making agent-based systems accessible to more people.’ Specialized silicon, he noted, is how you beat general-purpose GPUs and TPUs on both energy and latency.

Deleting the factor instead of amortizing it

PhantaField's Sophon starts from the position that the 1,000× factor is not a law of physics — it is a consequence of putting memory on the far side of a package. Sophon grows 330 GB of 2T0C DRAM inside the compute array, 32 memory tiers interleaved with 32 logic tiers, so every 256×256 MAC tile reads its weights through a private vertical via a few tenths of a micron long. The weight read costs 0.240 pJ against a 0.070 pJ MAC — a factor of about three, not a thousand.

Energy to deliver one 8-bit weight versus the multiply it feeds Log-scale bars: an HBM4 GPU spends roughly 40 to 80 picojoules moving one weight across the package, a 3 nanometer SRAM accelerator about 2 picojoules across millimeter wires, and Sophon about 0.31 picojoules through a 14 micron vertical hop. A dashed line marks the multiply itself at about 0.06 picojoules. Delivering one 8-bit weight + doing the math, per MAC at batch 1 0.01 pJ 0.1 pJ 1 pJ 10 pJ 100 pJ the multiply itself (≈0.06 pJ) 28 nm Sophon in-array DRAM · ~14 µm hop ≈0.31 pJ · ~3× the math 3 nm SRAM accelerator weights in on-die SRAM · mm wires ≈2 pJ · ~40× HBM4 GPU across the package · 5–10 pJ/bit ~1,000× the math — Dean's factor ≈40–80 pJ
Energy to deliver one 8-bit weight and perform the multiply-accumulate, per MAC at batch 1 (representative, order-of-magnitude). Everything to the right of the dashed line is data movement. HBM4's package crossing at ~5–10 pJ/bit is Jeff Dean's ~1,000× factor; a 3 nm SRAM accelerator's millimeter-scale wires still pay ~40×, because wires don't shrink like transistors; Sophon's 14 µm vertical hop from the DRAM tier above pays ~3×. The multiply itself varies less than ~2× across all three — 3 nm's smaller capacitance is largely offset by its higher voltage (energy is C·V²); the bars differ almost entirely in data movement.

A fair question about that chart: surely a 3 nm multiply is cheaper than a 28 nm one? Barely. Arithmetic energy is capacitance times voltage squared, and the two knobs pull in opposite directions — a 3 nm gate switches roughly 4.5× less capacitance, but it holds ~0.85 V to sustain multi-GHz clocks, while Sophon's 500 MHz arrays run at 0.6 V, and (0.85/0.6)² ≈ 2× hands most of the node's advantage straight back. Add bit-serial arithmetic that spends nothing on instruction fetch or unused precision modes, and the multiply lands within about 2× across all three architectures — roughly 0.05–0.10 pJ — while the data movement spans ~250×. The node race optimizes the one line of the ledger that stopped mattering.

Remove the factor and Dean's own logic runs in reverse: there is nothing left for batching to amortize. Sophon is compute-bound from the first stream — 14,438 tokens per second on an 80B-parameter model at batch one, FP8 — where an HBM4-bound GPU delivers roughly 300. That is a ~48× single-stream gap: almost exactly the 50× latency threshold Dean said would redraw the product boundaries of AI. And the domain where it lands — chat, agentic tool loops, long-context reasoning at batch 1–8 — is precisely the agent-based future Dean says inference hardware must serve.

Low precision, built into the arithmetic

Dean's other prescription — experiment with very low precision and build it into the hardware rather than supporting redundant options — describes Sophon's arithmetic literally. Computation is bit-serial: activations stream one bit at a time, so precision is the number of cycles, not a mode. FP8 takes eight cycles, BF16 sixteen, INT4 four — halving precision doubles throughput by construction, with no dark silicon spent on formats nobody uses.

References

Jeff Dean, ‘The 1% Rule for Building in AI.’ Y Combinator Startup Library, July 2026. ycombinator.com/library/Vy-jeff-dean-the-1-rule-for-building-in-ai

‘Google Chief Scientist Jeff Dean: Slashing AI Inference Latency by 50x Would Unlock Entirely New Applications, Specialized Hardware Is the Next Battleground.’ BigGo News, July 2026.

PhantaField, ‘PFG-1 Sophon Whitepaper,’ Rev 4.1 — energy model §3.C, batch-size analysis §5.A.5c. phantafield.com/whitepaper