When the person who built Google's AI infrastructure for two decades explains what is wrong with AI hardware, the industry listens. In a Y Combinator interview published this July, Google Chief Scientist Jeff Dean reduced the entire economics of AI inference to a single number — and it is the number PhantaField's architecture was designed to delete.

The penalty that invented batching

The accounting is simple. As interviewer Diana Hu recapped it, a calculation costs about one picojoule, and moving the data costs a thousand times that. Dean agreed, and said where the cost comes from: just getting the data from an accelerator's High-Bandwidth Memory into the processor that computes on it. Every architectural contortion of modern AI serving descends from that one ratio.

If you didn't have that 1,000× difference, then you wouldn't have to do batching.

Jeff Dean, interviewed by Diana Hu at Y Combinator, July 2026

Batching is not a feature. It is a coping mechanism — and it is why interactive AI feels the way it does. You batch many examples or many tokens at once, Dean explained, to amortize the data movement: the thousandfold cost is divided by the batch size. But every user waits behind the batch, and for really low latency, he said, batching is not very good. He expects more and more high-performance, low-energy inference hardware, because inference is what will bring agent-based systems to more and more people. Specializing the hardware, he noted, is a key way to beat general-purpose GPUs and TPUs on both energy and latency. Imagine, he said, what you could do with latency 50× better.

Deleting the factor instead of amortizing it

PhantaField's Sophon starts from the position that the 1,000× factor is not a law of physics — it is a consequence of putting memory on the far side of a package. Sophon grows 330 GB of 2T0C DRAM inside the compute array, 32 memory tiers interleaved with 32 logic tiers, so every 256×256 MAC tile reads its weights through a private vertical via a few tenths of a micron long. The weight read costs 0.240 pJ against a 0.070 pJ MAC — a factor of about three, not a thousand.

Energy to deliver one 8-bit weight versus the multiply it feeds Log-scale bars: an HBM4 GPU spends roughly 40 to 80 picojoules moving one weight across the package, a 3 nanometer SRAM accelerator about 2 picojoules across millimeter wires, and Sophon about 0.31 picojoules, reading each weight from the memory tier a third of a micron above along a 14 micron line inside the tile. A dashed line marks the multiply itself at about 0.06 picojoules. Delivering one 8-bit weight + doing the math, per MAC at batch 1 0.01 pJ 0.1 pJ 1 pJ 10 pJ 100 pJ the multiply itself (≈0.06 pJ) 28 nm Sophon in-array DRAM · ~14 µm read line ≈0.31 pJ · ~3× the math 3 nm SRAM accelerator weights in on-die SRAM · mm wires ≈2 pJ · ~40× HBM4 GPU across the package · 5–10 pJ/bit ~1,000× the math — Dean's factor ≈40–80 pJ
Energy to deliver one 8-bit weight and perform the multiply-accumulate, per MAC at batch 1 (representative, order-of-magnitude). Everything to the right of the dashed line is data movement. HBM4's package crossing at ~5–10 pJ/bit is Jeff Dean's ~1,000× factor; a 3 nm SRAM accelerator's millimeter-scale wires still pay ~40×, because wires don't shrink like transistors; Sophon reads each weight from the DRAM tier 0.35 µm above, along a ~14 µm read line inside the tile, and pays ~3×. The multiply itself varies less than ~2× across all three — 3 nm's smaller capacitance is largely offset by its higher voltage (energy is C·V²); the bars differ almost entirely in data movement.

A fair question about that chart: surely a 3 nm multiply is cheaper than a 28 nm one? Barely. Arithmetic energy is capacitance times voltage squared, and the two knobs pull in opposite directions — a 3 nm gate switches roughly 4.5× less capacitance, but it holds ~0.85 V to sustain multi-GHz clocks, while Sophon's 500 MHz arrays run at 0.6 V, and (0.85/0.6)² ≈ 2× hands most of the node's advantage straight back. Add bit-serial arithmetic that spends nothing on instruction fetch or unused precision modes, and the multiply lands within about 2× across all three architectures — roughly 0.05–0.10 pJ — while the data movement spans ~250×. The node race optimizes the one line of the ledger that stopped mattering.

Remove the factor and Dean's own logic runs in reverse: there is nothing left for batching to amortize. Sophon is compute-bound from the first stream — 14,438 tokens per second on an 80B-parameter model at batch one, FP8 — where an HBM4-bound GPU delivers roughly 300. That is a ~48× single-stream gap: almost exactly the 50× better latency Dean asked his audience to imagine. And the domain where it lands — chat, agentic tool loops, long-context reasoning at batch 1–8 — is precisely the agent-based future Dean says inference is the key to.

Low precision, built into the arithmetic

Dean's other prescription — go to very low precision, and build the one precision you need into the hardware instead of supporting many — describes Sophon's arithmetic literally. Computation is bit-serial: activations stream one bit at a time, so precision is the number of cycles, not a mode. FP8 takes eight cycles, BF16 sixteen, INT4 four — halving precision doubles throughput by construction, with no dark silicon spent on formats nobody uses.

References

Jeff Dean, interviewed by Diana Hu, ‘The 1% Rule for Building in AI.’ Y Combinator Startup Library, July 2026. ycombinator.com/library/Vy-jeff-dean-the-1-rule-for-building-in-ai

‘Google Chief Scientist Jeff Dean: Slashing AI Inference Latency by 50x Would Unlock Entirely New Applications, Specialized Hardware Is the Next Battleground.’ BigGo News, July 2026.

PhantaField, ‘PhantaField Sophon Whitepaper,’ Rev 4.1 — energy model §3.C, batch-size analysis §5.A.5c. phantafield.com/whitepaper