Sophon PFG-1: a monolithic-3D AI ASIC with 330 GB of on-die DRAM and no HBM

8. Energy-Constrained Ceiling on Model Size As transistor scaling slows and data-center power becomes the binding constraint, the practical ceiling on deployable model size is set not by silicon area but by the energy infrastructure — the power a grid, campus, or rack can deliver and cool. A model's lifetime energy splits into two regimes that scale differently and are bounded by different figures of merit: a recurring inference (serving) cost that is memory-bound and grows linearly with parameter count, and a one-time training cost that is compute-bound and grows roughly quadratically with model size at compute-optimal data. An architecture can dominate one regime without dominating the other, so we treat each in turn.

…

Excerpt limited to ~120 words for fair-use compliance. The full article is at Phantafield.