The AI infrastructure investment cycle of 2023 and 2024 was built around a specific assumption about hardware. Nvidia GPU clusters would remain the dominant, and in most cases the only practical, choice for both training and inference workloads at commercial scale for the foreseeable future. That assumption drove procurement decisions, financing structures, and competitive positioning across the neocloud sector, the enterprise AI market, and the hyperscaler capital expenditure programs that collectively defined the buildout. The assumption was reasonable when it was made. It is becoming less accurate faster than most of the people who made it expected.
Custom silicon programmes at the three largest US hyperscalers have moved from promising experiments to commercially significant infrastructure faster than even the engineers running those programmes expected. Google’s TPU v8, announced at Google Cloud Next 2026, Amazon’s Trainium 2 deployed at scale inside AWS, and Meta’s MTIA running production inference workloads for internal applications all represent hardware that delivers inference performance at costs reshaping the economics of AI workloads in the fastest-growing segment. Custom silicon AI inference economics are already shaping the competitive landscape of AI infrastructure for the next several years, and financial models that fail to account for them are already outdated.
The Cost Per Token Is the Number That Matters
Infrastructure providers win inference workloads by lowering the cost per output token, which measures how much it costs to generate one unit of AI output under production conditions while meeting the application’s latency and throughput requirements. For training workloads, operators use cost per GPU hour as the equivalent metric, measured at the utilisation levels required for training runs. Custom silicon programmes target these metrics directly and deliver measurable improvements over Nvidia GPU alternatives in the inference segment.
Google’s TPU architecture, optimized for the matrix multiplication operations that dominate transformer-based inference, achieves cost per token economics for certain model architectures that Nvidia GPU-based inference cannot match at equivalent quality levels. Anthropic’s commitment to run Claude model inference on Google Cloud TPUs, embedded in the expanded partnership announced earlier this month, reflects a commercial assessment that TPU inference economics are superior to GPU alternatives for the specific workload characteristics of large language model serving at the scale Anthropic now operates. That assessment, made by one of the most technically sophisticated AI model companies in the world, carries more signal about the actual state of custom silicon inference economics than any vendor benchmark.
What This Means for Neocloud Pricing Power
As Nvidia GPU performance becomes less of a differentiating factor in inference, neocloud operators whose business models rely on GPU-based infrastructure face a direct pricing power problem. A neocloud that charges enterprise customers a premium for dedicated GPU inference capacity comes under growing pressure as hyperscalers lower custom silicon inference costs below the GPU-based alternative. Enterprise customers who currently pay a premium for neocloud GPU inference to avoid the utilisation variability and pricing opacity of hyperscaler shared clouds will eventually reach a price threshold where the cost savings of hyperscaler custom silicon inference outweigh those disadvantages.
That threshold is not yet reached for most workload types. Nvidia GPU infrastructure still provides performance advantages for inference workloads requiring specific capabilities that custom silicon does not yet serve as efficiently. However, as each TPU and Trainium generation closes the performance gap with Nvidia GPUs on additional workload categories, the set of inference use cases where neocloud GPU infrastructure commands a durable premium is shrinking. As covered in our analysis of AI inference cost in enterprise infrastructure, the inference market is already highly competitive and margins are compressing. Custom silicon’s advance in inference economics compounds that compression in ways that neocloud operators with pure GPU capacity strategies cannot easily offset.
The Enterprise Budget Implications
For enterprise AI buyers, advancing custom silicon inference economics creates clear short-term benefits. Declining inference costs expand the set of AI applications that are economically viable to deploy at scale. Applications that were cost-prohibitive at 2023 inference pricing become commercially attractive as 2026 pricing reflects the cost improvements that custom silicon is delivering. The enterprise AI market will grow faster as a direct result of these cost improvements, which represents a genuine expansion of the addressable market for AI infrastructure broadly.
However, the distribution of that market expansion matters for infrastructure planning. The incremental inference demand created by declining costs will disproportionately flow to the platforms offering the lowest cost per token at acceptable performance levels. If hyperscaler custom silicon continues to drive inference cost declines faster than GPU-based alternatives, the incremental demand growth will flow toward hyperscaler platforms and away from the neocloud and independent inference operators who currently serve enterprise inference workloads at GPU-based economics. Enterprise AI budget growth does not automatically translate into neocloud revenue growth if the cost improvements driving that growth originate from infrastructure that neoclouds cannot replicate.
The Training Market Holds Longer
The training workload segment is where custom silicon poses the weakest competitive challenge to Nvidia GPU infrastructure, and that will remain true the longest. Large-scale distributed training requires tight coupling between thousands of accelerators across high-bandwidth networking fabrics, which places hardware demands that custom silicon programmes have not yet matched across all training workload types. Nvidia’s software ecosystem, particularly CUDA, also creates a major switching cost. Even when alternative hardware delivers competitive performance, enterprises struggle to adopt it because they cannot transfer the software optimisation investments they have made in CUDA without significant re-engineering.
However, stability in the training market does not protect neocloud operators from the inference economics of custom silicon. It simply means competitive pressure reaches inference first and training later. Operators who use the training market’s relative stability as an excuse to avoid adapting their inference strategy to the reality of custom silicon are only deferring a reckoning, not avoiding it. The operators who will define the next phase of AI infrastructure competition are building businesses that can thrive in an environment where custom silicon shapes inference economics and where GPU infrastructure plays an increasingly concentrated role in training workloads and specialised inference applications where its advantages last the longest.
