Something changed in enterprise AI budgets in 2025 and the change has accelerated into 2026. The monthly cloud bill arrived, and it was not what anyone had modelled. Enterprises that had deployed generative AI across customer service, internal workflows, and product features discovered that the cost of running AI in production, at the volume that production actually demands, bore no resemblance to the cost of running AI in a pilot. Some organisations reported monthly AI compute bills in the tens of millions of dollars. Others found that the token consumption of a single agentic workflow, running continuously across hundreds of concurrent enterprise processes, was generating costs that scaled faster than the revenue or productivity gains it produced.
The inference cost crisis is not a crisis in the sense of a sudden collapse. It is a structural tension between two trends running simultaneously in opposite directions. The unit cost of inference has fallen dramatically, dropping roughly 280-fold between late 2022 and late 2024 according to Stanford’s AI Index. The total cost of inference has risen sharply, because usage has grown faster than unit cost has fallen. Enterprises deploy more AI across more workflows and more users, generating more tokens, and the rising cost of those tokens now forces a fundamental rethink of where inference runs, how companies price it, and who owns the infrastructure that serves it.
Why the Cloud API Model Breaks at Enterprise Scale
The cloud API model for AI inference was designed for a world where AI consumption was experimental, variable, and low-volume. A developer building a prototype, a team running a pilot, a business unit testing a use case: all of these consumers benefit from the cloud API model because they need flexibility, they do not know their volume requirements in advance, and the economics of paying per token on demand are superior to owning infrastructure for uncertain workloads.
Production enterprise AI is a different situation. An enterprise that has integrated AI agents into its customer service operations knows, with reasonable precision, how many customer interactions it handles per day. An enterprise that has deployed AI across its document processing workflows knows approximately how many documents it processes per month. That predictability shifts inference economics from a variable-cost problem to a fixed-cost problem, and fixed-cost problems typically favor owned or committed infrastructure over on-demand consumption models. The cloud API model charges a premium for flexibility that enterprises running stable, predictable workloads do not need and cannot justify at scale.
AI workloads are breaking cloud abstractions that were designed on different assumptions, and the inference cost problem is the most visible sign of that shift. The cloud pricing model assumed AI workloads would mirror the utilisation patterns of web applications and data processing jobs, with variable demand that benefits from elastic, shared infrastructure. Continuous inference at enterprise scale does not follow those patterns. It runs continuously, at relatively stable volumes, with latency requirements that require reserved rather than shared compute resources. The infrastructure model that serves these requirements efficiently is not a shared cloud platform. It is dedicated, owned, or committed capacity that can be amortized over the volume it serves.
The Maths That Are Driving Infrastructure Decisions
The economic case for moving inference off the cloud is driven by a straightforward calculation that enterprises are performing with increasing frequency. Take a production AI deployment consuming 500 million tokens per day across enterprise workflows. At cloud API pricing for a capable model, the cost per million tokens ranges from roughly $1 to $15 depending on the model and provider. At even the lower end of that range, 500 million tokens per day generates $500 in daily spend, $15,000 per month, $180,000 per year. Scale that to a billion tokens per day, which is realistic for a large enterprise with AI integrated across multiple high-volume workflows, and the annual spend approaches $360,000 at the low end and several million at the higher end of current API pricing.
When Owned Infrastructure Becomes Cheaper Than Cloud
Against those costs, the economics of on-premises or colocation inference infrastructure become compelling at a much lower utilisation threshold than training workloads ever required. A well-specified inference server running an appropriately sized model with optimized serving software drives the marginal cost of additional tokens toward zero once operators cover the capital and operating costs of the hardware. The breakeven point at which owned inference infrastructure becomes cheaper than cloud API consumption varies by model size, hardware configuration, and utilisation rate, but for large enterprises running high-volume, stable workloads, that breakeven arrives at utilisation levels that are well within reach of production deployments.
The agentic AI multiplier makes this calculation more urgent. GenAI’s double role as load creator and load orchestrator means that agentic workflows consume tokens at a rate that direct user interactions do not. An AI agent completing a multi-step enterprise task may make dozens of inference calls, each generating and consuming tokens, to complete a single user-initiated workflow. At scale across hundreds of concurrent agents, the token consumption of an agentic deployment is an order of magnitude higher than a simple chatbot serving the same number of users. That multiplier effect is the primary driver of the cost escalation that enterprises are discovering in their AI infrastructure budgets, and it is the factor that is most consistently accelerating the shift toward owned inference infrastructure.
How Agentic Workloads Change the Cost Calculus
The specific characteristics of agentic AI workloads distinguish them from the general inference cost problem in ways that affect infrastructure decisions. Agentic systems do not simply respond to user queries. They plan, execute, evaluate, and iterate, making multiple inference calls in sequence to complete tasks that a direct user interaction would handle in one. The token consumption per user-initiated task is therefore not the token consumption of a single inference call but the aggregate of every call the agent makes across its planning and execution cycle.
That aggregate is substantial. Enterprises auditing their agentic workflows are discovering that many pipelines consume ten to fifty times more tokens per user-initiated task than they estimated during design. An agent that queries multiple data sources, synthesises information, drafts a response, checks it against policy guidelines, and routes it for approval may make eight to twelve separate inference calls in the process of completing a task that takes a user thirty seconds to initiate. At high volume, the token consumption of that workflow compounds in ways that simple per-query pricing models do not make visible until the bill arrives.
The On-Premises Case Is Stronger Than It Was
The argument for on-premises AI inference has historically been complicated by the hardware requirements of serving large language models. The GPU memory capacity required to run capable models, the networking infrastructure needed to serve low-latency inference at scale, and the operational expertise required to manage GPU clusters at production quality have all represented genuine barriers to enterprise on-premises deployment. Those barriers have not disappeared, but they have become lower in several important ways.
Hardware designed specifically for inference, rather than training, is now commercially available at a range of scales. NVIDIA’s DGX Station delivers substantial inference capacity in a form factor that does not require a dedicated data centre. Smaller, inference-optimised accelerators from multiple vendors are addressing the middle tier of enterprise deployment requirements. The model optimisation techniques that reduce inference costs, including quantisation, which reduces model precision to decrease memory requirements and increase throughput, and speculative decoding, which uses smaller models to generate candidate outputs that larger models verify, are now sufficiently mature and accessible that enterprises without specialist ML infrastructure teams can implement them using available tooling.
Open-weight models have changed the economics of on-premises inference in a second important dimension. Enterprises that depend on closed API models for their AI capabilities are permanently subject to the pricing decisions of the model provider. Enterprises that deploy open-weight models on owned infrastructure control their own cost structure. The performance gap between open-weight and closed models has narrowed substantially over the past two years, to the point where open-weight models are now viable for a wide range of enterprise inference workloads that previously required closed model capabilities. That viability gives enterprises the option to move from per-token API pricing to infrastructure-cost-only pricing for a meaningful share of their AI inference consumption.
What Colocation Offers That Cloud Cannot
The on-premises case is not the only alternative to cloud API consumption for enterprise inference. Colocation, placing dedicated inference hardware in a third-party data centre, offers a middle path that captures many of the cost advantages of owned infrastructure without the facility management requirements of on-premises deployment. For enterprises that have made the decision to own their inference infrastructure but do not want to operate their own data centre, colocation provides the power density, cooling infrastructure, and connectivity that modern inference hardware requires in markets where those resources are not available on enterprise premises.
The rise of inference clouds as a distinct infrastructure tier reflects the market’s recognition that the inference workload profile requires infrastructure that sits between the shared public cloud and fully owned on-premises deployment. Inference cloud providers, which operate dedicated inference infrastructure for enterprise customers on committed-capacity models, can offer per-token economics that are substantially better than public cloud API pricing while requiring less capital commitment than owned on-premises infrastructure. That middle tier is growing because it serves the economic requirements of a large number of enterprise customers who have the inference volume to justify dedicated infrastructure but not necessarily the capital or operational capability to own it outright.
Why Not Everything Moves Off the Cloud
The inference cost crisis and the economic case for owned infrastructure do not mean that enterprise AI workloads will migrate entirely off the cloud. The cloud API model retains genuine advantages for specific categories of AI usage that will persist regardless of the economics of on-premises inference.
Development and experimentation workloads are the clearest case. An enterprise exploring a new AI application, testing different model capabilities, or running evaluation workloads on varying datasets needs flexibility and access to a range of models that on-premises infrastructure cannot provide. The cloud API model is the right economic model for those workloads precisely because their volume is uncertain and their requirements change frequently. The capital commitment of owned infrastructure is inappropriate for workloads whose requirements may not exist in six months.
Access to frontier model capabilities is a second category where the cloud API model retains a durable advantage. The most capable foundation models are available only through cloud APIs. Enterprises that need the reasoning capabilities, multimodal performance, or specialised domain knowledge of frontier models cannot replicate those capabilities on owned infrastructure with open-weight models, at least not at current open-weight model capability levels. For workloads where frontier model capability is genuinely required, the cloud API model is not simply more convenient. It is the only option.
The infrastructure calculus for most enterprises will therefore not be cloud versus on-premises but a portfolio decision about which workloads run where. High-volume, predictable, stable workloads with known latency requirements are candidates for owned or committed infrastructure. Variable, experimental, frontier-capability-dependent workloads remain appropriate for cloud consumption. Managing that portfolio effectively, matching infrastructure to workload characteristics, is becoming a core enterprise IT capability in the same way that cloud cost optimisation became a core capability in the previous decade.
The Hybrid Model Taking Shape Across Enterprises
The leading organisations in enterprise AI deployment in 2026 are converging on a three-tier infrastructure model. Public cloud serves elastic training workloads, experimentation, and frontier model access. Private or colocation infrastructure serves predictable, high-volume inference with known latency requirements. Edge infrastructure serves the subset of AI applications with latency requirements so tight that even low-latency colocation cannot serve them adequately.
That three-tier model maps AI infrastructure decisions to workload characteristics in a way that optimises cost across the portfolio rather than optimising any single workload in isolation. It requires more sophisticated infrastructure management than a cloud-only strategy, and it requires the operational capability to run and manage AI inference hardware outside a public cloud environment. But for enterprises whose AI consumption has reached the scale where cloud API costs are material, the financial case for the additional management complexity is compelling.
The Neocloud Position in the Inference Market
The enterprise shift toward dedicated inference infrastructure is creating a distinct commercial opportunity for neocloud operators who can serve the inference workload profile more efficiently than either public cloud hyperscalers or traditional colocation providers. Neoclouds are redefining competition beyond hyperscalers by building infrastructure specifically optimized for AI inference, rather than for the general-purpose cloud workloads that hyperscaler infrastructure was originally designed to serve.
The neocloud advantage in inference is threefold. Purpose-built inference infrastructure, configured with the hardware, software, and networking specifically required for serving large language models at high throughput and low latency, achieves better performance-per-dollar than general-purpose cloud infrastructure serving the same workloads. Committed capacity pricing, which neoclouds can offer to enterprises with predictable inference volumes, generates better unit economics for both parties than the on-demand pricing model of public cloud APIs. And neocloud operations teams with deep expertise in AI inference optimisation can deliver the model serving performance and cost efficiency that enterprise customers need without requiring those customers to develop that expertise internally.
The neocloud inference market is growing faster than the general AI infrastructure market because it serves the specific economic problem that enterprises are encountering at production scale. An enterprise discovering that its monthly AI cloud bill has become unsustainable does not necessarily want to own and operate inference infrastructure. It wants the economics of owned infrastructure without the operational burden. Neocloud operators who can deliver that combination, providing dedicated inference capacity with managed operations at pricing that reflects the economics of purpose-built infrastructure, are building businesses that address a market need that neither hyperscalers nor traditional colocation can serve as efficiently.
What Enterprises Should Be Evaluating Now
The enterprises best positioned for the next phase of AI deployment are those that are conducting systematic audits of their current inference consumption before committing to infrastructure strategies. Understanding which workloads drive the majority of token consumption, which of those workloads have predictable volume and latency requirements suited to dedicated infrastructure, and which genuinely need the flexibility and frontier capabilities of cloud API consumption is the analytical foundation of an effective inference infrastructure strategy.
The optimisation layer deserves separate evaluation from the infrastructure layer. Model routing, which directs inference requests to the smallest model capable of handling each request adequately, can reduce token consumption by 30 to 50 percent for typical enterprise deployments without any infrastructure change. Semantic caching, which serves cached results for semantically similar queries rather than regenerating outputs, reduces effective API call volume for workloads with predictable query patterns. Quantisation and other model compression techniques reduce the hardware cost of serving capable models on owned infrastructure. These optimisations do not eliminate the case for infrastructure change, but they do change the economics of the decision and must be exhausted before companies make major capital commitments.
The Infrastructure Implications That Extend Beyond Cost
The inference cost crisis is the most visible driver of the enterprise shift toward dedicated inference infrastructure, but it is not the only one. Data sovereignty requirements are pushing regulated industries toward infrastructure models that give them full control over where their data is processed and stored. An AI agent processing healthcare records, financial transaction data, or legally sensitive documents cannot in many regulatory environments use shared cloud infrastructure where data residency and access controls are subject to the cloud provider’s architecture rather than the enterprise’s own.
Data Sovereignty, Security, and Latency Constraints
Intellectual property protection is a second non-cost consideration that is becoming more material as enterprises deploy AI on proprietary data. Enterprises that fine-tune models on their own data, train specialised models on proprietary datasets, or develop AI capabilities that represent genuine competitive differentiation are increasingly reluctant to run those workloads on shared infrastructure where the separation of their model weights and training data from other tenants’ environments depends on the cloud provider’s security architecture. Owned inference infrastructure eliminates that dependency entirely.
Latency requirements for the most demanding AI applications represent a third driver. Enterprise applications where AI response time directly affects user experience or business outcome, including voice AI, real-time financial decision systems, and customer-facing AI products where sub-second response is a product requirement, need inference infrastructure with latency budgets that shared cloud platforms cannot reliably guarantee at scale. Dedicated infrastructure with reserved compute capacity can provide the latency guarantees that these applications require, while shared cloud infrastructure serving variable demand from multiple customers cannot.
The inference cost crisis will resolve itself partially through continued unit cost reduction as hardware efficiency improves and model optimisation techniques mature. NVIDIA’s Vera Rubin platform delivers up to ten times lower token costs than Blackwell, reshaping the economics of cloud API consumption for enterprises that currently cannot afford high-volume deployment. But the combination of cost, sovereignty, intellectual property, and latency considerations driving enterprise infrastructure decisions extends beyond what hardware efficiency improvements alone will address. The enterprise inference infrastructure market is not simply a response to a temporary pricing problem. It is a structural shift in how enterprises think about owning and controlling the AI compute that is becoming central to their operations.
