Platform teams have always planned for peaks. What is different about model workloads is the shape of the demand and the price of getting it wrong. Accelerated compute is scarce, expensive and slow to provision, and a single popular feature can change the bill overnight.
Why inference changes the planning problem
Members of our Cloud Engineering employer panel described the same pattern again and again. Teams that had spent years making infrastructure elastic found themselves back in a world of reservations, quotas and waiting lists. The familiar habits did not transfer cleanly.
- Capacity is reserved rather than elastic, so over-provisioning is costly and under-provisioning shows up as latency users notice.
- Cost scales with usage in units product teams rarely plan in, such as tokens or requests per feature.
- Failure looks different: an endpoint can be healthy by every infrastructure metric and still be returning degraded results.
- Data gravity matters more, because moving large model artefacts and datasets between regions is neither quick nor free.
Nobody asked us to build a platform for inference. It simply became one, and the bill arrived before the plan did.
How it maps to the standard
We have not created a separate capability area for model workloads, and we do not intend to. The existing six areas already cover the ground. Platform design asks whether you chose an architecture that fits the workload rather than the trend, which for model serving often means deciding what should not run on accelerated hardware at all. Resilience asks what happens during a region outage when your reserved capacity sits in the region that failed. Cost discipline asks whether spend was visible and defensible before finance had to ask.
At BIPS Professional, where you own a platform in production including its availability and its bill, we are increasingly seeing inference capacity plans as the central piece of portfolio evidence. That is welcome, provided the reasoning is visible and not just the final configuration.
What we ask in the professional discussion
Assessors are interested in decisions, not tooling. Expect questions about how you forecast demand before you had usage data, what you did when reserved capacity ran out, and how you explained the unit cost of a feature to the people who asked for it.
At BIPS Specialist and BIPS Expert we also look for evidence that you set the rules others work within: quotas, cost allocation for model usage, and a clear position on which workloads the organisation runs itself and which it buys as a service. Those are strategy decisions, and at that level we expect you to own them.