Core42 has released a new whitepaper, ‘Efficiency Wins the Inference Era’, addressing the complex economics of scaling production AI. It advocates for assessing value through performance-to-cost metrics rather than simple token prices. The paper highlights how the Core42 Compass platform optimises workloads across various hardware accelerators, such as NVIDIA and Cerebras, to ensure high throughput, data residency, and robust security.
As generative AI becomes embedded in daily business operations, each interaction consumes tokens, the units a model processes to interpret prompts and generate outputs. Each interaction therefore carries a compute cost. That consumption can grow quickly as adoption expands, prompts and context become more complex, expectations for faster responses increase, and agentic systems turn a single request into multiple model calls behind the scenes.
The whitepaper’s central argument is that the quoted price of an individual token provides an incomplete view of the real cost of production AI. Instead, organizations should assess tokens per second per dollar: how much useful output a system delivers, what it costs to generate, and how quickly it reaches the user. This shifts the focus from simply purchasing lower-cost tokens to determining whether AI can deliver useful business outcomes repeatedly, economically, and at the speed each use case requires.
Achieving this requires organizations to move away from running every workload on the same model and infrastructure. Different tasks have different requirements, from real-time applications that depend on low latency to high-volume workloads that benefit from greater throughput. Core42 Compass is designed around this principle, routing each workload to the most appropriate combination of model, accelerator, and deployment path across diverse silicon, including NVIDIA, AMD, Qualcomm, and Cerebras.
The platform operates with a 99.5% availability commitment, serves more than seven million API requests and processes well over 100 billion tokens per week across production workloads. Its fastest Cerebras inference path reports throughput improvements of up to 20 times.
These decisions also determine where workloads run, how data is handled, who can access them, and how usage and costs are monitored. The whitepaper therefore argues that efficiency, governance, and sovereignty must be managed together. Compass combines centralized visibility into usage and cost with budget controls, access controls, guardrails, in-country data residency, encryption, zero logging of customer data, and more than 170 security policies.
