Fabricio Bloisi and Ioannis Zempekakis
Using a frontier model is the right way to start with AI. The models are powerful, easy to access, and need almost no infrastructure. But once AI runs the day-to-day operation of a business, the economics change. Billions of requests become millions of dollars.
At Prosus, AI agents now work alongside 40,000 people and 5 million partners, serving 1 billion customers. Our token usage grew 52x last year, and we now process 8.2 trillion tokens per month. Operating at that scale taught us something simple: to keep growing exponentially, we could not let our AI costs grow exponentially too. Four learnings got us to a 90 per cent cost reduction, without materially reducing quality. Here they are, so that others can do the same.
1. Not every problem needs the smartest model
Most AI applications send almost everything to their most capable model. That may be convenient, but it is expensive. Some tasks are hard: analysing a complex dataset, planning a series of actions, solving an unfamiliar problem. Most are not: extracting information from a document, rewriting text, producing a routine report.
So we built a simple router that chooses the model based on the objective. Hard, planning-heavy requests go to the strongest models. Simpler ones go to cheaper open-weight models that we run ourselves.
One warning from experience: a model that is cheaper per token is not always cheaper to use. Some open-weight models cost a fifth as much per token but need far more tokens, and sometimes more steps, to reach the same answer. The useful measure is not the price per million tokens. It is the cost to successfully complete the task.
Measuring that means evaluating each model on your own tasks, with real examples. When we did this across the models we use, including Luna, DeepSeek, and Kimi, the ranking changed completely. Models that look expensive by the token became competitive by the completed task, and in one case the premium reasoning setting cost 62 per cent more to gain half a percentage point of success. The public price list tells you very little; your own results tell you everything.
Our recommendation: invest real time in understanding the routing process. Know, for each model, when to select it, what its success rate is on each type of task, and what a completed task actually costs. The evaluation suite that answers those questions becomes a real competitive advantage, and once it exists, routing decisions become obvious. Ours are now good enough that we are teaching AI to make them automatically.
2. Half the memory, twice the capacity
An AI agent is not a search box. It can stay in a conversation for hours or days, carrying earlier messages, instructions, documents, and its own previous answers. In one production workload we studied, the average request carried roughly 115,000 tokens, about the length of a book, and every response reprocessed most of it.
The obvious fix, cutting off the longest conversations, made almost no difference. What worked was reducing what the average request carries: dropping information that is no longer useful, optimising file transfers, and retrieving data only when the model actually needs it. The average context fell from 115,000 to 60,000 tokens, and the same infrastructure handled almost twice the traffic at the same response times.
The question is not how much information the model can remember. It is how much information the model actually needs. Measure what your average request carries, not what your largest one does. At scale, that difference is worth millions of dollars of computing capacity.
3. Idle GPUs are still expensive GPUs
Running your own models changes what you pay for. You are no longer paying per request; you are renting GPUs, and a GPU costs the same whether it is working or waiting.
AI traffic is extremely uneven. In one workload, average demand was about one request per second while peaks were almost six times higher. Buy enough capacity for the peak and most of it sits idle for much of the day. Two things attack this problem from both sides:
First, make each request lighter. The context reduction above increased the throughput of each unit of capacity by around 100 per cent, which for our peak load meant needing half the machines. Same model, same chips, less unnecessary work.
Second, balance the load. Human users create the peaks: lunchtime, evenings, weekends. Repetitive machine work does not have to compete with them. Moving background tasks such as report generation, evaluations, and scheduled agent runs out of peak hours and into the idle valleys means the same GPUs serve customers at noon and run the routine work at four in the morning. The peak shrinks, the valley fills, and utilisation rises without buying anything.
4. Distil: use a specialist
A general-purpose model is often too much model: slower than it needs to be, and more expensive to run and update. Yet much of agentic work consists of narrow, well-defined tasks that repeat thousands or millions of times. Those repeated examples can train a much smaller model that does one job extremely well, at scale.
The first distillation of a frontier model into a task specialist cut inference cost about 10 times. The interesting part came after. A specialist does not need to reason about every possible domain, so we quantised it more aggressively and stripped out everything that does not contribute to its one task. A few rounds of tightening took the same task from 10 times cheaper to 40 times cheaper.
And the threshold keeps moving. Every time frontier models get smarter, the starting point for distillation improves, so the specialists trained from them get better too. The frontier model does the hard reasoning once; the specialist inherits the result and applies it millions of times at a fraction of the cost. That loop, frontier reasons, specialist inherits, cost collapses, is the part most people are underestimating.
Where the 90 per cent came from
There was no single breakthrough. The four disciplines compound, in approximate shares of the total saving:
- Routing tasks to the right model: 50%
- Reducing what each request carries: 20%
- Distilling specialists for repetitive tasks: 20%
- GPU utilisation and load balancing: 10%
Behind all four sit three questions, asked constantly:
- Are we using a more powerful model than this task requires?
- Are we giving it more information than it needs?
- Are we buying more computing power than the workload actually uses?
What 90 per cent cheaper unlocks
The value is not in the 90 per cent. It is in what the 90 per cent makes possible. At a tenth of the cost, AI stops being something you turn on for one query and turn off. It runs continuously: agents that wake up, check conditions, act, and report back; systems that maintain context across every conversation and surface the right thing before anyone asks. The economics flip from "can we afford to do this with AI?" to "can we afford not to?"
One warning to finish. This is not a project you complete. Usage grows, traffic shifts, models improve, and yesterday's optimisation goes stale. As with everything in AI, the feedback loop is what matters: keep asking the three questions, keep re-routing, trimming, and distilling, and the business keeps growing without the cost growing in proportion.
Everyone will have access to the best models. The advantage goes to whoever knows how much intelligence to spend on each problem, and never stops optimising how they spend it. That is where we are going.
