The first AI prototype is usually cheap. A small team connects an application to a capable model, tests a few hundred requests and sees a bill that looks almost irrelevant next to the potential value. Then the system reaches production. More users create more requests, conversations become longer and agents call more tools. What looked inexpensive in a demo becomes a serious operating cost at scale.
This is one of the biggest complaints we hear from early enterprise AI adopters: the system works, but the economics become harder to defend as usage grows. The answer is not to abandon AI or force every task through the cheapest model available. It is to design the system so that expensive intelligence is used only where it produces additional value.
The right goal is not the lowest possible token price. It is the lowest cost per successful business outcome.
Why AI costs grow faster than expected
An AI bill is not simply the advertised price of one model multiplied by the number of users. A production workflow may make several model calls to complete one task. Each can contain a large system prompt, tool definitions, retrieved documents, conversation history and previous tool results. Add retries, fallback models and agent loops, and one user request can quietly become tens of thousands of tokens.
A useful way to think about the cost is:
AI cost ≈ requests × calls per task × tokens per call × token price, plus tool and infrastructure costs
The exact formula differs between providers, but the lesson is the same. A 20% discount does not help much if the application sends five times more context than the model needs or allows an agent to loop without a clear stopping condition. Cost optimisation must happen at the system level: model selection matters, but so do context design, caching, batching, observability and whether an LLM should be used for a step at all.
Start by measuring cost per completed task
Before changing models or rewriting prompts, establish where the money is going. Record the selected model, input and output tokens, cached tokens, tool calls, agent steps, retries, latency and final status. Where possible, connect that data to a business outcome such as an invoice processed, support case resolved or report completed.
This changes the conversation from “we spent R200,000 on tokens” to “it costs us R3.20 to process an invoice that previously required R45 of staff time.” The second number can be managed and defended. It also reveals failures that aggregate API dashboards hide. A cheap model that causes two retries may cost more per successful result than a stronger model that succeeds the first time.
Providers expose increasingly useful measurement tools. Anthropic, for example, provides a token-counting endpoint that can estimate a request before it is sent, including system instructions, tools, images and documents. Whatever provider you use, build cost attribution into the application rather than waiting for the monthly invoice.
1. Route each task to the cheapest model that can do it well
The largest architectural mistake is sending every request to the most capable model. A complex contract review may justify a frontier reasoning model. Extracting five fields from a standard invoice, classifying an email or rewriting a short description may not.
Last year, we wrote about how we reduced LLM costs by up to 80% by building Aether, a routing layer that evaluates each request and selects an appropriate model. The principle remains one of the strongest cost levers available: reserve expensive models for the minority of tasks that genuinely require them.
A practical routing system starts with deterministic rules. Known low-risk tasks go to a smaller model, while regulated, ambiguous or high-value tasks go to a stronger one. A lightweight classifier can handle the requests between those cases, escalating when confidence is low or an evaluation fails. AWS offers a similar concept through intelligent prompt routing in Amazon Bedrock, which predicts response quality and routes between models to balance quality and cost.
Routing should always be tested against a representative evaluation set. The aim is not to maximise the percentage of traffic sent to the cheapest tier. It is to maximise savings while keeping the success rate above the threshold the business requires.
2. Stop sending the model context it does not need
Long context windows are useful, but they can encourage lazy architecture. It is easy to attach an entire policy library, a full customer history or every previous message “just in case.” At scale, that convenience becomes a recurring tax on every request. Worse, irrelevant context can make the model less reliable by burying the useful information in noise.
Retrieval should therefore be selective. Instead of sending fifty document chunks, retrieve a small set of relevant passages and rerank them before inference. Instead of replaying an entire conversation, retain recent turns and a compact summary of older information. Filter large tool results down to the fields needed for the next decision, and remove duplicate logs, markup and metadata.
Tool definitions also count as context. Giving every agent access to dozens of tools means repeatedly sending descriptions and schemas it may never use. Route requests to specialised agents or load tools only when relevant. This creates a smaller prompt, a clearer decision space and often a more reliable agent.
3. Design prompts so caching can actually work
Many production requests repeat the same material: a system prompt, company policies, a style guide, product documentation or a set of tool definitions. Prompt caching allows a provider to reuse the work already performed on that stable prefix rather than processing it from scratch each time.
OpenAI's prompt-caching documentation states that reused input can receive discounts of up to 90% on supported models. Google similarly recommends context caching for recurring queries over large document sets, long media files and chatbots with extensive system instructions.
Prompt structure matters. Put stable, reusable content first and variable user data later. Keep tool definitions, schemas and their ordering consistent. A changing timestamp or reordered tool list near the beginning can invalidate the reusable prefix. Monitor cached and uncached tokens separately; enabling caching does not mean the application is benefiting from it.
Response caching also helps with deterministic or slow-changing requests. If thousands of employees ask for the same leave policy, the application may serve a previously approved answer without calling a model. Use sensible expiry rules and access controls, especially for user-specific or sensitive data.
4. Control output, reasoning and agent loops
Input gets most of the attention because prompts are visible, but uncontrolled output can be equally expensive. A classifier that needs to return one of five labels should not generate a paragraph explaining its choice. A data extraction step should return a strict schema, not a conversational answer. Set appropriate output limits, ask for concise responses and use structured outputs where the task allows it.
Reasoning budgets should match the task. Deep reasoning is valuable for planning and difficult decisions, but wasteful for simple transformations. Give each agent a maximum number of steps, tool calls, retries and total tokens. Send ambiguous or high-risk exceptions to a person rather than allowing the system to keep guessing.
Retries deserve scrutiny. Timeouts and transient provider errors may justify another attempt, but malformed inputs, missing permissions and deterministic tool failures usually do not. Classify errors and ensure tool actions are idempotent so a retry cannot duplicate a payment, email or database update.
5. Move non-urgent work out of the real-time path
Not every AI task needs an answer in seconds. Document enrichment, evaluation runs, indexing and bulk summarisation can usually happen asynchronously. Separating these workloads from the interactive product path lets teams use lower-cost processing tiers and batch APIs.
OpenAI's Batch API currently offers 50% lower costs than synchronous requests in exchange for completion within a 24-hour window. Other providers offer their own batch or deferred-processing options. The exact discount will change over time, but the design principle will not: do not pay a real-time premium for work nobody is waiting to see.
6. Remove LLM calls that should have been normal software
LLMs are powerful, but they are not the right tool for every step. Exact calculations, database lookups, permission checks, duplicate detection and fixed business rules should generally remain deterministic code. Embeddings or conventional search may be enough to shortlist relevant material before a model sees it. A small classifier may replace a general-purpose agent for a narrow routing decision.
This matters because the cheapest model call is the one you do not make. In a well-designed workflow, conventional software handles predictable operations and the LLM is used for the parts that genuinely require interpretation, language or flexible reasoning.
Fine-tuning, distillation and self-hosted models can lower unit costs for stable, high-volume tasks, but they should follow measurement rather than fashion. Hosting introduces GPU, deployment, monitoring, security and maintenance costs. It becomes economical only when the workload is large and predictable enough to justify them.
Cost optimisation without quality collapse
These techniques compound, but their savings should not be added together as if they were independent discounts. Routing changes which tokens are billed at which rate. Context reduction changes the number of tokens available to cache. Batching affects when and how the remaining requests are processed. The correct way to calculate the benefit is to measure the final cost of the workflow after each change.
More importantly, every optimisation should be evaluated on quality. Track task completion, human corrections, escalations, user retries and downstream errors alongside token spend. A system that is 60% cheaper but creates twice as much manual review has not produced meaningful ROI; it has simply moved the cost somewhere else.
A practical 30-day optimisation plan
In week one, instrument the system and identify the five workflows responsible for most spend. In week two, remove unnecessary context, cap outputs and correct retry or looping problems. In week three, test two or three model tiers against a representative evaluation set and introduce routing with a safe fallback. In week four, restructure stable prompts for caching and move eligible workloads to batch processing.
At the end of the month, report cost per successful outcome, not only total tokens. Compare it with the original manual process and include the value created through faster turnaround, higher capacity or improved service. That is the number that determines whether an AI system can scale sustainably.
The real objective is meaningful ROI
Token prices will keep changing, so a durable cost strategy cannot depend on one provider remaining the cheapest. It needs an architecture that measures usage, routes work, controls context, reuses computation and enforces budgets at runtime.
Companies do not need every AI request to be cheap. They need the total cost of completing valuable work to remain lower than the value created. When that principle is designed into the system from the beginning, AI can scale without the token bill scaling blindly alongside it.
If your AI systems are producing value but the operating costs are becoming difficult to justify, Sulta Tech can help audit the workflow, identify the largest cost drivers and redesign it for sustainable scale.