Enterprise generative AI spending tripled to $37 billion in 2025, according to Menlo Ventures. At the same time, per-token prices have fallen roughly 1,000x over three years. Cheaper tokens haven't lowered bills, they've raised them, because usage grows faster than price drops. Kong's 2025 enterprise survey found 37% of companies already spend over $250,000 a year on LLM APIs, and 72% expect that number to climb.
The good news: most of that spend is optimizable. Teams that combine a few of the tactics below typically cut 30-70% of their LLM bill without touching output quality. This post walks through the tips, in the order that gives you the fastest payback first.
Quick summary
- Compress and trim prompts before they hit the API
- Cache both exact and semantically similar requests
- Route each query to the cheapest model that can handle it
- Cap output length and ask for concision explicitly
- Trim what you retrieve before you stuff it into context
- Batch non-urgent work for provider batch discounts
- Set hard budgets and watch spend per feature, not just per app
- Put guardrails on autonomous agents before they run unattended
- If you use agentic coding tools like Claude Code, watch cache windows, session length, and MCP servers, they quietly drive cost too
1. Compress your prompts, don't just shorten them by hand
Manually trimming a system prompt gets you marginal savings. Automated prompt compression gets you a lot more. Microsoft Research's LLMLingua uses a small model to score and drop low-information tokens, and reports up to 20x compression with only about 1.5% quality loss on the GSM8K benchmark at that compression rate. LongLLMLingua, the long-context variant, improved RAG retrieval performance by 21.4% while using a quarter of the original tokens.
Example: a RAG prompt with five retrieved chunks often runs 4,000-8,000 tokens of context. Compressed with LLMLingua, that same context can drop to 800-2,000 tokens while keeping the answer accurate, a 60-80% cut on context-heavy workloads.
2. Use prompt caching for repeated context
If your app resends the same system prompt, tool definitions, or long reference documents on every call, you're paying full price to re-process text the model has already seen. Anthropic discounts cached input tokens by 90%, OpenAI by 50%. The rule of thumb: if your system prompt is over roughly 1,000 tokens and you're making thousands of calls a day, caching pays for itself within hours.
Example: a support bot with a 2,000-token policy doc in every prompt, called 50,000 times a day, saves the equivalent of processing that doc from scratch 45,000 times a day once caching kicks in.
3. Add semantic caching for near-duplicate queries
Exact-match caching only fires when the prompt is character-for-character identical. Semantic caching (GPTCache, or Redis's RedisSemanticCache) embeds each query and matches it against previous ones by similarity, so "What's the weather in NYC?" and "Tell me New York City's weather" hit the same cached answer. In customer-facing chatbots, 20-40% of traffic is often near-duplicate, and GPTCache reports up to 10x cost reduction on that slice.
4. Route by task complexity, not by default to your biggest model
Most queries don't need your most expensive model. Classification, tagging, short extraction, and routine rewrites are commodity work for a small model. RouteLLM, from UC Berkeley's LMSYS team, uses a trained classifier to send simple queries to a cheap model and only escalate genuinely hard ones. Their published results: 85% cost reduction on MT-Bench while keeping 95% of GPT-4-level quality, 45% reduction on MMLU, 35% on GSM8K.
Example: processing 10,000 support tickets costs roughly $10 with a lightweight model like GPT-4o Mini, versus $1,300+ with a top-tier reasoning model, a 130x difference for functionally the same task.
5. Set a hard budget gateway, don't rely on end-of-month billing
Cloud-style billing dashboards show you cost 24-48 hours after it's already spent. LiteLLM, the most widely adopted open-source LLM gateway (~38,900 GitHub stars), sits in front of 100+ providers and lets you set max_budget and budget_duration per key, per user, or per project. Requests that would blow the budget get rejected instead of silently accumulating. This matters most for autonomous agents, which can burn hundreds of dollars in minutes with no human watching.
6. Control output length explicitly
Output tokens typically cost 3-10x more than input tokens across major providers. Telling the model to "be concise," setting an explicit max_tokens, and asking for structured output instead of prose all reduce the most expensive part of the bill. It's usually worth spending a few extra input tokens on instructions if it shortens the response.
7. Trim what you retrieve before it hits the prompt
In RAG pipelines, context bloat is often the single biggest cost driver, not the base query. Before sending retrieved chunks to the model, re-rank and truncate to the passages that actually answer the question, rather than sending everything the retriever returned. This is the same problem LLMLingua's long-context mode targets, but you can get partway there just by tightening your retrieval top-k and chunk size.
8. Batch what doesn't need a real-time answer
Anything that isn't user-facing in real time, nightly summarization, backfills, bulk classification, is a candidate for batch processing. Google offers a 50% discount for batch requests; OpenAI's Batch API offers a similar discount for jobs that can tolerate a delayed response. If a job can wait a few hours, batching is close to a free cost cut.
9. Track cost per trace, not just per app
A single agent workflow might call a classifier, a retriever, and a generator in sequence, each with different cost profiles. Langfuse (open source, ~23,000 stars, acquired by ClickHouse in early 2026) traces multi-step pipelines and calculates cost at ingestion time per generation, so you can see exactly which step in the chain is expensive, the classification call, the RAG retrieval, or the final generation, instead of guessing from a single blended number.
10. Watch for the multi-provider arbitrage gap
Three providers control the large majority of enterprise LLM usage, and list prices between them can span more than 300x, from a few cents per million tokens on lightweight models up to well over a hundred dollars per million tokens on frontier reasoning models. Layer in batch discounts, context caching rates, and priority-vs-flex tiers, and manually tracking the cheapest valid option per task becomes impractical. This is exactly what routing tools like LiteLLM and RouteLLM automate.
11. Fix your attribution gap before you try to optimize
You can't cut what you can't attribute. Native provider tools are thin here, OpenAI gives you only "user" and "project" as attribution fields, with no room for custom tags like feature=checkout or env=prod. Tools like LiteLLM's tag-based spend tracking and Langfuse's trace-level attribution fill this in at the code level; platforms like Finout's Virtual Tagging do it at the organizational level, allocating AI spend across teams and business units without waiting on providers to add native tagging.
12. Put guardrails on agents specifically
Multi-step, autonomous agents are the newest and least controlled source of AI spend. An agent that reasons, calls tools, and iterates can rack up hundreds of dollars in minutes if nothing stops it. Combine hard budget caps (tip 5) with anomaly detection that flags a cost spike in near real time, rather than waiting for the monthly invoice to surface it.
Bonus: tips specific to agentic coding tools (Claude Code, Cursor, and similar)
If your team is burning tokens through an AI coding assistant rather than a custom API integration, most of the above still applies, but there are a few extra levers that only show up once you watch a live session closely.
13. Work in tight bursts, not idle-then-return
Prompt caching in tools like Claude Code has a short window, five minutes in Claude's case. Step away for longer than that and come back, and the entire conversation history, including the roughly 30,000-token base system prompt, falls out of cache and has to be reprocessed at full price. In a live demo session, a simple one-line follow-up prompt cost about 2 cents when it hit the cache, and about 12 cents when it missed the cache after a short idle gap, same question, same model, the only difference was timing. Batch your related asks into one sitting instead of trickling them out across a day.
14. Keep sessions scoped to one task
Every prompt in a session resends the full history that came before it. A session that wanders across five unrelated tasks keeps paying to resend all of it, and gets progressively more expensive every time it misses the cache. Start a new session per task rather than letting one session run all day.
15. Audit which MCP servers are actually enabled, per project
Every MCP server you enable injects its tool definitions into the prompt on every call, whether you use that tool or not. In one test, adding just two lightweight MCP servers added roughly 1,000 extra input tokens per prompt, without the assistant touching either tool. Configure MCP servers per project directory (only what that specific project needs), rather than turning them all on globally "just in case."
16. Use the tool's own usage command before you guess
Coding assistants like Claude Code expose an in-session usage view that breaks down input, output, cache-read, and cache-write tokens per model. Check it before assuming where cost is coming from, the breakdown is usually more informative than intuition.
17. Turn on OpenTelemetry, then let the model analyze its own logs
Most AI coding tools now support OpenTelemetry, so you can pipe session-level cost and token metrics to a collector, an observability platform, or, as of a recent AWS integration, straight into CloudWatch. Once that data exists, you can point the assistant at its own session log files and OTel output and have it build a cost comparison across sessions instead of digging through raw JSON by hand.
18. Script the investigations you repeat
If you catch yourself asking the assistant to re-analyze usage logs the same way every week, have it write you a reusable script once. Then you run the script, instead of re-running (and re-paying for) the same investigative prompt from scratch each time.
19. Route enterprise usage through your cloud provider for attribution
Running a coding assistant through a cloud-hosted model (for example, Claude Code via Bedrock instead of directly against the vendor API) puts the spend into your existing cost and usage reports, where you can tag it by IAM principal, team, or project. That solves the attribution problem from tip 11 without waiting on the vendor to add native tagging.
20. Decide deliberately whether to capture prompt content in telemetry
OTel-based usage tracking is anonymous by default, it tells you a session happened and how many tokens it used, not what was in the prompt. Capturing full prompt text requires an explicit opt-in and comes with real confidentiality tradeoffs. Make that call on purpose rather than defaulting it on.
Where to start
If you're only going to do three things this week: turn on prompt caching, cap output tokens, and set a hard budget on your gateway. Those three take the least engineering time and typically produce the fastest visible drop in spend. Once that's in place, model routing and semantic caching are the next-highest-leverage additions for high-volume, repetitive workloads.
For a deeper, code-level walkthrough of the specific open-source tools mentioned above (LiteLLM, Langfuse, LLMLingua, RouteLLM, GPTCache), see 5 Open-Source Tools to Control Your AI API Costs at the Code Level.
cloud & AI spend
.svg)
