Prompt caching and cutting the bill
If you send the same long text every time, you are paying for one thing over and over.
If your app sends the same long system prompt or fixed document every time, you pay full price for that fixed text on every call. Prompt caching solves exactly that.
How it's billed
| Token kind | Rate |
|---|---|
| Normal input | The base input rate |
| Cache write | Slightly above normal input |
| Cache read | Far below normal input |
Each model's exact rate is on its page. If a model has no separate cache rate, caching buys nothing there.
What caching needs
Caching works only when the start of the prompt repeats byte for byte.
One small difference at the front voids the whole cache. So get the order right:
messages = [
# fixed: always identical, always first
{"role": "system", "content": LONG_INSTRUCTIONS},
# variable: after the fixed part
{"role": "user", "content": user_question},
]- Always put the fixed part first.
- Keep the current time, random ids or the user's name out of the fixed part.
- Keep the order and schemas of your tools the same between requests.
Keeping a conversation on one route
Send session_id in the body or an x-session-id header, and the turns of a conversation stay together so the cache stays warm. They also show up under that id in your usage reports. Without one, the system tells what it can from the start of the conversation.
Three more ways to lower the bill
- Give simple work to a small model. The gap between tiers is often several times over.
- Shrink images before sending. A 4000-pixel photo usually gives the same answer at several times the cost.
- Send only the tools needed at that step. Every tool's schema counts as input.
See the cache working0 of 3 steps
- Send a request with a long, fixed system prompt, and straight away repeat it with another question.
- In the logs, check the Cached column on the second request.
- If it stays at zero, the start of the prompt still differs per call, or this model has no cache.
What you learned
- The fixed start of your prompt must repeat unchanged.
- The Cached column shows what you saved.
- Small models, small images and fewer tools lower the cost too.
Keep your place
Sign in to the panel and each lesson and your path are saved, so you continue from the same spot on any device.