Prompt caching and cutting the bill

If you send the same long text every time, you are paying for one thing over and over.

5 minLesson 2 of 90 of 3 steps

If your app sends the same long system prompt or fixed document every time, you pay full price for that fixed text on every call. Prompt caching solves exactly that.

How it's billed

Token kindRate
Normal inputThe base input rate
Cache writeSlightly above normal input
Cache readFar below normal input

Each model's exact rate is on its page. If a model has no separate cache rate, caching buys nothing there.

What caching needs

Caching works only when the start of the prompt repeats byte for byte.

One small difference at the front voids the whole cache. So get the order right:

Python
messages = [
    # fixed: always identical, always first
    {"role": "system", "content": LONG_INSTRUCTIONS},
    # variable: after the fixed part
    {"role": "user", "content": user_question},
]
  • Always put the fixed part first.
  • Keep the current time, random ids or the user's name out of the fixed part.
  • Keep the order and schemas of your tools the same between requests.

Keeping a conversation on one route

Send session_id in the body or an x-session-id header, and the turns of a conversation stay together so the cache stays warm. They also show up under that id in your usage reports. Without one, the system tells what it can from the start of the conversation.

Three more ways to lower the bill

  1. Give simple work to a small model. The gap between tiers is often several times over.
  2. Shrink images before sending. A 4000-pixel photo usually gives the same answer at several times the cost.
  3. Send only the tools needed at that step. Every tool's schema counts as input.

See the cache working0 of 3 steps

  1. Send a request with a long, fixed system prompt, and straight away repeat it with another question.
  2. In the logs, check the Cached column on the second request.
  3. If it stays at zero, the start of the prompt still differs per call, or this model has no cache.

What you learned

  • The fixed start of your prompt must repeat unchanged.
  • The Cached column shows what you saved.
  • Small models, small images and fewer tools lower the cost too.
Streaming and toolsA model talks; a tool does things. This lesson shows the loop that joins the two.Done, next lesson