Holds and settlement: why max_tokens is money

One small number in your request decides how many concurrent calls your balance can carry.

5 minLesson 1 of 90 of 3 steps

Before any paid request goes out, an amount is set aside from your balance. When the answer finishes, the real cost replaces it and the rest is freed. That simple mechanism explains many behaviours that look odd at first.

How the hold is worked out

For chat, the hold is the cost of the prompt plus the output ceiling. The ceiling is your max_tokens; if you send none, the model's maximum output is used.

  1. An estimate is made and the amount is held from your balance.
  2. The request goes to the model and the answer comes back.
  3. The real cost is computed, the hold is released and the real amount is charged.

If you set max_tokens very high, a large amount is held for every request, even when the answer is short.

What it does to concurrency

Parallel requests each take their own hold. Say ten workers run in parallel, each with a large output ceiling. Most of your balance sits in holds and the eleventh request fails, even though real spending is low.

In that case the system behaves plainly:

  • If you send no max_tokens and the balance can't cover the ceiling, that request's limit is lowered to what the balance covers.
  • If you send max_tokens and the balance can't cover it, you get a 402, and the message states the number it can cover.
Set a realistic max_tokens for each kind of request. Holds get smaller and you meet insufficient_balance far less often.

The cost of a cut-off request

Once the first byte of an answer has arrived, the model can't be changed. If a stream breaks midway, the tokens produced until then are charged. For long answers, keep the received text on your side.

Classes of token

Every token falls into exactly one class: normal input, cache write, cache read, output, reasoning, image and audio. Some models charge a higher rate for very long prompts. Every rate is on each model's page, in rials.

See the real cost0 of 3 steps

  1. Send a request with max_tokens of 500 and get a short answer.
  2. See usage.cost_rial in the response.
  3. In the logs, compare the cost with what was held: only the real amount came off.

What you learned

  • The hold is the prompt's cost plus the output ceiling.
  • A realistic max_tokens makes concurrent work easier.
  • Next: making repeated prompts cheap.
Prompt caching and cutting the billIf you send the same long text every time, you are paying for one thing over and over.Done, next lesson