Latency, first token and sub-200ms decisioning, 4 of 26
The decision takes
0.4 milliseconds.
A client asks you to walk through accept and reject decisioning that has to answer in under 200 milliseconds at fifty requests a second. Here is the same endpoint measured twice, with identical code, from two places.
From the same machine it holds the budget everywhere. From a laptop on the other side of the world it misses everywhere, including at one request at a time.
the same machine, with no network in the way
| at once | requests | per second | p50 | p95 | p99 | the work itself |
|---|---|---|---|---|---|---|
| 1 | 20 | 286.4 | 3ms | 5ms | 6ms | 0.47ms |
| 5 | 20 | 432.1 | 11ms | 12ms | 13ms | 0.34ms |
| 10 | 40 | 513.7 | 18ms | 20ms | 22ms | 0.30ms |
| 25 | 100 | 467 | 46ms | 72ms | 75ms | 0.35ms |
| 50 | 200 | 588 | 79ms | 114ms | 127ms | 0.39ms |
held the 200ms budget at every level up to 50 at once
a laptop in India, over the public internet
| at once | requests | per second | p50 | p95 | p99 | the work itself |
|---|---|---|---|---|---|---|
| 1 | 20 | 4.2 | 228ms | 253ms | 347ms | 5.57ms |
| 5 | 20 | 18.7 | 227ms | 337ms | 354ms | 2.56ms |
| 10 | 40 | 37.9 | 232ms | 299ms | 367ms | 0.72ms |
| 25 | 100 | 93.3 | 230ms | 334ms | 343ms | 1.74ms |
| 50 | 200 | 155.1 | 240ms | 406ms | 524ms | 1.18ms |
missed the 200ms budget at every level, including one request at a time
What that actually means
The work is 0.39 milliseconds at the ninety-ninth percentile with fifty requests in flight. The round trip from India is 228 milliseconds with nothing else happening at all. Almost the entire budget is distance.
So the honest answer to the question is not about the code. Any competent function clears 200 milliseconds by two orders of magnitude. What decides whether the requirement is met is where the endpoint runs relative to whoever is calling it, and that is a deployment decision somebody has to make before the first line is written.
The code still matters for one reason: it has to leave the budget alone. Add one database lookup and the function is competing with the network instead of hiding behind it.
Four decisions
No model, and no database either
Most answers to this question start with a language model. A model call is four hundred milliseconds on a good day, so that answer has failed the requirement in its first sentence. The decision here is a fixed number of comparisons over values that arrive in the request. One lookup would hand the budget to the network.
The tail is flat because the work is bounded
No loop here depends on the input, so the slowest call is the same shape as the fastest. That is why the work percentiles barely move between one request at a time and fifty.
Percentiles are nearest rank, not interpolated
An interpolated p99 over two hundred samples is a number between two real measurements that nobody observed. Every figure on this page is a request that actually happened, which is the only kind worth quoting to somebody who will hold you to it.
The warm up is thrown away, and that is admitted
A serverless platform charges the first caller for a cold start, and including it reports a p99 that describes the deploy rather than the design. It is discarded here. A real user does hit it. Pretending otherwise gives you a benchmark that falls apart in week one.
What the numbers do not cover
Two hundred requests at each level is enough to see a tail and not enough to see a rare one. A p99 over two hundred samples is the second slowest request, so anything that happens to one call in a thousand is invisible here.
The load also comes from one machine on one connection, which measures that connection as much as the endpoint. Real load arrives from everywhere at once, and the way to know is to run it from everywhere at once.
The endpoint takes a payment and answers. The live system is what else is running on it. The whole list is 41 requirements from 114 job posts, with the gaps at the same size as the wins.