Serving configuration measured
| Model | Qwen3.8-27B, NVFP4 (4-bit) weights, 16-bit activations |
|---|---|
| Hardware | Four data-center GPUs, tensor parallel, owned and operated by Prefill Systems in the US |
| Engine | vLLM, FP8 (E4M3) KV cache |
| Speculative decoding | DFlash draft model, 7 draft tokens per step |
| Context | 262,144 tokens |
| Prefix cache | GPU memory, then host RAM, then NVMe; cached prefixes survive an engine restart |
| Gateway admission | Four concurrent requests, plus two more whose prompts are estimated at 32,768 tokens or longer |
Prefill speed
One uncached request at a time, measured on the test engine. Every prompt carries a unique prefix, so nothing is served from cache.
| Prompt tokens | Time to first token | Prefill tok/s | Measured |
|---|---|---|---|
| 8,155 | 2.03 s | 4,025 | Sep 26, median of 6 |
| 32,824 | 8.29 s | 3,960 | Sep 26, median of 6 |
| 64,043 | 18.6 s | 3,446 | Sep 24, median of 3 |
| 128,052 | 43.3 s | 2,955 | Sep 24, median of 3 |
Decode speed by concurrency
Steady-state generation on the test engine, September 26: 30-second windows after every stream had started, median of three rounds, with the production engine idle throughout.
| Concurrent streams | Prompt length | Per-stream tok/s | Aggregate output tok/s |
|---|---|---|---|
| 1 | 2,048 | 156 | 156 |
| 4 | 2,048 | 63.7 | 255 |
| 4 | 32,768 | 46.0 | 184 |
With speculative decoding, each step produced a median of 2.9 tokens at 2,048 tokens of context and 2.6 at 32,768. On September 25, the same engine without speculative decoding decoded a single 2,048-token stream at 64 tokens per second.
Sustained mixed load
Ten minutes at four concurrent requests on the test engine, September 24. All 134 requests completed, sustaining 107,328 prompt tokens and 8,220 completion tokens per minute at 13.3 requests per minute.
| Prompt avg | Output avg | TTFT p50 | TTFT p95 | Per-stream tok/s | Aggregate output tok/s |
|---|---|---|---|---|---|
| 8,091 | 620 | 2,273 ms | 4,232 ms | 40.0 | 137 |
This run predates a September 25 engine change to how drafts are verified when several requests decode together. That change raised four-stream decode at 32,768 tokens of context from 70 to 173 tokens per second, so the run understates current mixed-load throughput. The public model document declares capacity below it: 85,000 prompt tokens and 6,500 completion tokens per minute, 10 requests per minute, and 4 concurrent requests.
Context-length verification
Through the public path, September 29. Each prompt hides an eight-character code a third of the way in and asks for it back. The prompts share filler text ahead of the code, so part of each longer prompt was served from the prefix cache; the second column shows how much.
| Prompt tokens | From cache | Streamed | Time | Result |
|---|---|---|---|---|
| 7,107 | 0 | No | 1.9 s | Code recalled |
| 28,338 | 0 | No | 7.3 s | Code recalled |
| 113,276 | 8,192 | No | 34.2 s | Code recalled |
| 221,194 | 32,768 | No | 79.0 s | Code recalled |
| 221,192 | 73,728 | Yes | 67.0 s to first token | Code recalled |
| More than 262,144 | — | No | 3.1 s | Rejected with HTTP 400 above the context limit |
Tool and response conformance
The public-path matrix passed 18 of 18 cases on September 29, and every request was served by this backend with no fallback or retry:
- Streaming and non-streaming usage reporting
- Automatic, required, none, and named-function tool choice
- Parallel and streaming tool calls
- Tool-result round trips
- JSON object and strict JSON-schema responses
- Streaming and non-streaming reasoning fields
- Code recall at four prompt lengths, and an HTTP 400 above the context limit
Prefix cache
Measured on the production engine on September 25. A 59,982-token prompt with four planted codes reached its first token in 16.5 seconds cold. Resent, 57,344 of its tokens came from GPU memory and the first token arrived in 2.03 seconds. After a full engine restart, the same 57,344 tokens were restored from NVMe and the first token arrived in 3.20 seconds. All three answers were identical, and so were the first token's top five log probabilities. Under four concurrent requests, 20 of 20 answers restored from cache matched their cold answers.
Later that day the cache moved to a finer 2,048-token grid. On the production engine, resent, half-prefix, and extended prompts then hit the cache and produced token-identical output with unchanged log probabilities. The restart check was not repeated after that change.
The test engine replayed three concurrent agent sessions. Each started with its own 20,000-word system prompt, then ran 20 turns, each appending an assistant message and a tool result of 200 to 3,000 words. Across the 57 turns after each session's first, 89.7% of prompt tokens came from cache, and time to first token was 4.60 seconds at the median and 7.28 seconds at p95.
Output quality against BF16 weights
The same model with BF16 weights on the same GPUs, evaluated on the same problems with thinking on, temperature 1.0, and up to 65,536 output tokens, September 24. Differences are NVFP4 minus BF16, with paired bootstrap 95% intervals over problems.
| Benchmark | BF16 | NVFP4 | Difference |
|---|---|---|---|
| AIME 2025 (30 problems) | 92.2% | 93.3% | +1.1 pp [−5.6, +7.8] |
| AIME 2026 (30 problems) | 91.7% | 93.3% | +1.7 pp [−5.0, +7.8] |
| LiveCodeBench v6 (83 problems) | 81.9% | 80.7% | −1.2 pp [−6.0, +3.6] |
| Multi-key recall at 32K, 128K, and 224K tokens (60 answers, four codes each) | 100% | 100% | — |
Every interval includes zero: on these tests the 4-bit weights are within noise of BF16. AIME scores are per-problem means; BF16 was sampled about three times per problem and NVFP4 about once. Early in this evaluation, the NVFP4 engine stopped when a long uncached prompt arrived while other requests were decoding. The fault was fixed and the fix is in production. Ten 224K recall requests lost to it were run again and are counted above. After the fix, the engine ran 6.8 hours of evaluation load with no engine errors.
Interpretation
- Most rows are engine measurements. The public-path rows include the gateway and Cloudflare.
- Stream very long prompts. A non-streaming response that takes more than 100 seconds can be cut off at the Cloudflare edge; a stream receives a keep-alive comment every 15 seconds while the prompt is processed. The 221,194-token request above finished in 79 seconds with part of its prompt cached.
- Long prompts increase time to first token. Agent workloads that resend a growing conversation are mostly served from cache after the first turn.
- Capacity declarations are intentionally below measured peaks.
- No uptime SLA is implied. Current availability is published on the status page.