Serving configuration measured
| GPU | NVIDIA RTX PRO 6000 Blackwell, 96 GB |
|---|---|
| Model | Qwen3.8-27B, NVFP4 weights |
| Engine | vLLM, FP8 KV cache, MTP depth 3 |
| Context | 262,144 tokens |
| Measured path | Client → Cloudflare → Prefill Systems gateway → worker tunnel |
| Cache policy | Unique prompt nonce for throughput runs; prefix caching could not improve the result |
C8 sustained-load results
The C8 run kept at most eight sequences active. All requests completed with HTTP 200 and none spilled to the fallback provider.
| Workload | Prompt avg | Output avg | TTFT p50 | TTFT p95 | Per-stream tok/s | Aggregate output tok/s |
|---|---|---|---|---|---|---|
| Mixed chat | 1,043 | 77 | 954 ms | 3,909 ms | 52.5 | 199 |
| Prefill-heavy | 8,050 | 87 | 6,423 ms | 12,166 ms | 14.0 | 52 |
| Decode-heavy | 1,067 | 1,023 | 1,782 ms | 2,527 ms | 78.3 | 553 |
| Long prompt | 32,051 | 120 | 33.5 s | 55.1 s | 5.4 | 17 |
Aggregate prefill reached 290,808 prompt tokens per minute on the 8K workload. Aggregate decode reached 553 tokens per second on the decode-heavy workload. The public declaration keeps headroom at 240,000 prompt tokens, 26,000 completion tokens, 120 requests per minute, and eight concurrent requests per active worker.
Context-length verification
| Prompt tokens | Wall time | Result |
|---|---|---|
| 7,109 | 1.6 s | Needle recalled |
| 28,339 | 6.1 s | Needle recalled |
| 113,275 | 32.0 s | Needle recalled |
| 221,194 | 84.5 s | Needle recalled |
| ≈327,000 | 3.6 s | Rejected with HTTP 400 above the configured limit |
Tool and response conformance
The final public-path matrix passed 18 of 18 cases:
- Streaming and non-streaming usage reporting
- Automatic, required, none, and named-function tool choice
- Parallel and streaming tool calls
- Tool-result round trips
- JSON object and strict JSON-schema responses
- Streaming and non-streaming reasoning fields
Current cache proof
The current official NVIDIA worker artifact restored 217,600 tokens of a 220,033-token shared prefix from its filesystem tier after an engine restart. The repeat completed its cache restore in 2.582 seconds and returned the exact expected canary. That result validates cross-restart prefix reuse; it is separate from the cache-disabled throughput table above.
Interpretation
- These are end-to-end service measurements, not isolated kernel benchmarks.
- Long prompts increase time to first token even when aggregate prefill remains high.
- Capacity declarations are intentionally below measured peaks.
- No uptime SLA is implied. Current availability is published on the status page.