Same GPUs, More Traffic: BLACKBOX AI vs Other Inference Providers

The recap

Every inference provider serves the same open models on the same NVIDIA GPUs. The difference is the serving stack. We rented dedicated deployments from Together AI and Fireworks AI on the exact GPUs we run, replayed the same real production traffic against each, and measured how much work each machine does.

Bottom line: on our hardware, BLACKBOX AI carries more traffic than each provider at its best, while every user gets a faster first token. The biggest gap is DeepSeek V4.1 Flash, at 3.3× Fireworks' throughput per GPU; then Nemotron 3 Ultra, +54% over Together and +91% over Fireworks; then GLM-5.3-Flash, +23% over Together. Matched strictly to each provider's own per-request speed and first-token time, our GLM-5.3-Flash deployment carries 1.3× Together's traffic and 5.4× Fireworks'.

Why tokens per minute, at the same speed

GPUs cost the same per hour no matter who runs them. Tokens per minute (TPM) is how much work one machine does, so more TPM on the same GPUs means a lower cost per token. But any server can take more traffic by making every user wait longer. So we only count TPM at a fair operating point, where each user gets at least the experience the provider gives its own users: the same time to first token (TTFT) and the same output speed per request.

How we tested

DeepSeek V4.1 Flash

On DeepSeek V4.1 Flash we serve one replica on 2× B300; Fireworks' dedicated deployment uses 4× B300, so the comparison is per GPU, at the same number of concurrent sessions.

Agentic coding: production replay

Long, cache-heavy coding-agent sessions (70k–465k-token prompts, ~300-token replies): the workload the deployment was tuned for.

On agentic coding, each of our GPUs carries 3.3× the traffic of a Fireworks GPU at 64 sessions (16.1M vs 4.8M tokens per minute), with first tokens in 0.69s instead of 1.72s and similar output speed per request. Fireworks also failed about 10% of requests at both load levels; we failed none.

Agentic search: tool calling

A very different, non-coding agent workload: web search, where prompts are short (~11.5k tokens) and every reply is a short web-search tool call, with thinking off.

Here each of our GPUs carries 2.5× the traffic of a Fireworks GPU at 64 sessions, with first tokens in 0.44s instead of 1.76s. Fireworks' per-request output speed is higher on these short tool-call replies (141 vs 106 tokens per second), but each reply is only ~130 to 200 tokens, so first-token time dominates how long a call takes.

Nemotron 3 Ultra

On Nemotron 3 Ultra, BLACKBOX AI carries more traffic than every dedicated deployment we tested, with a faster first token and faster output per request.

GLM-5.3-Flash

For a single user the gap is wider: on 10k-token prompts, one BLACKBOX AI stream generates 451 tokens per second with the first token in 0.25s. Together's dedicated deployment on the same 8× B200 does 275 at 0.80s.

Against the lab's own recipe on the same 8× B200, the lab configuration keeps first-token time under a second up to about 21M TPM. Ours does 42.7M at 0.40s: twice the traffic at half the wait.

Fireworks AI has no validated GLM-5.3-Flash configuration for B200, so its 8× B200 deployment runs an untuned default. It peaks at 8.1M tokens per minute at 16 concurrent sessions and degrades beyond that (11 tokens per second per user at 32 sessions, with a 38% prefix-cache hit rate), which we reproduced on a second run.