The Blackbox Enterprise API runs on direct Vertex AI for Claude and direct Azure OpenAI for GPT, under strict Zero Data Retention, with everything else built around making that one stable surface look the same to every client. Here's what it does that generic public gateways structurally can't.
The scorecard
Seven capabilities a branded, multi-tenant LLM surface needs — and where public gateways structurally fall short.
1. Long-running thinking tasks survive end to end
Opus 4.7 / 4.8 reasoning, multi-step agents, and large contexts legitimately need 5–30 minutes of model time. Most public gateways enforce 60–300s request timeouts and don't proactively keep the SSE channel alive when the model is mid-thinking. The visible symptoms: the client receives connection closed, stream ended unexpectedly, or — worst case — gets only thinking blocks with no text and a 200 response, because the upstream timed out mid-generation. Clients waste budget retrying the same prompt.
The Blackbox API gives the model the time it actually needs. There's a 3600-second timeout on every upstream request; upstream keepalive SSE events are explicitly forwarded as comments (: keepalive) so intermediaries don't close idle connections; and the connection pool is tuned for long-lived streams — 300 sockets, 120s TCP keep-alive.
2. Multi-region Azure for GPT-5.x
Generic gateways run one deployment per model. Send gpt-5.4, get the single Azure deployment the gateway happens to have configured. A throttle on that deployment, a region outage, a parameter incompatibility, or a context-size limit on that region means your request fails. There's no notion of *where* beyond what the upstream picked.
The Blackbox API manages up to 10 Azure regions as a pool, with intelligent per-request routing:
Round-robin across healthy regions — failed regions cool down for 60 seconds and auto-recover.
Per-region capability maps — not every region has every deployment. Region-4 (Poland) carries gpt-5.4 / 5.4-pro / 5.5; other regions cover the long tail. The router knows which regions can serve which model.
Context-aware routing for gpt-5.4 — >300k tokens routes to region-4 only (the only region with the deployment quota); >200k tokens excludes Sweden Central (known to underperform on very long contexts).
Parameter-incompatibility awareness — region-4 rejects tools + reasoning_effort together on /chat/completions, so that region is silently dropped from the pool when both are present.
Cross-region ZDR safety — Azure regions have independent encryption keys, so we strip reasoning.encrypted_content from multi-turn inputs to keep a turn produced in one region from 400-ing on the next.
3. redacted_thinking never leaks to clients
Anthropic's redacted_thinking content blocks are an encrypted, summarized-thinking format that clients usually can't display. On OpenRouter and similar relays, Anthropic responses pass through largely unchanged, so callers receive these opaque blocks in content[] and as content_block_start SSE events. UI integrations show garbled output or have to silently drop blocks they don't understand.
On the Blackbox API they never leak — because Claude routes direct to Vertex with no third-party gateway in between to relay them.
4. Cross-provider failover under one model name
On a typical gateway, one model means one provider. If Vertex rate-limits, OpenRouter rate-limits, or Anthropic returns a 529 capacity-overload, your request fails. The most you get is a few retries against the *same* provider before the gateway gives up and forwards the error. There's no concept of "if Vertex can't serve this, try a different backend transparently."
The Blackbox API runs every model that matters as an ordered chain of providers:
Operationally, the orchestrator only fails to the client when *every* hop fails — a primary-provider blip becomes a slice of extra latency, not a customer-visible 5xx. Every fallback is provenance-tagged in the spend ledger (Fallback-From: vertex) so we can attribute traffic, bill the right rate for the path actually taken, and answer "how often did the primary need backup this week?" with one SQL query. Specific upstream signals are detected, not just generic 5xx — known soft-fail modes route around silently.
5. Automatic prompt caching on Vertex — that actually fires
Anthropic's cache_control shorthand turns long agentic prompts from full-rate input tokens into near-free cache reads. The catch: Vertex AI does not support the top-level cache_control shorthand that Anthropic's own API accepts. Public gateways forward the shorthand verbatim, Vertex ignores it, and callers pay full input-token rates on every turn — with no error, no warning, no log line.
The Blackbox API emulates it automatically, on by default. Every Vertex request gets explicit per-block cache_control breakpoints injected before forwarding: a tail slot (the last cacheable block in the conversation — the growing edge, matching Anthropic's native automatic-caching behavior) and a system slot (the last cacheable block in system, or the last tool definition if there's no system). It respects Anthropic's hard cap of 4 breakpoints per request and TTL ordering — auto-injected breakpoints inherit the maximum TTL already in use anywhere in the body, so a 5-minute auto-injection won't violate Anthropic's "longer TTL must come first" rule. Per-block markers from the caller are always preserved, and a top-level cache_control from the caller short-circuits auto-injection and expands into the explicit per-block form Vertex accepts. Net effect: the cache-hit pricing customers expect from Anthropic actually happens on Vertex, transparently.
6. Optional name remapping for privacy & security
True anonymization for brands, codenames, and customer IDs simply isn't available on public gateways — sensitive names go to the upstream model verbatim. The model sees "Apple", "Project Atlas", "customer 884221", and so does any incident review, abuse path, or rare logging surface on the provider side. The only workaround is for every client team to roll their own pre-/post-processor, which is fragile across multi-turn, breaks signed thinking blocks, and breaks code/diff offsets.
On the Blackbox API it's configured once and applied invisibly. A JSON rules file maps Apple → Orgxx, Project Atlas → Project Xxxxxx, customer 884221 → entity 000000. On the way in, every match is replaced; on the way out, every replacement is restored. The guarantees that make it production-safe:
Same-length, byte-deterministic substitution — code column offsets and diff line positions stay correct.
Per-character casing transfer — Apple→Orgxx, APPLE→ORGXX, ApPlE→OrGxX.
Letter-boundary matching — Apple is replaced; pineapple and AppleScript are not.
Streaming-safe — tokens split across SSE chunks are held back and reassembled.
Case-insensitive reverse — the model emits ORGXX, the client sees APPLE.
7. Inferred PII redaction
Name remapping catches the identifiers you know about in advance. But when your end-users paste an email thread, a CSV row, or a stack trace, you don't know what's in it. Public gateways forward that prompt verbatim — a real customer's address or an API key goes straight to the model, and to any incident-review or rare-logging surface, exactly as typed.
The Blackbox API offers an opt-in inference filter powered by OpenAI's open-weight Privacy Filter (Apache 2.0, runs locally — no third-party endpoint, no data exfiltration). It's a single forward pass, supports up to 128k tokens of context, and scores 96% F1 on the standard PII-Masking benchmark. It detects 8 categories: private_person, private_address, private_email, private_phone, private_url, private_date, account_number (cards/bank), and secret (passwords/API keys). On the way in, matched spans are replaced with semantic placeholders (<PERSON_1>, <EMAIL_2>); on the way back, placeholders are restored before the response reaches the client.
Cross-provider failover, not "try one and die"
The table above is the headline scorecard. The sections below unpack the architectural choices that make those rows hold up under real production traffic — starting with the one that decides whether a bad minute upstream becomes a bad minute for your customers.
A request to a typical gateway lands on one provider. If that provider returns 503, 429, or a capacity error, the failure is forwarded back to the caller. There is retry logic; there isn't fallback logic across independent providers under a single model name. The Blackbox API has both — when the first provider fails before any bytes hit the client, the orchestrator silently tries the next, and the client only sees a 5xx if every hop fails.
One API surface, every model
The biggest source of friction in multi-model applications is that every backend speaks a slightly different dialect — different request shapes, different response shapes, different field names for the same concepts. The Blackbox API absorbs all of it. Your client writes against one surface, and we handle translation and normalization on both sides.
On the way in: cross-API request translation
Three incompatible request shapes dominate today, and they disagree on system prompts, tool-call shapes (tool_calls vs tool_use/tool_result vs function_call/function_call_output), content-block types (text vs input_text/output_text, image_url vs input_image), reasoning surfaces, and usage reporting:
- OpenAI Chat Completions (/v1/chat/completions) — the lingua franca
- OpenAI Responses API (/v1/responses) — what GPT-5 reasoning models actually want
- Anthropic Messages API (/v1/messages) — what Claude wants
- Chat Completions → Messages for Claude on Vertex — system hoisted, tool_calls → tool_use, role:tool → tool_result, image_url → image blocks, signed thinking preserved byte-for-byte
- Chat Completions → Responses for Azure responses-only models — system → instructions, messages → input, tool_calls → function_call, image_url → input_image
- The reverse on every response — streaming and multi-turn round-trips work the same way
On the way back: response normalization
Translating the shape is only half the job. Each backend leaks its own quirks — different model strings, missing usage fields, different error envelopes, multiple reasoning-content homes, and provider-internal metadata. We normalize all of it: model names (openai/gpt-5.4, claude-opus-4-6, anthropic/claude-opus-4-7) all become the canonical Blackbox name with fast aliases preserved; Anthropic usage fields (cache_creation_input_tokens, cache_read_input_tokens, total_tokens) get filled in when upstream omits them; and reasoning fields (reasoning, reasoning_content, reasoning_details) are consolidated into provider_specific_fields per LiteLLM convention. Streaming works through all of it — SSE events get translated and normalized chunk-by-chunk without ever buffering the whole response.
“Write your integration once, against one shape, and we'll keep it working regardless of which provider serves each request.”
Direct access — fewer hops, more control
Every layer in front of a model adds a hop, a failure mode, and an abstraction deciding which features get exposed. We minimize all three by going direct: direct Vertex AI for Claude Opus 4.6 / 4.7 / 4.8 (Google service-account auth, full Anthropic Messages API surface end to end) and direct Azure OpenAI for GPT-5.x (our keys, our regional endpoints, our deployment naming).
Going direct is the cornerstone. Everything else — translation, normalization, ZDR enforcement, observability — gets to work in the clear because nothing in the middle is reshaping the request or swallowing fields. Anthropic-beta headers (interleaved-thinking-2025-05-14, context-1m-2025-08-07), thinking blocks with signatures, cache_control breakpoints, and fine-grained tool streaming all just work. And when an upstream ships a breaking migration — they do, several times a year — we patch the handler the same day, because the handler is ours.
ZDR enforced, not assumed
Zero Data Retention is the whole point. We picked Vertex AI and Azure OpenAI specifically because both offer enterprise-grade ZDR — your prompts and completions are not persisted by the upstream provider for training, fine-tuning, or human review. But a ZDR flag on a request is the easy part. Real ZDR across a multi-tenant, multi-region, multi-provider gateway is a lot more work than flipping a boolean — most of it only becomes visible the first time a multi-turn request crosses a region boundary and the upstream returns a confusing 400.
ZDR forced on, every request, every provider — Azure and OpenAI Responses-shape requests get store=false; Vertex never persists by design; clients can't turn it off.
Cross-region encrypted-content stripping — every Azure region has its own encryption key, so a reasoning.encrypted_content blob produced by region-2 would 400 on region-3 in the next turn. We strip it from multi-turn inputs automatically.
The bottom line
Once an LLM gateway leaves prototype territory and starts serving real customers under a real brand, it has six jobs: anonymize what goes to the model, survive long and expensive reasoning tasks, route intelligently across regions and capacity, surface only what the client can render, fail over when something does break, and save money on the prompts you send most often — all while staying close enough to the actual model that every one of those guarantees holds up under real load.
The Blackbox Enterprise API does all of them on one stable surface — backed by direct Vertex AI and direct Azure OpenAI, with Zero Data Retention that survives regions and multi-turn, region-aware routing, cross-provider failover under one model name, and a single API surface (Chat Completions, Responses, or Messages) that translates and normalizes every model into a consistent shape.
“Public gateways are excellent at making one request to one model work. The Blackbox Enterprise API is built for the case after that — multi-tenant traffic, branded surface, long-running agents, real money on the line, real customers depending on the response showing up.”