A benchmark run by the Azure Foundry team: encrypted Blackbox delivers 3.02× Azure’s mean throughput, with 13.3% lower throughput than regular Blackbox.
The Azure Foundry team ran this benchmark of DeepSeek v4.1 flash (xhigh), comparing Azure with regular and encrypted inference on BLACKBOX AI. In the reported runs, Blackbox’s encrypted path delivered 245.88 tokens/s mean generation throughput and 30.43 seconds mean query latency, versus 81.47 tokens/s and 122.15 seconds on Azure.
3.02×
mean throughput vs. Azure
4.01×
lower mean query latency
0.379 s
encrypted mean TTFT
These ratios compare the encrypted Blackbox path with Azure. The regular Blackbox path was faster still on mean throughput and query latency. The results below preserve all reported measurements, including the request counts: 230 on Azure, 179 on regular Blackbox, and 330 on encrypted Blackbox.
Time to first token
Time to first token (TTFT) measures how long a request waits for the first token, as reported by the benchmark. Mean TTFT was 10.71 seconds on Azure, 0.399 seconds on regular Blackbox, and 0.379 seconds on encrypted Blackbox. Azure’s P99 was 51.385 seconds, compared with 0.544 and 0.440 seconds on the two Blackbox paths.
Figure 1. First-token latency
Azure and encrypted Blackbox · time to first token (s)
Source: Azure Foundry team benchmark summary. All bars share a zero baseline and one linear scale. Exact values are labeled and available on hover or tap. Request counts: Azure 230; Blackbox encrypted 330.
At P99, the reported Azure TTFT is about 116.8 times the encrypted Blackbox value. Percentiles describe each run separately; they do not pair individual requests.
End-to-end query latency
Mean query latency fell from 122.15 seconds on Azure to 26.82 seconds on regular Blackbox and 30.43 seconds on encrypted Blackbox. At P99, the corresponding values were 414.70, 55.99, and 62.33 seconds.
Source: Azure Foundry team benchmark summary. All bars share a zero baseline and one linear scale. Request counts: Azure 230; Blackbox regular 179; Blackbox encrypted 330.
Encrypted Blackbox’s mean query latency was 13.5% higher than regular Blackbox’s, an absolute difference of 3.61 seconds. Its P99 was 11.3% higher. Both Blackbox paths had lower latency than Azure at every reported summary statistic.
Generation throughput
Regular Blackbox averaged 283.74 tokens/s, while encrypted Blackbox averaged 245.88 tokens/s. Azure averaged 81.47 tokens/s. Those are 3.48× and 3.02× the Azure mean, respectively.
Source: Azure Foundry team benchmark summary. All bars share a zero baseline and one linear scale. Request counts: Azure 230; Blackbox regular 179; Blackbox encrypted 330.
The encrypted path’s mean throughput was 13.3% lower than the regular path’s. Across the reported throughput percentiles, the decrease ranged from 12.6% to 14.1%. Higher throughput percentiles describe the faster end of the throughput distribution; they are not the equivalent of high-percentile latency tails.
What the encryption comparison measures
The source labels its final column “Encryption cost.” For latency, that is 100 × (encrypted / regular − 1). For throughput, the source reverses the sign to report a loss: 100 × (1 − encrypted / regular). Emitted-character changes use the first formula. All percentages below are transcribed from the source and rounded to one decimal place.
Encrypted TTFT was lower at every reported statistic, including a 5.0% lower mean. That observation does not establish that encryption itself improves TTFT. The two Blackbox runs have different request counts, and the supplied summary does not document matched prompts, concurrency, hardware, region, sampling settings, cache state, run order, or repeat trials.
Mean output length was similar: 34,522 characters on Azure, 34,435 on regular Blackbox, and 33,846 on encrypted Blackbox. The encrypted mean was 1.7% below regular Blackbox. Character counts provide output-length context, but they do not establish equal token counts, equivalent answers, or equal model quality.
Quality across the shared datasets
The supplied quality summaries cover 10 datasets on Azure and four on Blackbox. On the three shared datasets labeled with accuracy, the reported scores are close: MuSR ranges from 71.2% to 73.6%, tau2-telecom from 97.4% to 98.2%, and ChemBench from 70.4% to 70.7%. These are descriptive comparisons of the supplied runs; matching dataset names alone does not establish identical evaluation settings.
Figure 4. Azure and encrypted Blackbox
Reported accuracy (%) · every axis spans 0–100%
Figure 4. Azure and encrypted Blackbox
Dimension
Azure
Blackbox encrypted
MuSR
71.2%
73.6%
tau2
98.2%
97.4%
ChemBench
70.6%
70.4%
AzureBlackbox encrypted
Source: supplied quality summaries. tau2 denotes tau2-telecom. Azure uses Correct/Dataset. FrontierScience is excluded because its scoring definitions differ. Polygon area is not an aggregate quality score; use the bar chart for precise differences.
The radar chart shows the overall shape of reported accuracy on the three shared datasets. Its 0–100% axes preserve the scale, so small differences nearly overlap. The grouped bars below make the exact scores readable.
Figure 5. Reported accuracy on shared datasets
Percent correct · Azure uses the full-dataset denominator
Source: supplied Azure and Blackbox quality summaries. Zero baseline and shared scale. Azure Correct/Dataset is used, not Correct/Judged. FrontierScience is excluded because the tables use different score definitions. No confidence intervals or matched-run claims are inferred.
Using the reported full-dataset percentages, encrypted Blackbox is 2.4 percentage points above Azure on MuSR, 0.8 points below on tau2-telecom, and 0.2 points below on ChemBench. Relative to regular Blackbox, the corresponding differences are +0.8, −0.8, and −0.3 points.
FrontierScience has different scoring definitions
Blackbox reports a mean score including partial credit of 59.1% for regular inference and 57.7% for encrypted inference, with 74 and 71 fully correct answers out of 160 examples. Azure reports 90 correct, 155 judged, and 160 dataset rows, alongside 57.7% Correct/Judged and 55.9% Correct/Dataset. These Azure percentages are reproduced as supplied; they do not match simple division of the displayed counts (90/155 ≈ 58.1%; 90/160 ≈ 56.3%). The summary does not explain the discrepancy or establish that its score is comparable with Blackbox’s partial-credit metric.
Source: supplied Blackbox quality summary. 160 examples per path; fully correct answers: regular 74, encrypted 71. Mean scores include partial credit and must not be read as fully-correct accuracy. Azure is omitted because its score definition differs.
Evaluation conditions
Encrypted Blackbox recorded zero errors across 3,309 examples. Regular Blackbox recorded one error, on FrontierScience. Parallelism differed on three of the four datasets; only tau2-telecom used the same value on both paths. These settings do not establish a quality effect from encryption.
Source: supplied Blackbox quality summary. Parallelism is a run setting, not a quality score or throughput measurement. Example counts per path: MuSR 250; tau2-telecom 114; FrontierScience 160; ChemBench 2,785.
Full performance results
The following tables reproduce the supplied performance summary for DeepSeek v4.1 flash (xhigh).