BLOG
3 min read

DeepSeek v4.1 flash: Azure Foundry benchmark results

A benchmark run by the Azure Foundry team: encrypted Blackbox delivers 3.02× Azure’s mean throughput, with 13.3% lower throughput than regular Blackbox.

The Azure Foundry team ran this benchmark of DeepSeek v4.1 flash (xhigh), comparing Azure with regular and encrypted inference on BLACKBOX AI. In the reported runs, Blackbox’s encrypted path delivered 245.88 tokens/s mean generation throughput and 30.43 seconds mean query latency, versus 81.47 tokens/s and 122.15 seconds on Azure.

3.02×
mean throughput vs. Azure
4.01×
lower mean query latency
0.379 s
encrypted mean TTFT

These ratios compare the encrypted Blackbox path with Azure. The regular Blackbox path was faster still on mean throughput and query latency. The results below preserve all reported measurements, including the request counts: 230 on Azure, 179 on regular Blackbox, and 330 on encrypted Blackbox.

Time to first token

Time to first token (TTFT) measures how long a request waits for the first token, as reported by the benchmark. Mean TTFT was 10.71 seconds on Azure, 0.399 seconds on regular Blackbox, and 0.379 seconds on encrypted Blackbox. Azure’s P99 was 51.385 seconds, compared with 0.544 and 0.440 seconds on the two Blackbox paths.

Figure 1. First-token latency

Azure and encrypted Blackbox · time to first token (s)

Lower is better
Figure 1. First-token latency
EntryValue
Mean Azure10.71 s
Mean Blackbox encrypted0.379 s
P50 Azure7.437 s
P50 Blackbox encrypted0.378 s
P90 Azure22.902 s
P90 Blackbox encrypted0.404 s
P95 Azure30.197 s
P95 Blackbox encrypted0.416 s
P99 Azure51.385 s
P99 Blackbox encrypted0.440 s
Mean
Azure
10.71 s
Blackbox encrypted
0.379 s
P50
Azure
7.437 s
Blackbox encrypted
0.378 s
P90
Azure
22.902 s
Blackbox encrypted
0.404 s
P95
Azure
30.197 s
Blackbox encrypted
0.416 s
P99
Azure
51.385 s
Blackbox encrypted
0.440 s

Source: Azure Foundry team benchmark summary. All bars share a zero baseline and one linear scale. Exact values are labeled and available on hover or tap. Request counts: Azure 230; Blackbox encrypted 330.

At P99, the reported Azure TTFT is about 116.8 times the encrypted Blackbox value. Percentiles describe each run separately; they do not pair individual requests.

End-to-end query latency

Mean query latency fell from 122.15 seconds on Azure to 26.82 seconds on regular Blackbox and 30.43 seconds on encrypted Blackbox. At P99, the corresponding values were 414.70, 55.99, and 62.33 seconds.

Figure 2. Query latency, from median to tail

Query latency (s)

Lower is better
Figure 2. Query latency, from median to tail
EntryValue
P50 Azure94.87 s
P50 Blackbox regular25.49 s
P50 Blackbox encrypted30.94 s
P95 Azure290.03 s
P95 Blackbox regular49.97 s
P95 Blackbox encrypted54.28 s
P99 Azure414.70 s
P99 Blackbox regular55.99 s
P99 Blackbox encrypted62.33 s
P50
Azure
94.87 s
Blackbox regular
25.49 s
Blackbox encrypted
30.94 s
P95
Azure
290.03 s
Blackbox regular
49.97 s
Blackbox encrypted
54.28 s
P99
Azure
414.70 s
Blackbox regular
55.99 s
Blackbox encrypted
62.33 s

Source: Azure Foundry team benchmark summary. All bars share a zero baseline and one linear scale. Request counts: Azure 230; Blackbox regular 179; Blackbox encrypted 330.

Encrypted Blackbox’s mean query latency was 13.5% higher than regular Blackbox’s, an absolute difference of 3.61 seconds. Its P99 was 11.3% higher. Both Blackbox paths had lower latency than Azure at every reported summary statistic.

Generation throughput

Regular Blackbox averaged 283.74 tokens/s, while encrypted Blackbox averaged 245.88 tokens/s. Azure averaged 81.47 tokens/s. Those are 3.48× and 3.02× the Azure mean, respectively.

Figure 3. Mean generation throughput

Generation throughput (tokens/s)

Higher is better
Figure 3. Mean generation throughput
EntryValue
Mean Azure81.47 tokens/s
Mean Blackbox regular283.74 tokens/s
Mean Blackbox encrypted245.88 tokens/s
Mean
Azure
81.47 tokens/s
Blackbox regular
283.74 tokens/s
Blackbox encrypted
245.88 tokens/s

Source: Azure Foundry team benchmark summary. All bars share a zero baseline and one linear scale. Request counts: Azure 230; Blackbox regular 179; Blackbox encrypted 330.

The encrypted path’s mean throughput was 13.3% lower than the regular path’s. Across the reported throughput percentiles, the decrease ranged from 12.6% to 14.1%. Higher throughput percentiles describe the faster end of the throughput distribution; they are not the equivalent of high-percentile latency tails.

What the encryption comparison measures

The source labels its final column “Encryption cost.” For latency, that is 100 × (encrypted / regular − 1). For throughput, the source reverses the sign to report a loss: 100 × (1 − encrypted / regular). Emitted-character changes use the first formula. All percentages below are transcribed from the source and rounded to one decimal place.

Encrypted TTFT was lower at every reported statistic, including a 5.0% lower mean. That observation does not establish that encryption itself improves TTFT. The two Blackbox runs have different request counts, and the supplied summary does not document matched prompts, concurrency, hardware, region, sampling settings, cache state, run order, or repeat trials.

Mean output length was similar: 34,522 characters on Azure, 34,435 on regular Blackbox, and 33,846 on encrypted Blackbox. The encrypted mean was 1.7% below regular Blackbox. Character counts provide output-length context, but they do not establish equal token counts, equivalent answers, or equal model quality.

Quality across the shared datasets

The supplied quality summaries cover 10 datasets on Azure and four on Blackbox. On the three shared datasets labeled with accuracy, the reported scores are close: MuSR ranges from 71.2% to 73.6%, tau2-telecom from 97.4% to 98.2%, and ChemBench from 70.4% to 70.7%. These are descriptive comparisons of the supplied runs; matching dataset names alone does not establish identical evaluation settings.

Figure 4. Azure and encrypted Blackbox

Reported accuracy (%) · every axis spans 0–100%

Figure 4. Azure and encrypted Blackbox
DimensionAzureBlackbox encrypted
MuSR71.2%73.6%
tau298.2%97.4%
ChemBench70.6%70.4%
MUSRTAU2CHEMBENCH

Source: supplied quality summaries. tau2 denotes tau2-telecom. Azure uses Correct/Dataset. FrontierScience is excluded because its scoring definitions differ. Polygon area is not an aggregate quality score; use the bar chart for precise differences.

The radar chart shows the overall shape of reported accuracy on the three shared datasets. Its 0–100% axes preserve the scale, so small differences nearly overlap. The grouped bars below make the exact scores readable.

Figure 5. Reported accuracy on shared datasets

Percent correct · Azure uses the full-dataset denominator

Higher is better
Figure 5. Reported accuracy on shared datasets
EntryValue
MuSR Azure71.2%
MuSR Blackbox regular72.8%
MuSR Blackbox encrypted73.6%
tau2-telecom Azure98.2%
tau2-telecom Blackbox regular98.2%
tau2-telecom Blackbox encrypted97.4%
ChemBench Azure70.6%
ChemBench Blackbox regular70.7%
ChemBench Blackbox encrypted70.4%
MuSR
Azure
71.2%
Blackbox regular
72.8%
Blackbox encrypted
73.6%
tau2-telecom
Azure
98.2%
Blackbox regular
98.2%
Blackbox encrypted
97.4%
ChemBench
Azure
70.6%
Blackbox regular
70.7%
Blackbox encrypted
70.4%

Source: supplied Azure and Blackbox quality summaries. Zero baseline and shared scale. Azure Correct/Dataset is used, not Correct/Judged. FrontierScience is excluded because the tables use different score definitions. No confidence intervals or matched-run claims are inferred.

Using the reported full-dataset percentages, encrypted Blackbox is 2.4 percentage points above Azure on MuSR, 0.8 points below on tau2-telecom, and 0.2 points below on ChemBench. Relative to regular Blackbox, the corresponding differences are +0.8, −0.8, and −0.3 points.

FrontierScience has different scoring definitions

Blackbox reports a mean score including partial credit of 59.1% for regular inference and 57.7% for encrypted inference, with 74 and 71 fully correct answers out of 160 examples. Azure reports 90 correct, 155 judged, and 160 dataset rows, alongside 57.7% Correct/Judged and 55.9% Correct/Dataset. These Azure percentages are reproduced as supplied; they do not match simple division of the displayed counts (90/155 ≈ 58.1%; 90/160 ≈ 56.3%). The summary does not explain the discrepancy or establish that its score is comparable with Blackbox’s partial-credit metric.

Figure 6. FrontierScience on Blackbox

Mean score including partial credit (%)

Higher is better
Figure 6. FrontierScience on Blackbox
EntryValue
Blackbox regular59.1%
Blackbox encrypted57.7%
Blackbox regular
59.1%
Blackbox encrypted
57.7%

Source: supplied Blackbox quality summary. 160 examples per path; fully correct answers: regular 74, encrypted 71. Mean scores include partial credit and must not be read as fully-correct accuracy. Azure is omitted because its score definition differs.

Evaluation conditions

Encrypted Blackbox recorded zero errors across 3,309 examples. Regular Blackbox recorded one error, on FrontierScience. Parallelism differed on three of the four datasets; only tau2-telecom used the same value on both paths. These settings do not establish a quality effect from encryption.

Figure 7. Parallelism during quality evaluation

Reported concurrent evaluation settings

Relative scale
Figure 7. Parallelism during quality evaluation
EntryValue
MuSR Blackbox regular4
MuSR Blackbox encrypted1
tau2-telecom Blackbox regular4
tau2-telecom Blackbox encrypted4
FrontierScience Blackbox regular4
FrontierScience Blackbox encrypted5
ChemBench Blackbox regular4
ChemBench Blackbox encrypted10
MuSR
Blackbox regular
4
Blackbox encrypted
1
tau2-telecom
Blackbox regular
4
Blackbox encrypted
4
FrontierScience
Blackbox regular
4
Blackbox encrypted
5
ChemBench
Blackbox regular
4
Blackbox encrypted
10

Source: supplied Blackbox quality summary. Parallelism is a run setting, not a quality score or throughput measurement. Example counts per path: MuSR 250; tau2-telecom 114; FrontierScience 160; ChemBench 2,785.

Full performance results

The following tables reproduce the supplied performance summary for DeepSeek v4.1 flash (xhigh).

Serving pathRequests
Azure230
Blackbox regular179
Blackbox encrypted330

TTFT (s)

StatisticAzureBlackbox regularBlackbox encryptedEncryption cost
Mean10.710.3990.379-5.0%
P507.4370.3950.378-4.3%
P9022.9020.4580.404-11.8%
P9530.1970.4820.416-13.7%
P9951.3850.5440.44-19.1%

Query latency (s)

StatisticAzureBlackbox regularBlackbox encryptedEncryption cost
Mean122.1526.8230.4313.5%
P5094.8725.4930.9421.4%
P90254.6944.0249.312.0%
P95290.0349.9754.288.6%
P99414.755.9962.3311.3%

Generation throughput (tokens/s)

StatisticAzureBlackbox regularBlackbox encryptedEncryption cost
Mean81.47283.74245.8813.3%
P5072.48278.18242.612.8%
P90135.4319.75277.2113.3%
P95163.7346.12302.6112.6%
P99219.21378.94325.6514.1%

Total emitted characters (characters/response)

StatisticAzureBlackbox regularBlackbox encryptedEncryption cost
Mean34,52234,43533,846-1.7%
P5033,99534,69535,6822.8%
P9056,04655,04352,794-4.1%
P9560,03059,22358,458-1.3%
P9972,60469,34066,169-4.6%
The most secure infrastructure from training to inference

Ready to serve your first token?

Tell us the workload and the controls it has to satisfy. We come back with a deployment plan and a per-token commit.