Across MuSR, tau2-telecom, FrontierScience, and ChemBench, E2EE Blackbox scored within 1.4 percentage points of regular Blackbox, with zero reported errors.
E2EE Blackbox scored within 1.4 percentage points of regular Blackbox on each of four datasets: MuSR, tau2-telecom, FrontierScience, and ChemBench. The E2EE runs recorded zero errors across 3,309 examples, compared with one error in the regular runs.
4
datasets compared
3,309
examples per path
1.4 pp
largest score gap
0
E2EE run errors
This comparison covers regular inference and end-to-end encrypted inference (E2EE) on BLACKBOX AI. MuSR scored slightly higher on the E2EE path; the other three datasets scored slightly lower. Here are the results and the conditions that matter when interpreting them.
Scores at a glance
Regular and E2EE, side by side
Reported score (%) on each dataset
Higher is better
Regular and E2EE, side by side
Entry
Value
MuSR · accuracy Regular
72.8%
MuSR · accuracy E2EE
73.6%
tau2-telecom · accuracy Regular
98.2%
tau2-telecom · accuracy E2EE
97.4%
FrontierScience · mean score with partial credit Regular
59.1%
FrontierScience · mean score with partial credit E2EE
Source: reported evaluation summary. All bars share a zero baseline and scale. Compare paths within each dataset; FrontierScience uses a different scoring metric.
E2EE minus regular, the changes are +0.8 percentage points on MuSR, -0.8 on tau2-telecom, -1.4 on FrontierScience, and -0.3 on ChemBench, calculated from the reported rounded scores. FrontierScience includes partial credit, so its mean score should not be treated as accuracy or pooled with the other datasets into a single overall score.
Different run settings
Parallelism on the E2EE path
Regular Blackbox used parallelism 4 on every dataset.
Reported run settings, not measured throughput. Parallelism was matched only on tau2-telecom. These values do not establish a speed difference between the serving paths.
Inside each dataset
The score gaps correspond to small differences in correct-answer counts. Here is how those outcomes break down.
Correct-answer counts from the reported evaluation. Both columns start at zero. Neither run reported an error.
The E2EE run answered two more examples correctly out of 250, a gain of 0.8 percentage points. Neither run reported an error. Parallelism differed: 4 for regular Blackbox and 1 for E2EE Blackbox.
tau2-telecom
112
regular correct
111
E2EE correct
114
examples per path
At the same reported parallelism of 4, the E2EE run answered one fewer example correctly out of 114. Reported accuracy moved from 98.2% to 97.4%, with no errors on either path.
Counts shown in legend order. Remaining examples are total minus fully correct and include any partially correct outcomes or errors. This is not the partial-credit mean score of 59.1% versus 57.7%.
The E2EE run had three fewer fully correct examples. Its mean score, including partial credit, was 57.7% versus 59.1%, the largest observed score difference in this comparison at 1.4 percentage points. Regular Blackbox reported one error; E2EE Blackbox reported none. The summary does not identify the error's cause.
ChemBench
ChemBench: E2EE outcomes
2,785 examples with E2EE
ChemBench: E2EE outcomes
Segment
Value
Correct
1,962
Not correct
823
70.4%Accuracy
Correct1,962
Not correct823
823 not-correct answers = 2,785 total minus 1,962 correct. Zero errors were reported. Regular Blackbox answered 1,968 correctly out of the same total, at 70.7% accuracy.
On the largest dataset in this comparison, the E2EE run answered six fewer examples correctly out of 2,785. Reported accuracy differed by 0.3 percentage points, with zero errors on either path. Parallelism was 4 for regular Blackbox and 10 for E2EE Blackbox.
What these runs establish
The observed scores are close, and E2EE Blackbox completed the evaluated examples without reported errors. Zero errors does not mean every answer was correct: errors and answer quality are separate metrics in these results.
This is a comparison of the reported runs, not a statistical equivalence result. The evaluation summary does not specify the model or checkpoint, dataset versions and splits, harness configuration, prompts, sampling settings, evaluation date, or repeated-run measurements. Without those details and per-example outcomes, the score differences cannot be attributed to E2EE or distinguished reliably from run-to-run variation.
Parallelism also differs on three of the four datasets. Those settings describe the runs; they are not throughput measurements. No latency, tokens-per-second, or elapsed-time results are included, so this comparison does not quantify the speed difference or E2EE overhead.
Evaluate the E2EE path on your workload
These four datasets provide a useful starting point for comparing regular and E2EE inference. A stronger comparison would fix the model, harness, prompts, sampling, and parallelism across both paths, repeat the runs, and measure latency alongside answer quality.