A coding agent operates as an iterative control loop. At each iteration, it sends the accumulated conversation to a model, the model selects one action (open a file, run tests or edit code), and the resulting observation is appended to the conversation. Blackbox defines each model invocation as a step. Resolving one defect typically requires 20 to 80 steps.
Most steps are easy, such as listing files or rerunning a test, and do not need an expensive frontier model. A router looks at each step and sends it to the cheapest model that will still get it right. That saves money on every session, as long as the router itself is fast and cheap.
Blackbox used Microsoft Decision-1 as the routing model and optimized both the feature questions and the aggregation function applied to its outputs. The work produced two routers: a first router built on one coding agent with 20 questions, and a multi-agent router that extends it to eight coding agents with 44 questions. The results below state which one they use.
Share of steps where the router's classification matches GPT-6 Astra's label · higher is better
| Model | Value |
|---|---|
| Always small no router | 52% |
| Decision-1 asked directly | 49% |
| Jev asked directly | 57% |
| Qwen3.8 large prompted LLM | 62% |
| Decision-1 20 questions | 65% |
| Jev 20 questions | 71% |
| Decision-1 first router | 73.6% |
First six columns: Blackbox's initial 315 steps. First router: 20 questions and a tree model optimized on 2,540 steps. Every column is scored on repositories excluded from training.
A pick counts as right when it matches the tier GPT-6 Astra assigned to that step; the labels are explained below. Always picking the small model is already right about half the time, so that is the bar to beat.
Decision-1 is also the fastest router. One call returns all 20 answers, so a routing decision takes less than half a second. The multi-agent router's 44 questions add about 0.1 s to the same call. These times cover the Decision-1 call itself; measured end to end through the live gateway, a decision takes about 0.5 s.
Comparison with OpenRouter Auto Router
OpenRouter's Auto Router classifies the full prompt into one of about 30 task types and selects the model the OpenRouter community spends the most on for that type over the trailing seven days, keeping the same model for the rest of a session. Blackbox evaluated it against the multi-agent router on 1,200 locked test steps from the datasets described below (300 per set), restricted to the same three models (one closed, one big open-weight and one small open-weight; the tiers are defined below), and scored it against the same labels. The chart uses a 300-step sample per set; the multi-agent router's results on the full sets are in the Multi-Agent Optimization table.
The Auto Router selected Kimi K3 on about 97% of steps, so it was correct only on steps that require the big tier. Its choice reflects what the community is willing to pay for a task type, not the least-expensive model that completes a given step. Per-step routing captures the routine steps that a small model handles.
Terminal-Bench 2.0: Live Agent Runs
Matching labels is one measure; the outcome that matters is whether an agent still finishes its tasks. Terminal-Bench 2.0 is a public benchmark of 89 terminal tasks, each checked by automatic pass/fail tests. Blackbox ran a live agent on held-out Terminal-Bench 2.0 tasks with the multi-agent router, changing only which model handles each step. On the 16 tasks where every configuration produced a valid result, the router-driven agent completed 75% of tasks, compared with 69% when every step used Claude Opus; with one attempt per task, the two are within noise. Blackbox ran every configuration, including the Claude Opus baseline, on the same tasks, harness and infrastructure, so only the routing differs. These results are not drawn from public leaderboards and are not directly comparable to them.
The datasets
Every step comes from a real coding-agent session in a public dataset. Blackbox segmented each session into model invocations and used GPT-6 Astra to label the least-expensive tier expected to produce a correct next action.
SWE-smith [1]. SWE-agent sessions on defects injected into Python repositories. Blackbox optimized the router on 2,540 steps from 330 sessions across 125 repositories and reserved 765 steps from 100 held-out sessions for evaluation. These sessions were never used in training, but their repositories also appear in the training set.
Nebius SWE-agent [2]. The same agent on real GitHub issues, using an older version of its prompt and command format.
SWE-Gym OpenHands [3]. OpenHands, a different coding agent with its own prompts and tools, solving real GitHub issues from 11 Python repos.
SWE-rebench OpenHands [4]. OpenHands with native tool calls, the request format a production agent sends to an inference gateway, on real GitHub issues from several hundred repositories.
Terminal-Bench 2.0 [5]. Public trajectories from the Terminal-Bench 2.0 leaderboard: general terminal tasks such as building interpreters, cracking archives or training models, run by Claude Code, Codex, Gemini CLI, OpenHands, mini-swe-agent and Terminus. Two further agents, Judy and deepagent-harbor, are held out of training entirely.
The three tiers
Every step is assigned to one of three cost-ordered tiers. A closed model is a frontier model available only through its vendor's API, such as GPT-6 Astra or Claude Opus 5.5, the closed model in the live runs and the OpenRouter comparison. A big open-weight model has public weights that Blackbox can self-host, such as Kimi K3 or Nemotron 3 Ultra. A small open-weight model is the lowest-cost option, such as Nemotron 3.5 Lightning or gpt-oss.
Router Overview
Decision-1 is a 9B decision model from Microsoft. It does not generate conversational responses. It evaluates typed questions over an input sequence by selecting an option or estimating the probability of a binary statement. Blackbox compared it with TypeSafe Jev (jev-1.13), a decision model with the same API and price.
For every step, the router does three things:
1. Ask 20 yes/no questions in one call. Decision-1 reads the task, the recent messages and the latest tool output, and returns a probability for each question, from 0 (surely no) to 1 (surely yes). For example: "Does the latest output contain a traceback?", "Has the agent already tried a fix that still fails?" or "Will the next action edit source code?"
2. Add three counters. The turn number, the size of the conversation so far and the number of tool errors. These are read straight from the step, without any model call.
3. Turn the 23 numbers into a tier. A small tree model (300 shallow decision trees, a method called gradient boosting) takes the 20 answers and 3 counters and picks one tier. Each tree is a few if-then rules, such as "if a fix was tried and still fails, and the conversation is long, lean toward closed". The trees' scores are added up, and the tier with the highest total wins. It runs inside the BLACKBOX AI gateway in under a millisecond.
The 20 questions are manually specified. The gradient-boosted tree is the only learned component: Blackbox optimized it on 2,540 labeled steps to estimate which feature combinations require a higher-capability model tier. This describes the first router; the multi-agent router at the end of this post keeps the same design with more questions and counters.
Scoring Methodology
No human-generated tier labels are available, so Blackbox constructed reference labels with a frontier model.
Judge labels. GPT-6 Astra receives the same context as the router and identifies the least-expensive tier expected to produce the correct next action. The 73.6% result and all classification-accuracy values use these labels unless otherwise noted.
Repository-grouped cross-validation. Blackbox partitions repositories into five folds, optimizes the tree model on four folds and evaluates it on the fifth, then rotates the held-out fold. Every step is evaluated exactly once by a router that received no training samples from the same repository.
Ablation Study
Directly asking Decision-1 which tier a step requires produces 49% accuracy, below the always-small baseline. Prompt rephrasing and solved examples yield negligible improvement. Replacing the single subjective query with narrow state-feature questions and aggregating their probabilities produces the measurable gain. Each configuration below used the same evaluation protocol for both decision models on Blackbox's initial 315-step dataset:
Jev gains from the same changes and is better on hard steps: in the best early setups it caught 26 of the 32 steps labeled closed, against 19 for Decision-1. Qwen3.8 large, prompted as a router, reached 62% but labeled about 90% of steps small.
Blackbox then expanded the labeled dataset eightfold to 2,540 steps and replaced the linear heuristic with the gradient-boosted tree. On this dataset, counters alone reach 55%, Decision-1 probabilities with the heuristic reach 69%, and the same probabilities with the tree model reach 73.6%. This is the first router, the configuration first integrated into the BLACKBOX AI gateway.
The same 2,540 steps, each scored by a router that never saw its repo · higher is better
| Model | Value |
|---|---|
| Always small no router | 51.8% |
| Counters only no Decision-1 | 55.2% |
| Decision-1 answers simple formula | 68.8% |
| Decision-1 answers tree model | 73.6% |
Counters are the turn number, the conversation size and the tool errors so far. Both Decision-1 columns use the same single call with 20 questions; only the way the answers are combined differs.
Automated Feature-Question Search
Before collecting more labeled data, Blackbox also tried letting a model write the questions: over about 370 experiments, Kimi K3 proposed new yes/no questions, and a change was kept only if it raised accuracy. The gains did not hold up on eight repositories held back for checking, so the first router keeps the 20 hand-written questions; more labeled data improved accuracy far more than new questions did.
Error Analysis
Label Distribution
Per-Tier Recall
2,540 steps, 125 repos, 5-fold cross-validation grouped by repo · higher is better
| Model | Value |
|---|---|
| Small open-weight steps that need small | 84.5% |
| Big open-weight steps that need big | 66.6% |
| Closed steps that need closed | 51.2% |
| Right or stronger all steps | 84.6% |
Each column is the share of steps of that tier the router sends to that tier. Right or stronger counts a step as safe when the router picks the correct tier or a more capable one.
Small steps are the easiest to spot. Closed steps are the hardest, and most misses there go to the big tier. Rows are the correct tier, columns are the router's pick:
Routing Distribution by Action Type
Grouped by what the agent actually did next, the router keeps most commands and submits on the small tier and sends code edits up:
Feature Importance
Relative contribution of each input to the tree model, top 8, percent of total
| Model | Value |
|---|---|
| Fix tried and still failing | 15.1% |
| Context size | 10.9% |
| Turn number | 8.3% |
| Latest output shows the fix working | 8.2% |
| Next action writes code | 7.0% |
| Next action runs a command | 6.5% |
| Next action creates a file | 5.2% |
| Last action edited a source file | 4.6% |
Context size and turn number are counters. The other six are Decision-1 yes/no answers. Top signal in full: Has the agent already tried to fix the problem at least once, and the latest output shows it still failing?
Representative Routing Decisions
One correct, high-confidence pick per tier, each made by a router that never saw that repo. Excerpts are trimmed.
What the router saw (first step of the task, no tool output yet)
[coding agent call] turn=2 prior_tool_calls=0 prior_tool_errors=0 context_chars=1424
Decision-1 answers
0.99 Will the agent's next action most likely be opening or viewing a file?
0.01 Has the agent already found the exact file and function that need to change?
0.00 Has the root cause of the bug or the exact change needed already been identified in the conversation?
Router pick small open-weight (99.4% confidence), matches the label
Agent did viewed /testbed/pydantic/_internal/_known_annotated_metadata.pyWhat the router saw
[coding agent call] turn=14 prior_tool_calls=6 prior_tool_errors=3 context_chars=31919
[tool_call bash] {"command": "cd /testbed && python reproduce_error.py"}
...
File "/testbed/src/pythonjsonlogger/jsonlogger.py", line 273, in format
message_dict["exc_info"] = self.formatException(record.exc_info)
AttributeError: 'JsonFormatter' object has no attribute 'formatException'
Decision-1 answers
1.00 Does the most recent tool output contain a Python traceback or exception?
0.00 Does the latest output show the fix working (script or tests now pass)?
1.00 Will the agent's NEXT action most likely be writing or modifying source code (not just viewing, searching, or running)?
Router pick big open-weight (99.0% confidence), matches the label
Agent did created /testbed/reproduce_format_error.pyWhat the router saw
[coding agent call] turn=112 prior_tool_calls=55 prior_tool_errors=3 context_chars=116831
Let's run the reproduce script again to verify the fix:
[tool_call bash] {"command": "python /testbed/reproduce_bug.py"}
OBSERVATION:
Test case 1 cycle: []
Test case 2 cycle length: 3
Test case 3 cycle length: 2
Decision-1 answers
0.99 Did the agent's most recent action modify a source file?
0.98 Has the agent already tried to fix the problem at least once, and the latest output shows it still failing?
0.03 Does the latest output show the fix working (script or tests now pass)?
Router pick closed (96.9% confidence), matches the label
Agent did edited getcycle in /testbed/dask/core.pyRobustness Evaluation
Blackbox froze the first router without further parameter updates and evaluated it on previously unseen data.
Share of steps where the frozen router picks the right model tier · higher is better
| Model | Value |
|---|---|
| Original steps 2,540 steps · always small 51.8% | 73.6% |
| Held-out sessions 765 steps, repos seen in training · always small 49.7% | 76.3% |
| Other models Nebius SWE-agent, 380 steps · always small 42.1% | 66.3% |
| Other agent OpenHands, 400 steps · always small 47.0% | 60.8% |
The exact router deployed in the gateway, unchanged, in every column. New tests called Decision-1 through OpenRouter; it matched Blackbox's earlier endpoint to within a median of 0.007.
Accuracy holds on held-out sessions from the training agent. It drops on sessions from other models and other agents, which the multi-agent router below addresses.
The same 765 steps from held-out sessions, scored against different reference labels · higher is better
| Model | Value |
|---|---|
| GPT-6 Astra labels always small 49.7% | 76.3% |
| Second judge model always small 63.9% | 73.2% |
| Both judges agree 601 steps · always small 61.6% | 82.5% |
The two judges assign the same tier on 78.6% of steps, which bounds achievable classification accuracy.
Multi-Agent Optimization
The router above was optimized on a single agent. Blackbox extended the optimization to every agent in the dataset table, with three changes: steps from Nebius SWE-agent, OpenHands with native tool calls and seven Terminal-Bench agents were added to the training set; the tool-error counter now also reads tool results returned as plain user messages, which text-format agents use; and 24 binary questions were added to the same single Decision-1 call, for 44 in total. They came in two rounds of 12: the first targets agent state that the original 20 missed on other coding agents, such as an agent repeating a failing command; the second was added after the Terminal-Bench results below, and targets general task difficulty and the kind of work the next step involves. The counters grew from three to five, adding the number of tool results and whether the agent uses native tool calls.
Each locked test set uses repositories or tasks that appear in no training file, and its labels were produced only after the router was frozen.
On sessions from Claude Code, Codex, Gemini CLI and four other agents (Terminal-Bench 2.0), the first router picked the right tier 56% of the time, barely above always choosing the small model (53%). The multi-agent router raises this to 70%, with the same 70% on two agents excluded from training. On SWE-style agents, accuracy on held-out repositories reaches 76 to 80%. The general-task questions account for most of the Terminal-Bench gain: the original feature questions target repository bug fixing, while Terminal-Bench tasks range from cryptography to model training.
Sources
[1] SWE-bench/SWE-smith-trajectories, tool split.
[2] nebius/SWE-agent-trajectories.
[3] SWE-Gym/OpenHands-SFT-Trajectories.
[4] nebius/SWE-rebench-openhands-trajectories.
[5] yoonholee/terminalbench-trajectories, Terminal-Bench 2.0 leaderboard trajectories.