400 held-out BFCL v4 tasks, not used to train or to choose a recipe
| Entry | Value |
|---|---|
| Blackbox AI Fine-tuning | 54.0% |
| Together AI Fine-tuning | 48.0% |
| Baseline Qwen3.5-2B as released | 44.3% |
Same base model, same 4,750 ToolACE training conversations, same scorer on one NVIDIA H100 80GB. Blackbox AI fine-tuning vs Together AI fine-tuning: p = 0.001, exact McNemar test paired by task. October 24, 2026.
With the Blackbox AI fine-tuning API, we run agents that continuously fine-tune your model until it exceeds your performance targets. You provide the model, your training data, the tasks that matter and the scores you want to beat. Our agents train, evaluate each result, keep the improvements and use what they learn to guide the next round.
We ran it against Together AI fine-tuning, on the same model and data, scored on the same tasks.
The setup
Model. Qwen3.5-2B, which answers 44.3% of these tasks correctly before any training. We chose a small model so there was room to improve: on the larger Qwen3.5-9B, which already answers 68% of these tasks, neither side's fine-tune made a real difference.
Data. 4,750 tool-calling conversations from ToolACE, the same rows for both sides.
Tasks. 500 BFCL v4 tool-calling tasks, drawn across 14 categories with a fixed seed. Our agent saw scores on 100 of them to choose between its tries. Every accuracy in the headline chart and the results table comes from the other 400, which neither side trained on or used to choose a recipe. The try-by-try scores below are on the 100 selection tasks.
Together AI fine-tuning. One fine-tuning job on the same rows: a LoRA adapter (rank 64), one pass over the data, learning rate 1e-5.
Blackbox AI fine-tuning. An agent tested 11 recipes on one NVIDIA H100 80GB.
Continuous fine-tuning, driven by your targets
Blackbox AI fine-tuning handles the training, evaluation and iteration. Our agents use your task scores to decide what to change next and keep fine-tuning until the model exceeds the targets you set.
1. Set the targets. Define the tasks that matter and the scores the model needs to exceed. Blackbox AI fine-tuning evaluates the starting model to establish a baseline.
2. Choose the next experiment. The agent adjusts the training recipe, from learning rate and adapter size to data order and batch size, based on the results so far.
3. Train and evaluate. Each experiment trains an adapter and measures its performance on the same selection tasks, using a fixed scoring harness.
4. Keep the improvements. The agent retains the best result and discards changes that fail to improve it.
5. Repeat until the targets are exceeded. Each result informs the next experiment. We run the agents and manage the GPU infrastructure throughout the process.
What the agent tried
The starting recipe made the model worse: 46% on the selection tasks, against 51% for the baseline model. On its second try the agent cut the learning rate tenfold, to 2e-5, and reached 59%. Nine more ideas, including data order, adapter size, batch size, context length and training time, didn't beat it, so we returned try 2.
Score on the agent's 100 selection tasks, by try
| Try | Value |
|---|---|
| Baseline | 51% |
| 1 | 46% |
| 2 | 59% |
A try is kept only if it beats the best so far; a tie counts as not better. These are selection scores, used to choose a try. The headline numbers come from the 400 held-out tasks.
Results
Blackbox AI fine-tuning vs Together AI fine-tuning: +6.0 points, p = 0.001. Blackbox AI fine-tuning answered 38 tasks correctly that the Together AI fine-tuned model got wrong, and the reverse happened on 14.
Blackbox AI fine-tuning vs the baseline model: +9.75 points (39 more tasks), p < 0.001.
Together AI fine-tuning vs the baseline model: +3.75 points (15 more tasks), p = 0.08, which could still be chance.
What the agent chose differently
Both sides trained LoRA adapters on the same 4,750 conversations. The Together AI fine-tuning job used a fixed recipe. Our agent tested each recipe on the selection tasks and kept the best one: rank 16, learning rate 2e-5 held constant, trained on the first 2,464 conversations.
The Together AI fine-tuning recipe differed in several ways: rank 64, learning rate 1e-5 on a cosine schedule, one full pass over the data. This run does not show which difference matters. Matching the recipe on Together AI means choosing each of those settings and testing the result yourself, which is the work Blackbox AI fine-tuning automates.