What's New: BC-Bench (2026 release wave 1)
BC-Bench is a benchmarking tool for AI coding agents fixing real Business Central AL bugs from Microsoft's backlog. The video covers how it works, the leaderboard metrics (mean resolution rate, pass at five, execution time), and how to access or fork the MIT-licensed GitHub repository.
- Introduction to BC Bench and its purpose
- Why Business Central needs its own benchmark
- How BC Bench works with bug-fixing methodology
- Getting started with BC Bench and accessing the tool
- Leaderboard results and interpreting the metrics
- Model improvements over time and customization options
Overview
The video presents BC-Bench, a reproducible evaluation framework that tests which combination of LLM model, MCP server, instructions and agent harness works best for AL development. General coding benchmarks such as SWE-bench do not reflect what a Business Central developer faces, and the AL dataset is smaller (330 tasks at recording time). Each task is built from a real bug and includes the problem statement, the unit test written by the BC developer, and the expected fix.
The presenter explains the leaderboard: results are aggregated over five runs because language models are non-deterministic. Overlapping confidence intervals mean two setups cannot be confidently ranked. The video also covers the source repository, the supported harnesses, model comparison across Anthropic and OpenAI models, and how to contribute or fork with a custom dataset.
Key points
- BC-Bench tests combinations of LLM model, MCP server, instructions and agent harness on real Business Central bugs from Microsoft’s backlog.
- Each benchmark task contains the bug description and repro steps as the problem statement, the BC developer’s unit test, and the expected fix.
- Mean resolution rate is averaged over five runs and shown with a 95% confidence interval; one example setup reached 68.5%.
- If confidence intervals overlap, the setups cannot be confidently compared, since the difference could be a fluke; non-overlapping intervals showed Opus 4.6 is an improvement over Opus 4.5.
- Pass at five measures how consistently the agent resolves the same tasks across all five runs; average execution time is also reported.
- Compared with a GPT-4.1 run, the pass at five metric increased tenfold in about 12 months.
- A version column records BC-Bench and agent harness updates instead of rerunning all results after every update.
Features
| Feature | Status | At |
|---|---|---|
| BC Bench benchmarking tool | status not stated, demoed | 0:40 |
| BC Bench bug-fixing dataset | status not stated, demoed | 4:00 |
| Mean resolution rate with confidence intervals | status not stated, demoed | 7:02 |
| Pass at five consistency metric | status not stated, demoed | 9:22 |
| Average execution time metric | status not stated, demoed | 9:58 |
| GitHub repository access for BC Bench | status not stated, demoed | 4:57 |
| Model comparison across AI providers | status not stated, demoed | 6:28 |
| Custom dataset contribution capability | status not stated | 12:02 |
Quotes
- 0:40 “To find out which combination is the best, we’ve developed a benchmarking tool that lets us test all these combinations as an agent solves”
- 2:21 “The reality here is that general coding benchmarks like SweetBench do not reflect the realities that a Business Central developer faces every day.”
- 3:01 “Whereas in AL, we only have 330 as of the time of this recording.”
- 7:27 “This specific setup managed to achieve 68.5% resolution rate on average, meaning on average of five runs, it resolved 68% of the all the”
- 8:47 “We cannot confidently say which one of the setup is better because, well, it could be a fluke because of those five runs.”
- 10:53 “So, if you look at the pass at at five metrics, uh it increased tenfold uh just in 10 month 12 months.”
Disclaimers in the video
- 0:21 other: As more and more development in Business Central AL is done by agents or assisted by agents, there’s a need to find out which model which MCP servers, which instructions, and which agent harness is the best combination.
- 7:50 other: There are lots of fluctuations on the accuracy between runs because AI models or language models are non-deterministic.
- 10:39 other: By the time you are watching this video, you will be exactly a year before GPT-4o model was released.
Presenters (as heard): Joost, Horan.