Microsoft presents: BC-Bench: Can coding agents solve real-world AL tasks?

Overview

The session explains how Microsoft built BC-Bench, an evaluation framework for coding agents working on Business Central AL tasks. It started about six months before the talk and follows SWE-bench methodology. Agents get a bug report and the code base from before the fix. Their fixes are checked against unit tests written by human engineers, in Docker environments of Business Central set up with BC Container Helper.

The talk covers the bug-fixing dataset and how it was filtered, results across models and agent harnesses, multi-run metrics, and the effect of AL-MCP. It then moves to a code review agent evaluated offline and monitored online, a self-improvement pipeline, the roadmap, and a Q&A on source code, data collection, costs and GitHub models.

Key points

• Bug-fix verification: the code base is reset to before the fix, the agent gets the same bug report and produces a fix, and the original human-written tests are run. Tests must fail before and pass after the fix, and code must build in Docker.
• Dataset: single-bug pull requests that include tests, from the public BC apps GitHub repo and a private Azure DevOps repo. About 20% inventory and 19% finance. Bug reports are manually verified for sufficient context.
• Leaderboard: latest models reach about 68% (another 55%) on bug fixing, versus 16% (another 5%) for the GPT-4 model at the bottom.
• Multi-run metrics: AI is non-deterministic, so they compare mean accuracy, a consistency metric (task resolved in all five runs) and a best-case metric (resolved in any run). Bug fixing reports mean and the consistency metric. Two or three runs can still give statistical significance.
• AL-MCP (compile and symbol search) with GitHub Copilot CLI and Opus 4.6 raised mean resolution rate about 5% (non-overlapping 95% CI) and pass@5 about 10%.
• With Opus 4.6 fixed, GitHub Copilot CLI and Claude Code showed no significant difference; model choice is probably more significant.
• Complexity: agents reach about 80% on bugs needing fewer than 10 lines of code, dropping to around 50% for larger fixes.

Features

AL objects mentioned

As heard in the captions. A name that matches one object page by exact type and name links to it; the others stay as named.

• other “BC-Bench” at 0:28
• other “BC Container Helper” at 10:30
• other “BC apps” at 10:30
• other “SWE-bench” at 5:08
• other “SweepBench” at 8:14
• other “AL-MCP” at 19:57
• other “GitHub Copilot CLI” at 20:11
• other “Opus 4.6” at 20:11
• other “Cloud Code” at 21:22
• other “GPT-4 2” at 22:31

Quotes

• 0:53 “the basic question we want to uh to answer is how good are coding agents. And specifically at uh AL tasks.”
• 1:04 “one of the things that uh would or is interesting to know is how much does it depend on the agent harness, right?”
• 8:14 “We started the effort about 6 months ago. That was heavily inspired by SweepBench. We followed the same methodology, but adapted for Business Central”
• 13:18 “So it’s 68% almost or uh another one is 55%. That’s almost 10 times higher.”
• 14:17 “The first part is the functional fix that actually fix the behavior and the second part is some union test to verify okay the”

Disclaimers in the video

• 3:13 coming-later: there’s a GPT-6 Codex. Well, there will be, but I don’t know when it will be released.
• 17:11 other: coding agents can have random behaviors due to non-deterministic AI
• 29:13 coming-later: we are going to refresh the data set with more issues for for the bug-fixing benchmarking from uh BC apps
• 31:42 preview: we are actually considering that on the leaderboard you saw that we demoed that we can potentially have a partner …
• 34:21 coming-later: we’re moving more and more things on on GitHub. So, I think these will be available soon

Presenters (as heard): Klaus, Horan, Harun, Claus, Haron.