Business Central Under the Hood episode 15: BC-Bench: How we evaluate AI on AL tasks

Overview

Episode 15 of Business Central Under the Hood explains how BC-Bench evaluates AI on AL tasks. The method is inspired by Sweep Bench: bugs are collected from internal repositories (Azure DevOps rather than GitHub), the code is restored to the state when the bug occurred, an agent tries to fix it, and the original unit tests validate the result. A task qualifies only if it resolves exactly one bug, has full context, and its tests are independent of the fix.

The team uses the benchmark to compare models and to decide whether tooling such as the AL2 MCP server, built by the developer experience team, is worth supporting. The speakers report an increase in resolution rate with the MCP server, but the confidence intervals overlap, so the effect is not conclusive. They also describe plans for more categories (test generation, code review, refactoring, possibly feature addition), running tests through the MCP server, a public release and a partner dataset.

Key points

• BC-Bench is an offline evaluation framework for coding agents on real AL engineering tasks. It is an internal developer tool, not a product feature. It currently covers bug fixing only, with 66 tasks at the time of recording and a dataset that keeps growing.
• Method (adapted from SWE Bench): take a historical bug, recreate the code state at that time from a base commit, let the agent fix it, and validate with the original unit tests. Tests run through AL-Go and BC Container Helper.
• Task criteria: a pull request must resolve exactly one bug, the bug must include full context, and the tests must not depend on the fix (for example, exact error messages).
• The AL2 MCP server brings AL language extension tools such as compile and symbol finding to agents. It showed a higher average resolution rate, but the confidence intervals overlap, so the improvement is not conclusive. A new version has been built based on BC-Bench feedback.
• Comparisons used a mini BC agent (about 200 lines of Python, with only PowerShell and Microsoft Foundry endpoint access) and a terminal/pipeline variant of the GitHub Copilot agent. Custom instructions and skills can also be varied.
• Claude Opus 4.5 is the best-performing model among those available through GitHub Copilot. The ranking is close to SWE Bench Python results, and AL performance is roughly equal to Python, even though there are over 2 million public Python repositories and only about 300 AL ones.
• Planned items: test generation, code review and refactoring categories, possibly feature addition, running tests via the MCP server, a public BC-Bench and a partner dataset. No timelines were given.

Features

Every feature below: status not stated.

AL objects mentioned

As heard in the captions. A name that matches one object page by exact type and name links to it; the others stay as named.

• other “AL2 MCP Server” at 13:33
• other “BC Bench” at 14:01
• other “mini BC agent” at 17:05

Quotes

• 0:41 “BC Bench is a offline evaluation framework that measures how coding agents performs on real world engineering tasks in Business Central or AL programming”
• 1:10 “if a model perform better than others in AL tasks or if we can provide custom instructions or different tooling to improve the coding”
• 1:49 “this is about measuring um how good the deep the energetic experience is when you do AL development”
• 4:24 “what they did is they go to GitHub and find some popular Python repositories. Uh they find previous pull requests done by human engineers”
• 9:13 “last time I checked is 66 tasks in the data set. Well obviously the number is growing every day”
• 9:48 “one of the criteria is it’s a pull request must be resolving exactly one bug. So so we know this PR is fixing this”
• 13:16 “we are currently focusing on tool that built by our developer experience team that’s called AL2 MCP server”
• 14:01 “having BC bench because we’ll know that we we we are going to invest into adding more tools to to the Visual Studio code”
• 14:41 “the short answer is not conclusively. We we run them we actually did a rigorous analysis on on this AL2 MCP server. We run”
• 14:52 “We did see an increase on the average accuracy or what we call resolution rate. But there is a overlap on the confidence interval.”
• 15:31 “We provided some feedback to the developer experience team and now we have a new version of AL2 MCP server.”
• 16:08 “If we figure out this tool doesn’t make any difference then we don’t need to support it anymore and then we can focus our”
• 17:56 “No, it is a well, it’s similar to GitHub copilot in Visual Studio code but designed for to be running in terminal and in”
• 21:00 “the next on the list is to run tests uh, through the MCP server to ask to allow coding agent to iterate on the”
• 26:59 “Cloud Opus 45 is the best model we have access so far and it outperforms than any other models we have access to through”
• 27:45 “the ranking we see for for PC bench is more or less the same, isn’t it? Yes. Yes. And that’s So that’s that’s comforting”
• 27:45 “the the ranking we see for for PC bench is more or less the same, isn’t it?”
• 28:17 “there is more or less you know, equivalence between you know, handling writing code in Python and writing code in AL”
• 29:59 “we aim to make BC Bench public. Um well, first of all a bench benchmark is not uh useful unless people can reproduce the”
• 30:31 “Um now with BC Bench they can verify if it helps in solving what bugs in AL.”
• 31:13 “where partners can simply plug in this own uh their own code base in in”

Disclaimers in the video

• 6:51 coming-later: So one day hopefully we’ll make it public
• 14:41 subject-to-change: We did see an increase on the average accuracy or what we call resolution rate but not conclusively
• 20:14 coming-later: hopefully we have much more work as a software engineer. We want to bring test generation
• 20:26 coming-later: we also want to build a a category called code review
• 21:00 coming-later: the next on the list is to run tests uh, through the MCP server
• 25:09 coming-later: we also have code refactoring on the agenda and potentially feature addition
• 26:19 coming-later: that’s a bit uh, probably in the future. We haven’t really thought about that yet
• 26:47 coming-later: agents looking at pages is more in the future
• 29:59 coming-later: we aim to make BC Bench public
• 31:13 coming-later: create a partner data set for BC-Bench is part of the plan

Presenters (as heard): Håkon, Erhard.