What's new in BC-Bench (2026 release wave 2)
BC-Bench is an open-source, reproducible evaluation framework for AI coding agents on real AL development tasks (bug fixing, test generation, code review). The video covers bug fixing results with the AL MCP server, a harness vs model comparison, a contamination check, the new code review category judged by an LM-as-judge, and how partners can fork the repo to benchmark their own AL apps.
- Introduction to BC-Bench and its purpose
- Why BC-Bench matters and coverage areas
- Bug fixing category and AL MCP server
- Bug fixing results and agent harness comparison
- Contamination detection strategy
- Contamination detection results
- Code review category introduction and approaches
- Code review methodology and results
- How to use BC-Bench for your own AL apps
Overview
BC-Bench is an evaluation framework for AI coding agents working on AL development, inspired by SWE-bench. It covers bug fixing, test generation and PR reviews. The video walks through the bug fixing category, which uses 101 tasks taken from real BC app repositories where human engineers solved the problems, and shows how the AL MCP server affects results.
It then explains how the team checks whether models memorized the public data set, and introduces the code review category with three approaches. Code review results were still being finalized at recording time. The last chapter covers how to use BC-Bench for your own AL apps.
Key points
- Bug fixing category: an agent fixes a bug in the real codebase, using 101 tasks from real BC app repositories solved by human engineers.
- The AL MCP server exposes compiling, publishing and simple search to coding agents, so they can work without opening Visual Studio Code.
- With GitHub Copilot CLI and Claude Opus, the AL MCP server improved pass@5 by almost 10% and clearly improved mean resolution rate. Tasks take longer because agents publish apps and run tests.
- Comparing agent harnesses (GitHub Copilot CLI vs Claude Code) with the same models showed no significant difference, which suggests model choice matters more than harness choice.
- Contamination check: three models with knowledge cutoffs after BC-Bench went public were tested, and only 1 of 101 tasks matched across all three. Agents can deduce file paths from a bug description without having memorized the solution. Contamination will be monitored over time.
- Code review category evaluates AI review of AL pull requests with three approaches: baseline GitHub Copilot UI, the BC Quality plugin on Copilot CLI, and a specialized AL review agent focused on security, privacy and performance.
- Code review quality is judged with an LM-as-judge that checks whether generated comments are semantically equivalent to the expected ones. Final results were not ready at recording time and are to be checked on the GitHub repository.
Features
| Feature | Status | At |
|---|---|---|
| Bug fixing category | status not stated, demoed | 2:46 |
| AL MCP server | status not stated | 3:19 |
| Pass@5 improvement with MCP | status not stated, demoed | 3:46 |
| Agent harness comparison | status not stated, demoed | 4:40 |
| Contamination detection strategy | status not stated, demoed | 5:44 |
| Contamination results across three models | status not stated, demoed | 7:38 |
| Code review category | status not stated, demoed | 8:34 |
| BC Quality plugin | status not stated, demoed | 9:47 |
| AL review agent | status not stated, demoed | 10:20 |
| LM-as-judge for code review evaluation | status not stated, demoed | 12:24 |
Quotes
- 0:19 “BCBench is a reproducible evaluation framework for AI coding agents working on real world AI development task inspired by SVBench”
- 1:46 “BCbench covers the following categories of AL development work. First of all, there’s bug fixing and there’s test generation and also there’s PR reviews”
- 4:17 “the pass head five matrix improved almost 10 uh and there’s also a significant improvement”
- 4:57 “So the choice of models probably matter more than the choice of agent harness at least using our data set.”
- 7:54 “they had a chance to memorize our data set and the solutions and as you can see”
- 8:46 “the motivation for us to introduce use the code view category is because we are getting more and more poll requests on BC apps”
- 13:47 “you want to fork our repository and replace the data set that we have created for our own AL app issues”
Disclaimers in the video
- 12:54 not-in-this-release: At the time of the recording of this session we are still working hard on the results for code review so please check back on our GitHub repository if you’re curious
Presenters (as heard): Yos, Mharen, Javan.