---
id: video/yIYUCSK0Bh4
type: video
title: "Microsoft presents: BC-Bench: Can coding agents solve real-world AL tasks?"
summary: BC-Bench is Microsoft's open-source offline evaluation framework for coding agents on AL tasks (bug fixing, AL test generation, code review, code from natural language), adapted from SWE-bench. The session covers bug-fix methodology and dataset, multi-run metrics, AL-MCP gains, the finding that harness choice (Copilot CLI vs Claude Code) made no significant difference, complexity limits, and code review agent evaluation with a self-improving pipeline.
tier: community
language: en
tags:
  - coding agents
  - benchmarking
  - al development
  - bug fixing
  - bc-bench
  - code review
  - test generation
  - agent evaluation
  - llm
  - github
  - al-mcp
  - multi-run metrics
system: development
review:
  state: reviewed
  by: opus
  at: "2026-10-07T23:17:21.330Z"
  flags: []
generated:
  at: "2026-10-07T23:17:21.405Z"
  pipeline: 0.2.0
  prompts:
    extract-video: 1
    summarize-video: 2
  input_hash: 9dead8270a808a628e8548ae2ed98bb82f93e5977444d7d6a0d29ad536b40fcc
evidence:
  - kind: video
    url: https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=2087s
    title: "Payables and Sales Agents Code Availability: announced"
    date: "2026-10-01T05:38:52.000Z"
    commit: null
    t: 2087
    quote: Yes, so we're moving more and more things on on GitHub. So, I think these will be available soon.
  - kind: video
    url: https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=53s
    title: "Microsoft presents: BC-Bench: Can coding agents solve real-world AL tasks?"
    date: "2026-10-01T05:38:52.000Z"
    commit: null
    t: 53
    quote: the basic question we want to uh to answer is how good are coding agents. And specifically at uh AL tasks.
  - kind: video
    url: https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=64s
    title: "Microsoft presents: BC-Bench: Can coding agents solve real-world AL tasks?"
    date: "2026-10-01T05:38:52.000Z"
    commit: null
    t: 64
    quote: one of the things that uh would or is interesting to know is how much does it depend on the agent harness, right?
  - kind: video
    url: https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=494s
    title: "Microsoft presents: BC-Bench: Can coding agents solve real-world AL tasks?"
    date: "2026-10-01T05:38:52.000Z"
    commit: null
    t: 494
    quote: We started the effort about 6 months ago. That was heavily inspired by SweepBench. We followed the same methodology, but adapted for Business Central
  - kind: video
    url: https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=798s
    title: "Microsoft presents: BC-Bench: Can coding agents solve real-world AL tasks?"
    date: "2026-10-01T05:38:52.000Z"
    commit: null
    t: 798
    quote: So it's 68% almost or uh another one is 55%. That's almost 10 times higher.
  - kind: video
    url: https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=857s
    title: "Microsoft presents: BC-Bench: Can coding agents solve real-world AL tasks?"
    date: "2026-10-01T05:38:52.000Z"
    commit: null
    t: 857
    quote: The first part is the functional fix that actually fix the behavior and the second part is some union test to verify okay the
links:
  learn: []
  objects: []
  features: []
  topics: []
  localizations: []
  videos: []
  posts: []
  guidelines: []
video_id: yIYUCSK0Bh4
channel: yt-mibuso
source_name: mibuso.com / BC TechDays
url: https://www.youtube.com/watch?v=yIYUCSK0Bh4
published_at: "2026-10-01T05:38:52.000Z"
duration_s: 2140
captions: derived
audience:
  - developer
  - partner
  - decision maker
chapters:
  - t: 0
    title: Introduction and agenda
  - t: 53
    title: Evaluating coding agents for AL tasks
  - t: 193
    title: Benchmarking methodology and examples
  - t: 308
    title: SWE-bench and BC-specific requirements
  - t: 439
    title: BC-Bench framework and capabilities
  - t: 574
    title: BC-Bench architecture and stages
  - t: 691
    title: Live leaderboard and results
  - t: 813
    title: Bug fixing methodology and verification
  - t: 919
    title: Dataset creation, quality verification, and diversity
  - t: 1086
    title: Multi-run metrics and AL-MCP evaluation
  - t: 1274
    title: Complexity analysis and code review agent introduction
  - t: 1485
    title: Code review agent evaluation and online monitoring
  - t: 1678
    title: Self-improvement pipeline and BC-Bench roadmap
  - t: 1920
    title: "Q&A: Source code, data collection, costs, and GitHub models"
features:
  - name: BC-Bench benchmarking framework
    status: unclear
    t: 494
    verified: false
    status_source: video
  - name: Bug fixing agent evaluation
    status: unclear
    t: 41
    verified: false
    status_source: video
  - name: AL test generation evaluation
    status: unclear
    t: 544
    verified: false
    status_source: video
  - name: Code review agent evaluation
    status: unclear
    t: 544
    verified: false
    status_source: video
  - name: Natural language to code generation
    status: unclear
    t: 562
    verified: false
    status_source: video
  - name: Docker environment setup for BC-Bench
    status: unclear
    t: 615
    verified: false
    status_source: video
  - name: Multiple agent harness support
    status: unclear
    t: 64
    verified: false
    status_source: video
  - name: Model comparison capability
    status: unclear
    t: 87
    verified: false
    status_source: video
  - name: LLM as judge evaluation
    status: unclear
    t: 691
    verified: false
    status_source: video
  - name: GitHub integration and public repository
    status: unclear
    t: 587
    verified: false
    status_source: video
  - name: AL-MCP for coding agents
    status: unclear
    t: 1197
    verified: false
    status_source: video
  - name: Multi-run metrics for coding agents
    status: unclear
    t: 1063
    verified: false
    status_source: video
  - name: Dataset for bug fixing benchmark
    status: unclear
    t: 943
    verified: false
    status_source: video
  - name: Coding agent complexity analysis
    status: unclear
    t: 1317
    verified: false
    status_source: video
  - name: Online evaluation of AI features
    status: unclear
    t: 1657
    verified: false
    status_source: video
  - name: Self-Improving Code Review Agent Pipeline
    status: unclear
    t: 1689
    verified: false
    status_source: video
  - name: BC-Bench Dataset Expansion
    status: unclear
    t: 1753
    verified: false
    status_source: video
  - name: Prompt Compression Experiments
    status: unclear
    t: 1783
    verified: false
    status_source: video
  - name: Partner Dataset Leaderboard
    status: unclear
    t: 1902
    verified: false
    status_source: video
  - name: Payables and Sales Agents Code Availability
    status: announced
    t: 2061
    verified: true
    status_source: video
objects_mentioned:
  - other BC-Bench
  - other BC Container Helper
  - other BC apps
  - other SWE-bench
  - other SweepBench
  - other AL-MCP
  - other GitHub Copilot CLI
  - other Opus 4.6
  - other Cloud Code
  - other GPT-4 2
quotes:
  - t: 53
    text: the basic question we want to uh to answer is how good are coding agents. And specifically at uh AL tasks.
    check: exact
  - t: 64
    text: one of the things that uh would or is interesting to know is how much does it depend on the agent harness, right?
    check: exact
  - t: 494
    text: We started the effort about 6 months ago. That was heavily inspired by SweepBench. We followed the same methodology, but adapted for Business Central
    check: exact
  - t: 798
    text: So it's 68% almost or uh another one is 55%. That's almost 10 times higher.
    check: exact
  - t: 857
    text: The first part is the functional fix that actually fix the behavior and the second part is some union test to verify okay the
    check: exact
---

# Microsoft presents: BC-Bench: Can coding agents solve real-world AL tasks?

> BC-Bench is Microsoft's open-source offline evaluation framework for coding agents on AL tasks (bug fixing, AL test generation, code review, code from natural language), adapted from SWE-bench. The session covers bug-fix methodology and dataset, multi-run metrics, AL-MCP gains, the finding that harness choice (Copilot CLI vs Claude Code) made no significant difference, complexity limits, and code review agent evaluation with a self-improving pipeline.

[Watch on YouTube](https://www.youtube.com/watch?v=yIYUCSK0Bh4) · mibuso.com / BC TechDays · 2026-10-01 · 35:40 · tier community · reviewed (checked by Opus)

## Overview

The session explains how Microsoft built BC-Bench, an evaluation framework for coding agents working on Business Central AL tasks. It started about six months before the talk and follows SWE-bench methodology. Agents get a bug report and the code base from before the fix. Their fixes are checked against unit tests written by human engineers, in Docker environments of Business Central set up with BC Container Helper.

The talk covers the bug-fixing dataset and how it was filtered, results across models and agent harnesses, multi-run metrics, and the effect of AL-MCP. It then moves to a code review agent evaluated offline and monitored online, a self-improvement pipeline, the roadmap, and a Q&A on source code, data collection, costs and GitHub models.

## Key points

- Bug-fix verification: the code base is reset to before the fix, the agent gets the same bug report and produces a fix, and the original human-written tests are run. Tests must fail before and pass after the fix, and code must build in Docker.
- Dataset: single-bug pull requests that include tests, from the public BC apps GitHub repo and a private Azure DevOps repo. About 20% inventory and 19% finance. Bug reports are manually verified for sufficient context.
- Leaderboard: latest models reach about 68% (another 55%) on bug fixing, versus 16% (another 5%) for the GPT-4 model at the bottom.
- Multi-run metrics: AI is non-deterministic, so they compare mean accuracy, a consistency metric (task resolved in all five runs) and a best-case metric (resolved in any run). Bug fixing reports mean and the consistency metric. Two or three runs can still give statistical significance.
- AL-MCP (compile and symbol search) with GitHub Copilot CLI and Opus 4.6 raised mean resolution rate about 5% (non-overlapping 95% CI) and pass@5 about 10%.
- With Opus 4.6 fixed, GitHub Copilot CLI and Claude Code showed no significant difference; model choice is probably more significant.
- Complexity: agents reach about 80% on bugs needing fewer than 10 lines of code, dropping to around 50% for larger fixes.

## Chapters

- [0:00](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=0s) Introduction and agenda
- [0:53](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=53s) Evaluating coding agents for AL tasks
- [3:13](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=193s) Benchmarking methodology and examples
- [5:08](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=308s) SWE-bench and BC-specific requirements
- [7:19](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=439s) BC-Bench framework and capabilities
- [9:34](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=574s) BC-Bench architecture and stages
- [11:31](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=691s) Live leaderboard and results
- [13:33](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=813s) Bug fixing methodology and verification
- [15:19](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=919s) Dataset creation, quality verification, and diversity
- [18:06](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=1086s) Multi-run metrics and AL-MCP evaluation
- [21:14](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=1274s) Complexity analysis and code review agent introduction
- [24:45](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=1485s) Code review agent evaluation and online monitoring
- [27:58](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=1678s) Self-improvement pipeline and BC-Bench roadmap
- [32:00](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=1920s) Q&A: Source code, data collection, costs, and GitHub models

## Features

| Feature | Status | At | Evidence |
|---|---|---|---|
| BC-Bench benchmarking framework | status not stated, demoed | [8:14](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=494s) |  |
| Bug fixing agent evaluation | status not stated, demoed | [0:41](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=41s) |  |
| AL test generation evaluation | status not stated | [9:04](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=544s) |  |
| Code review agent evaluation | status not stated, demoed | [9:04](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=544s) |  |
| Natural language to code generation | status not stated, demoed | [9:22](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=562s) |  |
| Docker environment setup for BC-Bench | status not stated | [10:15](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=615s) |  |
| Multiple agent harness support | status not stated, demoed | [1:04](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=64s) |  |
| Model comparison capability | status not stated, demoed | [1:27](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=87s) |  |
| LLM as judge evaluation | status not stated | [11:31](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=691s) |  |
| GitHub integration and public repository | status not stated, demoed | [9:47](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=587s) |  |
| AL-MCP for coding agents | status not stated, demoed | [19:57](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=1197s) |  |
| Multi-run metrics for coding agents | status not stated, demoed | [17:43](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=1063s) |  |
| Dataset for bug fixing benchmark | status not stated | [15:43](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=943s) |  |
| Coding agent complexity analysis | status not stated, demoed | [21:57](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=1317s) |  |
| Online evaluation of AI features | status not stated | [27:37](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=1657s) |  |
| Self-Improving Code Review Agent Pipeline | status not stated | [28:09](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=1689s) |  |
| BC-Bench Dataset Expansion | status not stated | [29:13](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=1753s) |  |
| Prompt Compression Experiments | status not stated | [29:43](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=1783s) |  |
| Partner Dataset Leaderboard | status not stated | [31:42](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=1902s) |  |
| Payables and Sales Agents Code Availability | announced | [34:21](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=2061s) | "Yes, so we're moving more and more things on on GitHub. So, I think these will be available soon." ([34:47](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=2087s)) |

## AL objects mentioned

As heard in the captions. A name that matches one object page by exact type and name links to it; the others stay as named.

- other "BC-Bench" at [0:28](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=28s)
- other "BC Container Helper" at [10:30](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=630s)
- other "BC apps" at [10:30](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=630s)
- other "SWE-bench" at [5:08](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=308s)
- other "SweepBench" at [8:14](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=494s)
- other "AL-MCP" at [19:57](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=1197s)
- other "GitHub Copilot CLI" at [20:11](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=1211s)
- other "Opus 4.6" at [20:11](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=1211s)
- other "Cloud Code" at [21:22](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=1282s)
- other "GPT-4 2" at [22:31](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=1351s)

## Quotes

- [0:53](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=53s) "the basic question we want to uh to answer is how good are coding agents. And specifically at uh AL tasks."
- [1:04](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=64s) "one of the things that uh would or is interesting to know is how much does it depend on the agent harness, right?"
- [8:14](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=494s) "We started the effort about 6 months ago. That was heavily inspired by SweepBench. We followed the same methodology, but adapted for Business Central"
- [13:18](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=798s) "So it's 68% almost or uh another one is 55%. That's almost 10 times higher."
- [14:17](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=857s) "The first part is the functional fix that actually fix the behavior and the second part is some union test to verify okay the"

## Disclaimers in the video

- [3:13](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=193s) coming-later: there's a GPT-6 Codex. Well, there will be, but I don't know when it will be released.
- [17:11](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=1031s) other: coding agents can have random behaviors due to non-deterministic AI
- [29:13](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=1753s) coming-later: we are going to refresh the data set with more issues for for the bug-fixing benchmarking from uh BC apps
- [31:42](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=1902s) preview: we are actually considering that on the leaderboard you saw that we demoed that we can potentially have a partner ...
- [34:21](https://www.youtube.com/watch?v=yIYUCSK0Bh4&t=2061s) coming-later: we're moving more and more things on on GitHub. So, I think these will be available soon

Presenters (as heard): Klaus, Horan, Harun, Claus, Haron.