What's new in BC-Bench

0:06 Hi everyone and welcome to this session on what's new in BCbench. My name is Yos and I'm joined today with my colleague >> Mharen. >> Before we cover what's new in BCbench, let's start with a recap on what BCbench is and how we use it. BCBench is a reproducible evaluation framework for AI coding agents working on real world AI development task inspired by SVBench. This means that we use it to test the uh the agents and harnesses and models that we use to develop um AL features and bug fixes and we measure the performance of these different models and harnesses

0:39 on real problems. Um BCbench is there to quantify the impact of tooling changes. So if we're changing the tools or models or instructions or agents that are working on these coding problems, does uh the result change? Do they get better or worse over time? Um, and this in turn allows us to track the progress with comparable metrics over time. It also makes it possible for us to iterate rapidly. So if we see scores going up or down because of a change we've made, we can quickly revert or continue with a certain change. Now why do we need our own benchmarking tool in addition to SWEBench?

1:13 This is because the the general coding benchmarks like SWEBench don't uh reflect the AL development realities. AL is a smaller ecosystem than other uh coding languages and there's a lot of domain specific tooling for AL features. This is best illustrated by simply looking at the number of open- source repositories for different coding languages in GitHub. As you can see, there's only 400 um that are written in AL. Well, there's 3.3 million in Python and 650,000 in C. So, what do we cover with BCBench?

1:46 So today BCbench covers the following categories of AL development work. First of all, there's bug fixing and there's test generation and also there's PR reviews or code reviews. And throughout these categories, BC bench follows the principles of representative real world data sets. Meaning that the data sets that are part of this benchmark come from real books and real PRs and real tests uh that were written in AL uh that BCbench runs on. These data sets are open source, meaning that BC bench can run can be run transparently and the results can be reproduced by anyone that runs this uh benchmark.

2:20 And finally, we make sure that these evaluations are rigorous by following industry best practices. In this session, we don't have the opportunity to go into the deep details of how BCBench was built, uh how it works and why it works. If you are interested in learning the details of this, we have published a research paper on aka.ms/bcbench paper that is available uh for all to read uh that contains all the details. Thanks. A quick recap on the since last time we were here we introduced the bug fixing category. Um as used covered um a quick recap of what it is about.

2:58 An agent is tasked with fixing a bug in the real codebase just like how human engineers are deal with day-to-day. Uh and we uh use 101 tasks from the real box where real human engineer solve in our n repositories and busy [clears throat] apps. And finally, uh if you notice it, haven't noticed it, uh there is the ALMCP server that uh thanks to our beloved developer experience team that expose AL development tools like compiling,

3:33 publishing and simple search to those coding agents. So you don't need to open your visual studio code and the agents will have the capability to um publish and um compel just like uh what our engineers do dayto-day. So what has the result uh showed us? Um during over the last few months we have done lots of experiments and find out uh using the GitHub copilot CLI and cloud opus for six

4:07 uh we can we find that AMCP uh gives the coding agents a meaningful improvement over uh when the coding agent don't have access to it. As you can see that um the pass head five matrix improved almost 10%. Uh and there's also a significant improvement uh in the mean resolution rate. Well, it does take a bit longer but I think that makes sense because the agent now has to uh publish apps and uh and run tests and things like that.

4:40 Uh what else did we discover? uh we find that um and using the same model and but with different agent harnesses like GitHub copilot CLI versus cloud code uh we find no significant differences uh between them. Uh all the numbers are fairly close when using the same models. So the choice of models probably matter more than the choice of agent harness at least using our data set.

5:10 And what else did we do? uh over the last few months we got lots of questions about now that uh our repositories are open source because we want to be transparent there is a risk of the models will be trained on the public available data so they will memorize our solutions and that is called uh contamination we um followed a precedence using this paper it's a three bench illusion so for folks who are interested give it a read.

5:44 Uh we follow their method uh as much as possible and uh derived a contamination detection strategy of our own. So how does it work? uh we give them a template, give the coding agent a template where we uh uh feed it with where which repository is this task coming from like for example BC apps and a problem statement uh the buck description just like on the

6:14 accuracy test part and finally we ask the coding agent to come up with the file is supposed to modify but this time without the code base itself. So there's no way the coding agent can figure this out by itself except for memorize it from the training. And what is the result? Here is a run a rerun in one of our tasks where the agent managed to uh response with a fairly close uh file path that looks correct. Uh but um as you can see the file is supposed to modify our our three files and uh even the closest one did not match exactly.

6:58 So it deduced the path from the bug description but did not memorize the exact file path. As some of us may be wondering now all our source code on PC apps maybe the agent managed to figure out from there that is not the case either uh because on PC apps the the file path starts a bit differently too than uh what the agent managed to come up with. So for this specific example, we we decide the agent didn't manage to memorize the problem. So this specific task was not contaminated.

7:38 And here are the results uh over 101 tasks in our data set. We chose three models. Um and they are chosen because their knowledge cut off date uh is after we made BCbench public. So they had a chance to memorize our data set and the solutions and as you can see the all for all three models they only managed to uh come up with one out of 101 tasks uh that's a match and in this specific case it was actually the task actually

8:14 contain the exact file path is supposed to modify and with that this is not a one-time effort we will be continuous continuously monitoring the signs of contamination in BCB bench to make sure our data reflect the performance of the models and the agent but not the how the model remembers our data set and uh you mentioned previously there is a new uh code review category that we have on boarded to BCbench

8:46 and I want to cover a bit more details here uh the motivation for us to introduce use the code view category is because we are getting more and more poll requests on BC apps. As you can see from the screenshot here that um uh the number of poll requests on BC apps are growing very fast especially now that we have all the source code uh on BC apps. So a natural question we come up with is now that AI can help us create those poll requests, can AI help us review the AL code changes as well.

9:24 So uh this doesn't become a bottleneck. So we implemented three way to review AL pests um in BC bench. Uh they are all slightly different. The first one is the baseline or the the controlled experiment where we use the barebone GitHub copilot UI. Uh it doesn't have any specialized AI knowledge. It just download from uh internet directly. And then we have the second approach of BC quality where all um lots of our partner community has been contributing to with all the best practices and things to pay attention to

10:08 and we installed that onto the copilot CLI using the plug-in approach. So uh this is the second uh way uh that we can review AL pull requests. And finally that our teams have been working on the AL review agent. If you have contributed to BC apps recently, you should see some comments from this review agent that is a specialized orchestration that um designed to give BC quality a further boost on focus on security, privacy and performance. individually uh instead of installing the BC quality all together and how does the review uh methodology works in BC bench.

10:55 So here is a simplified task we have uh for the code review data set and the agent is tasked to review this specific function that's introduced in a poll request. Uh and um for app developers you can pause the video to see uh if you can recognize uh what is the issue with this code. Uh and now uh we will uh review uh so this is u for every task in the code review data set it comes with the expected comment

11:30 meaning what is what is wrong with this piece of code and in this scenario is the log table uh line that um is unnecessary because the whole procedure is a read transaction so there's no need for the log table here in the code and we run this specific task in our pipeline and here are the results. Uh the first one is from GPT56 Luna. He categorized the issue to be cocurrency and he left a comment that's um semantically similar to uh what is expected

12:08 uh between line seven and 8 and similarly GPT56 terror uh categorized into a performance issue and then uh left also similarly semantically on line number seven. So well the those uh comments uh looks different but uh they both capture the exact same expected comment. So here we use a LM as judge. So different than the bug fixing um category. Uh here we use the LMS stretch to determine the comments are uh expected and in both scenario here we determine they are both um generated the correct comment.

12:54 At the time of the recording of uh this um session uh we are still working hard on the results. Uh so please check back uh on our GitHub repository uh if you're curious. Now back to you. >> Thank you Javan. So as a partner that is um getting started with Aentic engineering for your own uh apps in AL you would want to experiment uh with the tools instructions MCP server models that you're using to do so. And to experiment um you should visit our BC bench repository on GitHub and first of all check out the results that we publish. So for the the test cases, the bug fixes, the PR reviews, the test generated

13:34 uh that we publish, um there's results that are there which already should give you an idea of the models and harnesses that are most effective at developing in AL code. Of course, these results are the results from running BCB on our data set. Uh but what you want to do is you want to fork our repository and replace the data set that we have created for our own AL app issues uh with your own data set that includes your own bug fixes, your own generated tests and your own PRs. So you can see how BCBench produces results for different models and harnesses and tools for your specific app problems. Uh and then finally, as you do this, if your repository uh is open source over time, you want to make sure you check for contamination as well.

14:16 so that uh newer models aren't trained on your open source data set um as you rerun BCBench in the future. Um finally, this is the link where you can find BCBench. You can find uh the entire tool is open source. Our data set is open source and also you can see the results of our own BCBench runs. Uh the link for this is github.com/microsoft/bcbench. And that concludes our session on BCBench. Thanks everyone. >> Thanks.