What's New: Testing Copilot in Business Central (2024 release wave 2)
Copilot Test Toolkit in Business Central (2024 release wave 2), introduced in preview: an AL extension for testing Copilot features with data-driven tests, AI test context, Azure AI Studio external evaluation and multi-turn tests. Demonstrated on sales line suggestion and marketing text.
- Introduction to Copilot Testing
- Challenges in Testing Large Language Models
- Test Case Creation and Azure AI Studio
- Types of Tests and Data-Driven Testing
- Introducing the Copilot Test Toolkit
- Copilot Test Toolkit Interface and Components
- Test Suite Actions, Data Set Format, and Architecture
- From Manual Tests to Data-Driven Testing with AI Test Context
- Running Tests and Evaluating Results in Business Central
- External Evaluation with Azure AI Studio
- Multi-Turn Testing for Conversational Scenarios
- Best Practices and Future Roadmap
Overview
The video explains why testing Copilot features differs from normal testing: large language models are sensitive to input variations, so the same test needs to run against many inputs. It introduces the Copilot Test Toolkit, an AL extension installed in a sandbox, with source in the BC apps repo. It is presented as being introduced in preview.
The session covers the toolkit interface, test suites, the data set format, the AI test context codeunit, and running tests in Business Central. It demos sales line suggestion (28 tests, 27 passing, 1 failing) and marketing text, which is evaluated externally in Azure AI Studio. It then covers multi-turn testing for conversations and best practices, and ends with a roadmap toward test automation.
Key points
- The Copilot Test Toolkit is an AL extension, installed in a sandbox, with source in the BC apps repo (aka.ms/bcapps, AI Test Toolkit folder). It is introduced in preview in this release.
- Data-driven testing runs each test procedure once per entry in a data set, which covers LLM sensitivity to input variations.
- The AI test context codeunit gives test procedures their inputs from the data set (question, answer, context, ground truth, expected data) and can set test output for external evaluation.
- Data sets are uploaded in JSON lines format and can be marked as sensitive. Test suites can be imported and exported as XML files, and results can be downloaded for external evaluation.
- Marketing text is evaluated through Azure AI Studio using built-in or custom evaluators. This is subject to separate billing, and different metrics need different input sets.
- Multi-turn testing supports several inputs and outputs per test case for conversational scenarios. Calling the next turn API on the AI test context moves the test to the next question.
- The demos mock responses, so no tokens are consumed. Use data sets with many entries, because small prompt changes can change results.
Features
| Feature | Status | At | Evidence |
|---|---|---|---|
| Copilot Test Toolkit | preview, demoed | 8:26 | “in this release, we are introducing the Copilot Test Toolkit in preview that allows you to test Copilot features” (8:26) |
| Sales Line Suggestion Feature | status not stated, demoed | 14:11 | |
| Data-Driven Testing for Copilot | status not stated | 6:55 | |
| Accuracy Testing for Copilot | status not stated | 5:03 | |
| Quality Testing for Copilot | status not stated | 5:56 | |
| AI Test Context Code Unit | status not stated, demoed | 16:08 | |
| External Evaluation with Azure AI Studio | status not stated, demoed | 19:04 | |
| Marketing Text Feature | status not stated, demoed | 20:00 | |
| Multi-Turn Testing Support | status not stated, demoed | 22:54 | |
| Test Suite Import and Export | status not stated, demoed | 25:19 | |
| API Pages for Test Toolkit | status not stated | 25:19 |
AL objects mentioned
As heard in the captions. A name that matches one object page by exact type and name links to it; the others stay as named.
- codeunit “AAT test context” at 13:25
- codeunit “AI test context” at 13:25
- codeunit “AI test” at 15:51
- page “AI test suites” at 17:48
Not found in BC28-30: codeunit “AAT test context”, codeunit “AI test context”, codeunit “AI test”, page “AI test suites”.
Quotes
- 0:57 “there are some new considerations in the world of large language models and co-pilot features that require attention due to the high degree of”
- 7:37 “large language models are sensitive to input variations and we want to cover how well the prompt generalizes and adapts to various contexts”
- 8:26 “in this release, we are introducing the Copilot Test Toolkit in preview that allows you to test Copilot features in an efficient way”
- 8:40 “It’s an AL extension that can be installed in a sandbox environment, and you can find the source for this in our BC apps”
- 13:14 “Each test procedure will be executed as many times as you have the entries in the data set.”
- 15:40 “the question and the expected data, these are specific values supported by the tool, but you can format this however you want in your”
- 18:27 “for this specific one, we are mocking the response, so there’s no tokens consumed, but you would be able to see how many tokens”
- 19:15 “our choice for external evaluation is Azure AI Studio. In Azure AI Studio, you can assess and compare AI application performance using built-in evaluators”
- 23:17 “In multi-turn, you might have multiple inputs and outputs per test case. This is complex conversational scenarios.”
- 24:41 “Ensure that your data sets have a substantial number of entries. As mentioned before, even slight variability in prompts can impact how your feature”
Disclaimers in the video
- 8:26 preview: we are introducing the Copilot Test Toolkit in preview
- 19:38 subject-to-change: It’s subject to separate billing, but you can learn more about that at aka.ms/azureaistudio.
- 25:42 coming-later: We aim to improve the tool to enable the test automation and make it even smoother for you to test your Copilot features with less effort.
Presenters (as heard): Peter Borring, Julia Møller, Steen Klitgård, Julian, Peter.