Getting Started With Agent Testing: Test Setup and Structuring (part 3)
Structuring and setting up data-driven tests for Business Central agents: YAML data sets, XML suite configuration, reusable AL codeunits, P0/P1/load/variation categories, and a PowerShell runner for CI/CD against sandboxes. Part 3 of an agent testing series, using a sales return agent as the example.
- Test structure and three core components
- AI Development Toolkit and dependencies
- Configuration file example and code unit reuse
- Installation code unit and YAML file loading
- Test categorization: P0, P1, load and variation tests
- Sales return agent test example
- PowerShell support for CI/CD pipelines
- Test runner location and sample availability
- Future plans: LLM judge and Visual Studio Code integration
- Call to action and feedback request
Overview
The video explains how an agent test is built from three parts: data sets that define the input, an AL codeunit that loops through the turns and verifies the outcome, and configuration that ties them together. Tests depend on the AI Development Toolkit Evaluation app, which brings the AI Test Toolkits and the agent test library with utility methods. An installation codeunit uses the AI Test Suite Management methods to load YAML and XML files into the environment.
It then covers categorizing tests (P0, P1, load, variation), shows a sales return agent example where one codeunit is reused across several data sets, and describes a PowerShell module for running tests in CI/CD pipelines. The presenters note this is a first version, ask for feedback, and list planned work: LLM as judge, running tests from Visual Studio Code, and more samples.
Key points
- An agent test has three core parts: data sets define the input, an AL test codeunit loops through the turns and verifies the outcome, and configuration files wire them together.
- Add a dependency on the AI Development Toolkit Evaluation app, which the presenters recommend. It provides the AI Test Toolkits and agent test library utility methods.
- YAML files hold data sets and XML files hold test suite configuration. An installation codeunit imports them using the AI Test Suite Management methods.
- One codeunit can be reused across multiple data sets, so adding a new test case mostly means adding data.
- Categorize tests: P0 for happy-path tests that should usually pass and run fast and often, P1 for specific scenarios and edge cases, plus load tests (slower, run less often) and variation tests.
- The PowerShell test runner supports CI/CD with token-based authentication, but only against sandboxes. It does not run on-prem or on AL-Go images.
- The sales validation agent test sample is on BCT. The sales return agent sample is not on BCT yet and is available on request. The PowerShell runner ships in the image under Applications, test framework, test runner. LLM as judge, running tests from Visual Studio Code, and agent-assisted agent implementation are planned.
Features
| Feature | Status | At | Evidence |
|---|---|---|---|
| AI Development Toolkit Evaluation app | status not stated | 0:49 | |
| AI Test Toolkits | status not stated, demoed | 0:49 | |
| Test Suite Management methods | status not stated | 2:02 | |
| Data-driven test structure with YAML | status not stated, demoed | 1:26 | |
| P0 test category | status not stated | 2:35 | |
| P1 scenario tests | status not stated | 2:45 | |
| Load and variation tests | status not stated | 3:12 | |
| PowerShell test runner for AL tests | status not stated | 4:06 | |
| LLM as judge for test validation | announced | 5:29 | “we are planning to introduce LLM as a judge and this one is to verify the user intervention if it is containing the text” (5:29) |
| Visual Studio Code test execution | status not stated | 5:49 | |
| Agent-assisted agent implementation | status not stated | 5:59 |
AL objects mentioned
As heard in the captions. A name that matches one object page by exact type and name links to it; the others stay as named.
- other “AI Development Toolkit Evaluation” at 0:49
- other “AI Test Toolkits” at 1:09
- codeunit “AI Test Suite Management” at 2:02
- other “Sales Validation Agent Test” at 5:01
- other “Sales Return Agent” at 5:13
Not found in BC28-30: codeunit “AI Test Suite Management”.
Quotes
- 0:29 “the AL test is what ends looping through the turn verifying their outcome. So the data sets are defining the input in the form”
- 0:49 “you need to depend on this new app this AI development toolkit evaluation which is uh the app uh we recommend using”
- 1:50 “we’re actually reusing one code unit with multiple data sets, which so which highlights also how easy it is now to to create new”
- 2:35 “we usually define a set of P 0 tests. Those are the ones that you should expect to to pass pretty often. Uh, they”
- 4:18 “Currently it’s only possible to run these tests in the sandboxes. Unfortunately, it’s not possible to run them on prem for the algo images”
- 5:01 “So you can find a sample test which is called the sales validation agent test at the BCT.”
- 5:13 “this one is not on BCT. If you need it, reach out to us and we can provide the codes to you”
- 5:29 “we are planning to introduce LLM as a judge and this one is to verify the user intervention if it is containing the text”
Disclaimers in the video
- 4:18 not-in-this-release: Currently it’s only possible to run these tests in the sandboxes. Unfortunately, it’s not possible to run them on prem
- 5:13 not-in-this-release: sales return agent sample is not on BCT yet; available on request
- 5:29 coming-later: planning to introduce LLM as a judge for test validation
- 5:49 coming-later: looking into enabling running these tests from the visual studio code
- 5:59 coming-later: want to ship more samples
- 6:14 coming-later: making this fully possible for the agents to implement
- 6:29 preview: Keep in mind this is the first version. We are really looking forward to your feedback
Presenters (as heard): Quin, Nicolola.