Getting Started With Agent Testing: How to Write Agent Evals in AL (Part 2)

Overview

Part 2 of the agent testing series shows how to structure an agent eval. The test follows the agent’s flow: a message arrives, the agent acts, and the output is verified as a message, a review request or an intervention request. Multi-turn conversations are covered by mocking user interventions in YAML. The presenter notes that data can change between turns in a busy ERP.

The video then walks through the YAML test definition (suite setup, test name, description, turns) and the AL test code that reads it. It covers suite setup with custom data, turn setup, running a turn, validating results, error logging, downloaded logs for LLM judge analysis, and the library agent’s public methods.

Key points

• Create custom master data for the suite setup instead of relying on demo data. In the team’s testing, the agent recognized customer 10000 from public demo data, probably from model training.
• Tests are driven by YAML with predefined sections (turn setup, message arrival with sender, title, body and attachments, expected data) plus free-format sections that you implement in AL code.
• Attachments can come from resources or from AL-generated reports converted to attachments. Dynamic attachment generation needs custom AL code.
• Date and datetime placeholders in YAML calculate dates from the current work date and a formula, so hardcoded dates need no manual updates. They also work in data-driven tests that are not agent tests.
• Instead of asserts, tests check data and accumulate issues in an error reason field, returning true or false. The finalize turn keyword stores results in the history log, and a fact box shows why a test failed (recently added).
• Downloaded logs contain serialized tasks, steps, inputs, expected data and error messages, and can be uploaded to Azure AI Foundry for LLM judge analysis.
• The library agent has 27 public methods, with asynchronous ones ending in ‘and wait’. Checking the text of an intervention request message is currently not possible, and the presenter says it is planned.

Features

AL objects mentioned

As heard in the captions. A name that matches one object page by exact type and name links to it; the others stay as named.

• other “animal data sets” at 4:12
• other “library agent” at 4:30
• other “Get eval suit setup data input” at 17:34
• other “Run turn and wait” at 20:54
• other “Get expected data” at 22:03
• other “Finalize turn” at 22:59
• other “Wait for the task to complete” at 25:49

Quotes

• 0:41 “going to end up with one of the following it is either going to create a message for you to review it”
• 1:37 “ERP is quite busy and there is a lot of users and code that is changing the data, data might be changed between the”
• 3:43 “in our testing we discovered that it wasn’t searching for the customer 10 000 it knew that”
• 4:30 “are going to be actual data-driven tests specified in YAML. The next component is library agent, which is going to help you to automate”
• 10:48 “Unfortunately, currently it’s not possible to check on the message if the message of the intervention request as as you expecting it. But we”
• 14:00 “now you can replace the dates with placeholders like this. And this is going to calculate dynamically the work date based on the current”
• 14:00 “the problem is that in the tests, I would need to change this one quite often. And the solution that we did is that”
• 14:13 “this is going to calculate dynamically the work date based on the current work date and the formula that you provide here”
• 16:25 “you can test You can run the same tests on different agents, change their instructions, maybe change the model, and see how that the”
• 18:28 “we are going to default the other fields like posting groups units of measure automatically by using test libraries”
• 22:26 “you’re not using the asserts to check the data. We are actually using the text field, which is called error reason.”
• 25:24 “we have added the library agent. It has 27 public methods which will allow you to automate every single interaction that you can do”
• 26:02 “this one is quite good if you are going to invoke the agent through the action or AL code. So, you would like to”

Disclaimers in the video

• 10:48 coming-later: it’s not possible to check on the message if the message of the intervention request as as you expecting it. But we are planning that.
• 23:51 preview: this fact box was not visible. We have added it recently

Presenters (as heard): Quentin.