Getting Started With Agent Testing: Why Are Evaluations Important? (Part 1)
Part 1 of an agent testing session on why evaluations matter for AI features in Business Central. It explains how non-deterministic AI behavior differs from traditional pass-fail testing, then outlines evaluation types: human evaluation, programmatic data-driven tests in AL using YAML (the main focus of the session), LLM-as-judge scoring, and a brief mention of agent benchmarking.
- Introduction to AI evaluations
- Traditional testing approaches
- Why AI testing differs from traditional testing
- Types of evaluations for AI
- Programmatic and data-driven testing
- Using LLMs to judge outputs
- Advanced benchmarking scenarios
Overview
This five-minute introduction explains why evaluations matter when building AI features and agents. The speaker argues that the quality of the evaluations matters more than the quality of the prompt or instructions. Traditional tests assume deterministic behavior, while AI has non-deterministic behavior, varying inputs, and subjective quality, so metrics matter more than a simple pass or fail.
The video then outlines the types of evaluations. Programmatic evaluation builds tests in a deterministic way, with variation in the input, using data-driven test cases defined in YAML. LLM-based evaluation has a language model judge whether an output, such as an email or a product description, reflects what was asked. Benchmarking agents across setups, with metrics like latency, cost and error rates, is mentioned only briefly and is left for later content.
Key points
- The speaker’s central claim: the important part is not how good the prompt or instructions are, but how good the evaluations are.
- Traditional testing checks deterministic behavior with exact assertions that are repeatable and either pass or fail.
- AI is different: behavior is non-deterministic, inputs and outputs vary, results are not repeatable, and quality is subjective, so metrics matter more than pass-fail.
- Evaluations should also measure quality and safety, for example how the feature behaves when someone tries to fool it.
- Human evaluation is possible but hard to scale.
- Programmatic evaluation: write the tests in a deterministic way in AL, with varied inputs and biases, using data-driven test cases defined in YAML. This is the main focus of the session.
- LLM-based evaluation: use a language model (for example Foundry functionality) to score whether generated text such as emails, product descriptions or project descriptions reflects what was asked.
Features
Every feature below: status not stated.
| Feature | At |
|---|---|
| AI evaluations framework | 0:04 |
| Programmatic evaluation with data-driven testing via YAML | 3:20 |
| LLM-based output evaluation | 4:09 |
| Agent benchmarking and observability | 4:49 |
Quotes
- 0:04 “the important part is not necessarily how good your prompt is or your instructions, but how good your evaluations are”
- 1:46 “AI is different in that AI has non-deterministic behavior. The uh the inputs will vary.”
- 2:37 “the behavior isn’t repeatable, and the quality is subjective, and the metrics actually matter more than just a pass-fail”
- 3:20 “you need to program the tests in a deterministic way. You can have variation in the input.”
- 3:35 “building it in AI AIL and using um data-driven via YAML gives you a framework for basically testing in a deterministic way”
- 3:57 “that is what we’re going to focus most of today’s office hour how you can build your own evaluations for your AI features here”
- 4:33 “you can use LLMs to try to judge, okay, I asked for this, does the answer reflect what I asked for?”
Disclaimers in the video
- 5:00 not-in-this-release: But we’re not really going to cover that today. The main focus is on the on the second part.
Presenters (as heard): Nikola.