20251208 - You don't have to make Test Plans anymore...
AI-assisted test plan generation from requirements, shown by 4PS and a partner in an Areopa webinar. It covers the test plan method, structured system prompts, and VS Code with an Azure DevOps MCP server, and it says AI output is a draft that testers review.
- Introduction and upcoming webinars
- Presenter introductions: Luke Fanfoot
- Presenter introduction: Tina Starridge and project overview
- Agenda and project goals
- What is a test plan and why it matters
- Benefits of test plans and scenario types
- Test plan structure and components
- Scenario IDs, descriptions, and test types
- Challenges of manual test plan creation
- Introduction to AI-assisted test plan generation and structured prompts
Overview
The webinar starts with what a test plan is and why 4PS writes one from requirements before coding. It lists all user scenarios, split into sunny-path (positive) and rainy-path (negative), so reviewers can confirm nothing is missing. The manual process takes time, and the size of the requirements affects how much.
The presenters then show how AI can generate a first draft. They use a structured system prompt with five parts: instructions, steps, format requirements, examples and notes. In Visual Studio Code, an MCP server connects the model to Azure DevOps, and a prompt file is run with a slash command such as /create test plan. Domain experts review the output and the prompt is refined over several rounds. The talk ends with plans to distribute the prompts through a VS Code extension.
Key points
- A test plan lists all user scenarios from the requirements before coding. Once reviewers agree on the list, they know no scenario is missing.
- Structure the system prompt with instructions, steps, format requirements, examples and notes. Without structure, running the same prompt again gives different results.
- Language models cannot reach Azure DevOps on their own. An MCP server acts as a toolbox that lets the model read requirements in VS Code.
- Output quality depends on the user stories. If a requirement lacks detail, the AI may make content up.
- The first prompt will give poor results. Refine it through dialogue and expert feedback, for example by asking for one line per user action and no ‘attempts to’ wording.
- A prompt file can be run with a slash command like /create test plan, so testers need not be prompt engineers. A central team maintains the prompts.
- The suggested rollout has three stages: keep prompts in the project, move them to user settings, then distribute updates with a VS Code extension, which is still being planned.
Features
| Feature | Status | At |
|---|---|---|
| AI-assisted test plan generation | status not stated, demoed | 4:48 |
| Test plan methodology at 4PS | status not stated, demoed | 7:25 |
| User scenario classification | status not stated | 11:23 |
| Structured system prompts for test planning | status not stated | 22:37 |
| MCP integration for test planning in Visual Studio Code | status not stated, demoed | 26:09 |
| System prompts in VS Code for structured test plan generation | status not stated, demoed | 30:44 |
| AI-assisted prompt generation and refinement | status not stated, demoed | 33:09 |
| AI-structured requirement analysis into test scenarios | status not stated, demoed | 35:56 |
| Expert feedback loop for test plan improvement | status not stated, demoed | 39:01 |
| Prompt-based slash commands for AI tasks | status not stated | 46:59 |
| Multi-use case AI prompts | status not stated | 46:44 |
| AI as junior developer paradigm | status not stated, demoed | 47:58 |
| AI reduces manual test planning effort | status not stated, demoed | 48:55 |
| Centralized prompt management for teams | status not stated | 50:25 |
| VS Code extension for prompt distribution | status not stated | 57:58 |
| Phased prompt deployment strategy | status not stated | 57:18 |
AL objects mentioned
As heard in the captions. A name that matches one object page by exact type and name links to it; the others stay as named.
Quotes
- 0:13 “This one is entitled You don’t have to make test plans anymore.”
- 4:48 “this is the point that really brought us together and that’s where we started to explore this”
- 8:10 “what i want to achieve is i want us to have all user scenarios listed”
- 9:00 “if you have requirements you need to have a test plan”
- 10:37 “If we have reviewed the requirements and agree upon this is the full list of user scenarios then we all know there’s nothing missing.”
Disclaimers in the video
- 23:13 other: Whatever comes out of AI is not going to be perfect. We’re not creating a test plan that we immediately …
- 24:16 subject-to-change: the technology has evolved a lot in the last six months, let alone the full year. So for today, I’m …
- 57:58 coming-later: how can I then centrally update these uh these prompt files within the user directory
- 58:15 coming-later: what if we create a VS code extension that’s going to do do just that
Presenters (as heard): Henrik Ellison, Luke Fanfoot, Tina Starridge, Ron vonor, Jeremy, Waldo, Hen Helgerson.