Business Central Under the Hood episode 4: How we built Copilot Chat in Business Central

Overview

Episode 4 of Business Central Under the Hood (published 2024-08-02, 26 minutes) walks through how the team built Copilot Chat. It started about 18 months before the recording, after the launch of ChatGPT, and followed earlier AI work such as marketing text suggestions. Copilot Chat is an open chat for looking up documentation, understanding terminology and finding data. It is in public preview, US environments only.

The speakers explain how the conversation is condensed into a single intent message, how skills are selected, and how meta prompts, system prompts and user prompts are layered. They then cover testing: test-driven prompt development, datasets of thousands of harmful inputs, two LLMs simulating jailbreak conversations, and a third LLM grading the results. They close with advice for partners who want to build their own AI features.

Key points

• Copilot Chat is in public preview and, at recording time, only available in US environments. The team hoped to roll it out to more regions during the wave.
• Copilot Chat is scoped to two skills: documentation lookup (grounded in learn.microsoft.com) and finding data in Business Central. It runs asynchronously so users can keep working.
• Intent condensation sends the whole conversation to the model to produce one chat message with the user’s current intent. Other components then receive that message.
• Because the model is a blackbox, prompts are developed test-first. Datasets of inputs for supported question types are checked with assertions, for example that the right skill is chosen or off-topic queries are handled.
• Jailbreak prevention uses user roles versus system roles. User input goes into user prompts, and system prompts restrict scope to Business Central and forbid harmful content.
• Security testing runs thousands of potentially harmful inputs through the test suite. Two LLMs (Copilot Chat and an attacker) converse, and a third LLM grades the responses against harm categories.
• Automated grading is needed because manual grading does not scale and behavior changes when models are upgraded.

Features

AL objects mentioned

As heard in the captions. A name that matches one object page by exact type and name links to it; the others stay as named.

• other “learn.microsoft.com documentation” at 4:12
• other “Copilot Chat” at 13:31
• other “Copilot Developer Tools” at 24:13

Quotes

• 0:50 “about 18 months ago uh we started like playing around with some ideas after seeing like the launch of chat GPT”
• 1:30 “right now uh obviously it’s like an open chat so you can type whatever but we we have kind of scoped it down to”
• 2:31 “we wanted it to be non-blocking intentionally so you can uh like offload these queries and it might take a little bit to run”
• 2:51 “it was the last release wave so 2024 release Wave 1 yes um it’s currently in a public preview um and um it’s only”
• 3:52 “Bing chat and um co-pilot chat with business Central are kind of similar in the sense that they both use external data sources to”
• 5:31 “we take the whole conversation and we actually we go to the model and and ask it to um bring condense this whole conversation”
• 8:54 “the model itself is effectively a blackbox um so the best way to do it is to you know go back to your Basics”
• 12:31 “a meta prompt right now is a it’s effectively a prompt that is not visible to the end user”
• 13:31 “if uh if it’s a user role uh that that the prompt is saying like ignore the instructions the model knows that like w”
• 13:31 “if it’s a user role uh that that the prompt is saying like ignore the instructions the model knows that like w this is”
• 15:29 “so it’s it’s like thousands of uh inputs um but but um that we know can potentially produce some harmful response or undesirable response”
• 16:10 “we set two L&M to talk to each other our own our own feature the BC co-pilot chart right and we have another LM”
• 17:51 “so just to summarize there actually three LMS involved you say so there’s one asking the question yeah to BC chat right yeah uh”
• 18:32 “we have to automate these things because uh we see like when model upgrades things typically do change um it’s not that it’s”
• 19:33 “it’s definitely it is it is very different to kind of your classical uh software engineering um because uh yeah you you have this”
• 21:10 “you can you can ask it like please be more concise with this language it’ll fix any kind of grammatical issues and that kind”
• 24:53 “you could put them in the in say your key Vault if if you’re an appsource extension um and that way when you distribute”

Disclaimers in the video

• 2:51 preview: it’s currently in a public preview um and um it’s only available in US environments for now uh but we hope over the wave to roll it out to more regions
• 22:51 other: unfortunately that is like part of the our Microsoft uh IP uh at the moment um so uh so no
• 23:11 other: but they’re definitely not ready for production use um but it gives an idea of kind of the things you can do

Presenters (as heard): Viston Nicholas, Sam.