Business Central Under the Hood episode 8: AI Small Language Models and Fine-tuning

Overview

Episode 8 of Business Central Under the Hood explains what small language models are, how the Phi series relates to larger models, and how fine-tuning and model distillation can lower the cost of AI features. It then applies this to the sales order agent, which is in preview.

The demo covers generating test data from seed examples, running test cases with the AI test toolkit, and using successful runs as fine-tuning data for GPT-4 mini. Results are preliminary and the speaker warns about overfitting to the toy data. The closing advice for partners is to try prompt engineering first and to treat data quality as the main work.

Key points

• Small language models have fewer parameters (billions). Phi 4 has 14 billion, runs on a single GPU, and is described as outperforming GPT-3. They are weaker at reasoning and world knowledge, and are about 10 times faster.
• Model distillation uses a larger model such as GPT-4 as teacher to train a smaller one such as GPT-4 mini. The price difference between them is a factor of 10-15. The larger model must already work well on the use case.
• OpenAI models such as GPT-4o and GPT-4 mini can be fine-tuned in Azure AI Foundry, deployed from there, and consumed in Business Central through the AI toolkit like any other model.
• Sales order agent test on 10 emails: GPT-4 reached about 85% accuracy at about 22 cents, and GPT-4 mini reached about 30% at about 1 cent. Fine-tuned GPT-4 mini reached over 90% on execute task steps at about twice the cost of plain GPT-4 mini.
• Fine-tuning data came from successful test case runs, taking input-output pairs from the execute task steps. Hundreds of test cases measure accuracy. The AI test toolkit runs data-driven AL tests and also supports LLM-based checks of natural-language output such as reply emails.
• The fine-tuned results are preliminary and trained on a limited data set. The model may overfit to the toy customer and item names and not generalize to other customers’ master data.
• Try prompt engineering first (few-shot prompting, Chain of Thought). Fine-tune only if the model does not do a good job. Training data must be representative, accurate and kept separate from test data.

Features

AL objects mentioned

As heard in the captions. A name that matches one object page by exact type and name links to it; the others stay as named.

• other “Phi models” at 7:35
• other “AI Foundry” at 15:00
• other “AI toolkit” at 15:20
• other “AI test toolkit” at 17:39

Quotes

• 0:50 “i started working on ai and machine learning in 2005 so it’s 20 years ago so before”
• 4:53 “um this is 4 years ago that that had 175 billion parameters”
• 7:35 “was released um this december with i think 14 billion parameters and that uh outperforms it clearly outperforms something like”
• 8:16 “these 14 billion that that’s small enough that you can run it efficiently on a single GPU so it means that it’s so you”
• 10:37 “we can do the same as open AI does on this pre-trained LM and we can also do it in a more efficient way”
• 12:18 “you would do that only if you find out that the model is not doing a good job to start with”
• 13:39 “you take a big model and uh which performs for your particular application well yes and you take the output of that and then”
• 14:19 “the difference between in pricing of dg4 for example in tpg4 mini is is a factor 10 15”
• 15:00 “you can deploy them also from AI Foundry and then you can consume them just like any other model”
• 16:59 “for um the sales order agent that we are releasing now we have a measure of accuracy that is um you know given an”
• 20:29 “we also have support in the AI test to kit for using an llm to to test features”
• 22:11 “we have test cases so we have hundreds of test cases that run mhm and if these test cases pass that’s actually a good”
• 24:33 “uh um here on the left we ran through 10 emails and in approximately 85% of the cases gbgo did the right thing so”
• 25:13 “tt4 mini achieves around 30% accuracy so that’s a quite a significant drop and so useless”
• 25:33 “if we do that we get more than 90% accuracy on on these execute task steps which is pretty good and in this case”
• 26:54 “risk that when we have been training with this data then small the small language model that get fine tune is very good at”
• 27:34 “gbgo uh achieved around 85 85% accuracy but it cost um around 22 cents to run these these 10 emails in this case whereas”
• 27:55 “dg4 mini then the fun version then hits kind of a sweet spot because it’s it’s a little bit more expensive than dg4 mini”
• 28:35 “the first thing to do is always prom prompt engineering you know few shot prompting Chain of Thought and and all those things”
• 28:55 “if there is a feature uh that uses gg4 um that has high accuracy but is expensive yeah then that’s that’s and you have”
• 29:35 “if you have a scenario where you need speed MH then it’s also Factor 10 or so faster so yeah so from from you”
• 30:36 “you have to you know have good data you have to make sure that it’s representative that it’s accurate you have to make sure”

Disclaimers in the video

• 7:35 other: phi4 was released this December with I think 14 billion parameters
• 19:33 preview: it’s a feature which is in preview
• 25:53 subject-to-change: these are preliminary uh results um so I think in order to fully you know bake this and and and ship it we we need to train this dt4 mini on more data
• 26:13 subject-to-change: there’s a risk that because we have these data so it’s you know we have toys and and and so the toys have certain names and that’s the items we have and the customers have certain names so there’s a risk that because we have been training with this data then small the small language model

Presenters (as heard): Claus, Claus Marius Hansen.