Business Central Under the Hood episode 3: How Many Users Can Business Central Handle in the Cloud?

Overview

Episode 3 of Business Central Under the Hood (31 minutes, published 2024-05-31) explains how many users Business Central can handle in the cloud. It describes the cloud topology: a cluster of Azure VMs in a virtual machine scale set, each hosting web server and NST components, with a load balancer that understands BC sessions. Clusters typically start at five VMs and can grow to 30 or more. The presenters show real cluster examples for scale-out and VM provisioning time, and explain how database capacity is monitored and increased automatically. The speakers note that some of this machinery is not publicly documented.

The second half covers sources of performance problems and the tools to investigate them: the in-client profiler, the database locks page, the missing indexes page and Application Insights telemetry. It closes with user scalability (customers with hundreds or thousands of users), the scalability guide, and the performance toolkit for testing large customers before go-live.

Key points

• The cloud runs on an Azure VM scale set. Clusters have at least five VMs, and 10, 20, 30 or 40 VMs is normal.
• The load balancer stops sending new sessions to a VM above 60% CPU. Existing sessions stay on that VM. When all VMs exceed the threshold, more VMs are added. Scale-in is intentionally slower than scale-out.
• Database CPU, IO, data IO and log IO are monitored, and database capacity is increased automatically when thresholds are exceeded.
• Capacity is rarely the cause of a performance problem. Many short queries are typically worse in the cloud because of datacenter latency, usually under a millisecond.
• Diagnostic tools: in-client profiler, database locks page (refresh with F5), database missing indexes page (needs several days of running), and Application Insights telemetry. By default only SQL queries slower than 750 ms are emitted.
• There is no hard user limit. Customers have hundreds or thousands of users. Locking between concurrent users, which depends on AL code, is the main constraint.
• The performance toolkit simulates what-if workloads in a cloud environment before go-live, in principle with around a thousand users. The scalability guide includes a real example of 5,000 sales invoices posted in one hour.

Features

AL objects mentioned

As heard in the captions. A name that matches one object page by exact type and name links to it; the others stay as named.

• page “database locks page” at 20:54
• page “database missing indexes page” at 22:15
• table “mixer table” at 21:34

Not found in BC28-30: page “database locks page”, page “database missing indexes page”, table “mixer table”.

Quotes

• 2:09 “we have a VM cluster technically it’s a virtual machine scale set that has a number of VMS and all our VMS are configured”
• 4:10 “we have set a threshold at 60% and if VM goes above 60% CPU load we say it’s it’s getting starting to become overloaded”
• 5:11 “if all VMS get busy like if there’s really a lot of activi everybody’s working you know hard doing a lot of sales order”
• 11:14 “at a12 we say we need three more VMS again the algorithm which we change from time to time it in this case it”
• 11:14 “we can see in this case it took 10 minutes and then 11 minutes before we got the last two VM so okay after”
• 12:35 “uh we we are pretty mature in this area and uh yeah capacity is rarely the source of a performance problem”
• 14:36 “how come we hear reports every now and then that uh our customers they have you know they think they experience BC as being”
• 14:57 “the source of a performance problem can be uh a million things okay but uh we have some uh things we see all the”
• 16:18 “many short queries so for example if you iterate over you have maybe thousands of sales orders or sales lines and somehow in your”
• 16:59 “the third category many short queries I would say it’s probably typically worse in the cloud”
• 17:19 “in the cloud with buildings Data Center buildings far apart there can be a latency typically less than a millisecond but it’s there yeah”
• 24:50 “normally we would not emit these events when a SQL query takes less than 750 milliseconds because if we did we would just overflow”
• 26:30 “we don’t have a hard limit so you can have we have customers with hundreds or thousands of users”
• 26:30 “we have customers with hundreds or thousands of users um adding more users it’s not really a problem because we just scale out and”
• 26:51 “the more users you have yeah that work at the same time the more likely they are to step on each other’s toes”
• 26:51 “that work at the same time the more likely they are to step on each other’s toes and uh”
• 27:31 “we have customers with more than thousand of us more than thousand users many many customers with more than 100 users”
• 27:51 “scalability guide that we released recently”
• 27:51 “5,000 sales invoices posted in one hour it’s a real example from some customer”
• 29:12 “if you are bringing a large customer and you are want to make absolutely sure that they will run fine from the day one”
• 29:32 “we have a toolkit for that it’s called the performance toolkit and it is assigned one of the main goals is to just to”
• 29:52 “in principle yes yes yeah um and U while it runs you can also log in yourself right it’s running in an environment just”

Disclaimers in the video

• 1:09 other: some of the Machinery that is in the cloud is not publicly documented but we are going to talk a little bit about it

Presenters (as heard): Christian.