Business Central Under the Hood 9: The Engineering Systems Powering Business Central Development

Overview

Part 9 of the Under the Hood series describes how the Business Central Engineering Systems team builds, tests and releases the product. It covers build infrastructure and repositories, the move from Source Depot (based on Perforce) to Git, the branch and release strategy, and the scale of builds and automated testing, with up to 3,000 machines running daily.

The talk also covers security (permission baselines, sign-off rules, component governance, SBOMs, certificate rotation), the past use of Business Central as a UI for the internal build system, the Test Tolerance project, incremental builds in AL-Go for GitHub, developer environments (DevBox, virus scanner exclusions, a shared test tenant), and advice for partners on engineering systems.

Key points

• Scale: about 100 repositories, about 70 build pipelines, about 30 active branches, and up to 3,000 machines per day; every commit triggers builds across 30+ languages.
• Testing runs about 40 million tests per day on average. One day, January 9, reached almost 80 million.
• PR validation uses turn-based test selection. Code coverage data and file inputs of build tasks decide which tests run for the changed files.
• Git migration from Source Depot needed history replay, special handling of binaries with symlinks, and bridges to view pre-Git history.
• Security: a full-time employee must sign off all code, two people for release branches, permission changes are reviewed daily, and component governance generates SBOMs. Certificates are kept in key vaults and rotated automatically.
• Test Tolerance uses heuristics to tell whether a failing test comes from the developer’s change or from other instability. AL-Go for GitHub is getting incremental builds that rebuild only the modified project.
• For partners: use an existing solution such as AL-Go for GitHub or Azure DevOps rather than building custom engineering systems. The internal application team is gradually migrating from Azure DevOps to AL-Go with GitHub, and the migration is still in progress.

Features

AL objects mentioned

As heard in the captions. A name that matches one object page by exact type and name links to it; the others stay as named.

• other “Developer Tools Team Repository” at 2:51
• other “Application Developer Team Repository” at 2:51
• other “Platform Runtime Repository” at 3:11
• other “Engineering Systems Repository” at 3:11
• other “Control Plane” at 3:11
• other “AL-GO” at 23:43
• other “NST” at 26:58
• other “Master branch” at 28:19

Quotes

• 2:11 “we have about a 100 repositories with code bases that we build on a regular basis”
• 6:50 “binaries and stuff in Source Depot that would take up something like 30 gigabytes on your disc for business Central”
• 7:10 “git doesn’t really do binaries well like it does text really well but not binaries”
• 9:51 “in the last 60 days we did that almost 800 times in master and in the 25x branch we did it almost 600 times”
• 11:53 “we use a lot of compute Power Engineering system teams like we have up to 3,000 and machines running on a daily basis doing”
• 12:53 “january 9 for instance we ran almost 80 million tests wow uh on that single day on the product”
• 13:13 “we run yeah I think on average yeah 40 million tests every day something like that”
• 13:54 “with the pr validation we do turn based test selection so depending on what you change in the product we run different tests and”
• 14:14 “we collect code coverage for all of the test that runs in the product and then we also know all of the inputs and”
• 15:55 “supply chain attack is on the rise so there are lots of examples where companies manage to compromise Engineering Systems Packaging Systems npm n”
• 17:17 “no code gets into repository without at least one full-time employee signs off on it and if it’s release branches at least two different”
• 20:09 “we found out that BC is really good at datadriven UI presentation declarative data driven UI present so we essentially connected business Central on”
• 22:30 “if you run 80 million test cases every day if just a single of those are unstable then you get into some issues right”
• 23:43 “the team is working on that Freddy is is driving now is incremental bills so if you have a lot of projects that you’re”
• 25:43 “you just go to a website and you you say Okay create Dev box you give it a name and you give a size”
• 26:38 “when compiling you know it was the our app it was really really slow to deploy right”
• 27:18 “we add exclusions for the virus scanner for specific folders and types of and processes based on our experiences”
• 27:18 “this is still very secure this is not we do V scanning of everything just because this is just a development environment”
• 28:19 “share test tenant they can access practically wide away with the latest bits from from”
• 29:40 “i would piggy back on one that is already out there and something like aure devops”
• 30:20 “I don’t understand how you would how you would be able to work in today’s landscape without having automated bills tests deployments that happens”
• 31:20 “we’re gradually moving them out of our internal asro devop solution slowly it takes time we have a large Legacy system but eventually the”

Disclaimers in the video

• 21:29 not-in-this-release: we don’t use this anymore no unfortunately not no we don’t
• 31:20 coming-later: gradually moving them out of our internal asro devop solution slowly it takes time - migration still in progress

Presenters (as heard): Vincent, Yonas, Freddy.