BC Observatoryunofficial, made by waldo
community postAL Developmentcommunity - not Microsoftreviewed - checked by Opus

AL Runner: A Weekend of Autonomous Agents, a Windows Cross-Check, and 59% of Microsoft's Tests Passing

Stefan Maron describes a weekend of autonomous agent loops on AL Runner, shipped as v2.11.0, and its companion AL language test suite, which grew by 832 tests. A new nightly Windows workflow found a missing TestPermissions property in about 100 test codeunits and 8 behavioral differences between BC on Linux and real BC. A full run of Microsoft's BaseApp tests now passes 59.2%, and a hang was traced to a dropped report MaxIteration property.

Open in Claudehttps-stefanmaron-com-posts-al-runner-autonomous-agent-weekend--030ecee688.md

post Stefan Maron

Stefan Maron · 2026-09-08 · 1,359 words · about 6 min

"Every one of those tests pins its answer against a real BC service tier"
Show the post hereRead on Stefan Maron

Key points

  • AL Runner shipped v2.11.0 after 351 merged PRs. The companion test suite went from 2,346 to 3,178 tests in five days.
  • A manual check against a SaaS sandbox, followed by a nightly Windows container workflow, exposed tests passing on Linux CI but failing on real BC. Most failures came from a missing TestPermissions property.
  • Eight Linux-versus-Windows differences were filed, including lost SingleInstance codeunit state and IsolatedStorage encryption doing nothing under Linux. Five are closed.
  • AL Runner passes 23,985 of 40,530 Microsoft BaseApp tests (59.2%). Microsoft itself disables 16,141 of them, so a figure adjusted to the tests Microsoft actually runs is still being worked out.
  • Rebuilding report metadata from SymbolReference.json dropped MaxIteration, so one report dataitem looped 101,001 times. The bug was fixed at 5 call sites.

Quotes

  • “Every one of those tests pins its answer against a real BC service tier” (Shows the rigor of the test suite - tests validate against actual running BC instances, not mocks, ensuring reliability.)
  • “59.2% is coverage of everything AL Runner can discover, not coverage of what Microsoft actually runs day to day.” (Clarifies what test coverage percentages mean and that Microsoft itself disables over 40% of available tests, providing context for improvement targets.)

AL objects mentioned

As named in the post. A name that matches one object page by exact type and name links to it; the others stay as named.

  • other “SymbolReference.json”

Context

  • Features: AL Runner testing framework, Autonomous agent loops for implementation and review, Windows and Linux cross-platform testing, Nightly automated test workflows, Permission-based test validation, Report metadata reconstruction, TestPermissions property handling, Microsoft base-app test execution
  • Versions: v2.11.0

Source: Stefan Maron, community blog. Summary, key points and quotes are derived (CONTENT-NOTICE.md); read the original for the full text.