Optimizing BC Code History Downloads with HTTP Range Requests

Key points

• Removing the global SilentlyContinue error setting exposed hidden bugs: an arithmetic parsing mistake, null array indexing, missing Applications folders, and a vNext grep mismatch that caused re-processing and duplicates.
• Because a ZIP keeps its central directory at the end, the pipeline fetches the last 64KB, finds the Applications entries, and downloads only that byte range with HTTP Range requests.
• In a German benchmark the download dropped from 18 seconds to about 1 second, with roughly 88% less bandwidth. Combined compute across both daily workflows fell from nearly 12 days to about 38 hours.
• A small stdlib-only Python script parses the ZIP and decompresses in memory, a PowerShell wrapper and curl handle the HTTP calls, and every failure falls back to the old full download.
• Targeted git fetches and the cleanup of 196 obsolete vNext branches also cut run time.

Quotes

• “The error suppression had been papering over null references, parsing bugs, missing folder handling, and incorrect commit patterns - all at once.” (Demonstrates how silent error handling can hide systemic problems in automation scripts, requiring explicit error checking to reveal cascading issues.)

AL objects mentioned

As named in the post. A name that matches one object page by exact type and name links to it; the others stay as named.

• other “MSDyn365BC.Sandbox.Code.History”
• other “MSDyn365BC.Code.History”
• other “Auto_load_versions.ps1”
• other “Auto_load_versions_vNext.ps1”
• other “download_range_helper.py”
• other “Download-ApplicationsRange.ps1”

Context

• Features: HTTP Range request optimization, ZIP central directory parsing, GitHub Actions matrix parallelization, PowerShell error handling, Git workflow automation, Bandwidth reduction, In-memory decompression, Selective artifact extraction

Source: Stefan Maron, community blog. Summary, key points and quotes are derived (CONTENT-NOTICE.md); read the original for the full text.