Skip to main content

Production readiness

Status (11 November 2026): not production-ready. The scoped rollout is Windows WEF/WEC (ForwardedEvents) and firewall syslog → Bower collector → Microsoft Sentinel through the Logs Ingestion API. All seven readiness gaps have a guide and live evidence. The 72-hour re-soak (8f615b8) failed on both collectors: growth and an unexplained DCR alert on both, and the WEC's exact-delivery check is blocked by a missing native export window. WEC latency now passes, consistent with #178's group commit but not isolated from the change from credit-limited t3 to m6i.large hosts. The earlier 24-hour soak also failed. Release check 7 stays open. Release packages are unsigned; signing is deferred. The C# collector is the only implementation; the Rust collector was retired in November 2026.

Each guide is the step-by-step operator procedure for its gap, for the environments that apply (Linux packages, Windows MSI and WEC hosts, Docker, Kubernetes, AWS, Azure VMs and on-premises), and records how it was tested. Evidence covers only the build, scope and scenarios it names. Fixes made after a live run are not live-qualified unless the guide says so.

#GapGuideCurrent resultEvidenceStill open
1Reliable firewall deliverySyslog delivery reliabilityQualified for the tested scope over RELP at commit 24ece92. Forced kill, TCP reset, sender reboot, collector stop with sender reboot, restart, collector reboot, blackhole, idle gaps of 5, 30 and 65 minutes, a 15-minute outage and queue-full refusal all had 0 missing events from source to Sentinel, with rsyslog 8.2610.0 as the sender. rsyslog 8.2204.0 lost 6 of 600 at collector stop plus sender reboot. The 8.2606.0 minimum comes from the rsyslog changelog and was not tested. Five RELP review fixes after 24ece92 were tested locally against librelp, not re-run live. Plain TCP/TLS syslog is not crash-lossless: 21 of 6,000 lost at forced kill, 4 of 1,200 at reset, 2 of 600 at collector stop plus sender reboot.fw-20261101-111911, plain TLS final campaignA real firewall appliance and the appliance → relay hop; sender power loss with an unacknowledged RELP window; collectors other than Linux EC2.
2Duplicate (replay) handlingReplay-safe detectionsQualified live for the content at 4dabe15 (a pre-rebase commit; git diff 4dabe15 cc5e57e -- deploy/sentinel/analytics is empty): zero duplicate incidents and zero missed late events. ws2c included one live owned-window brute-force alert; ws2d validated queries with the rules disabled. Neither qualifies scheduled operation of the final content. No rule has yet run as a scheduled rule on the final content. The fixes in 206684a and e12afe5 were committed after the ws2d run and have not run live.ws2b, ws2c, ws2dScheduled runs of the firewall rule, catch-up path and late-event monitor on the final content; query cost at production volume.
3Full queues and low diskStorage pressureTested live on Linux and Windows at 261f61a. Every accepted event reached Sentinel, and the drain-deadlock and stall fixes worked. 7,109 plain-TCP syslog frames were lost before acceptance when the collector restarted while its volume was full. Review fixes after 261f61a are unit-tested only.ws3b, ws3Plain TCP/TLS loss at restart under backpressure (use RELP); ForwardedEvents overflow when Bower stalls.
4Upgrades and recoveryUpgrade, backup and recoveryTested live in one low-rate run. RPM and Docker upgrades under load, crash, forced stop/start and backup/restore lost no accepted events. The MSI kept operator settings only after the bower.env fix: the 264 → 270 upgrade dropped 12 of them.ws4MSI downgrade blocking and release versioning; WEC host loss and disaster-recovery replay; Kubernetes, Azure VM and DEB live runs; post-review fixes not re-run live.
5Credential rotation and security failuresCredential rotation and security failuresTested live, not a production approval. Covered: expired, revoked and disabled Entra credentials; client-secret and certificate rotation; syslog mutual-TLS rotation; HMAC rotation; and an adversarial privacy sweep. Pack rejection and trusted-key rotation were unit-tested only. The Azure collectors ran on macOS, not as packaged services. Replacing the Azure ingestion credential certificate file needs a restart; syslog TLS material reloads automatically. rsyslog drops one frame per refused TLS session: 40 lost during revocation.ws5A live Windows/WEC run; post-campaign fixes not re-run against the cloud; managed identity, workload identity, Key Vault and Secrets Manager live; HMAC dual-key period; pack version pinning; CRLF frame forging under LF syslog framing (use octet counting).
6Monitoring and runbooksMonitoring and runbooksDrilled live on earlier rule versions. Alerts fired and recovered for silent sources, failed delivery, backlog and collector down. The storage alert did not fire for an already-full queue in that drill. #161's revised queue-storage query fired for both full collectors in the gap 3 run (ws3b, 261f61a); later query and reserve fixes are not live-validated. The TLS-expiry false positive the 24-hour soak found was fixed on main by #171: in the 72-hour re-soak (8f615b8, which includes #171, rules deployed from its monitoring.bicep), syslog-tls-expiry did not fire in 72 hours on the Linux RELP collector, including the F04 syslog TLS certificate rotation. That shows no false positive in that lab configuration; the rule was not tested against a certificate near expiry.ws6, final campaignAlerts marked "unverified" in the guide; changes since the drills; management API with Entra sign-in.
724-hour soak and release validationSoak and release validation72-hour re-soak failed on both collectors (8f615b8, 2026-11-06 to 11-09, eleven scheduled faults, all fault outcomes met). Both failed criteria 3 (growth) and 5 (one unexcused dcr-request-errors alert, cause not determined). The firewall was exact (2,693,520 at every stage). The WEC had 2,693,520 accepted and at Sentinel with 0 missing and 0 duplicates, but criterion 1 is blocked: the last native window was not exported before the hosts terminated. Latency passed on both (p99 7.9 s and 11.5 s); restarts were only the scheduled ones; no privacy-marker matches. Earlier 24-hour soak also failed on both collectors (soak of 1a4c4c2). The firewall failed criteria 3 (growth) and 5 (alerts: the TLS-expiry false positive). The WEC failed 3, 4 and 5: whole-run p99 latency was 881.8 s against an 18.16 s limit, on a CPU-credit-limited t3 host, first attributed to reader cost growing with ForwardedEvents (not profiled), and two true alerts came from retention contention. All 896,400 source events per collector reached Sentinel. Firewall identity sets matched exactly; the WEC had none missing, but accepted-stage extras are unverified. There were no process restarts and no privacy-marker matches.resoak-20261106-092539, soak-20261102-015614Firewall and WEC growth slopes; the cause of the DCR alert; a harness that exports its last native window without an operator; token refresh and journald rotation, registered natural events that the re-soak did not show; the WEC on burstable hosts, not re-tested; the hourly-retention purge scenario, not repeated (defect 2); suppression replay around a bookmark update, unqualified (defect 5). Release check 7 needs a re-soak that passes all six criteria for both collectors.

Before you release​

  1. Work through the release checklist for the exact build you are shipping, and record its version and artifact hashes.
  2. Confirm every "Still open" item above is either closed by new evidence or accepted as a documented risk for your deployment.
  3. Re-run the parts of gaps 1–6 that your build or configuration changes, using the procedure in each guide, and keep failed attempts.
  4. Remove every temporary cloud resource and keep the cleanup audit.

The operations qualification record keeps the history of earlier gates and campaigns.