Production readiness
Status (11 November 2026): not production-ready. The scoped rollout is Windows WEF/WEC
(ForwardedEvents) and firewall syslog → Bower collector → Microsoft Sentinel through the
Logs Ingestion API. All seven readiness gaps have a guide and live evidence. The 72-hour
re-soak (8f615b8) failed on both collectors: growth and an unexplained DCR alert on both,
and the WEC's exact-delivery check is blocked by a missing native export window. WEC latency
now passes, consistent with #178's group commit but not isolated from the change from
credit-limited t3 to m6i.large hosts. The earlier 24-hour soak also failed. Release check 7
stays open. Release packages are unsigned; signing is deferred. The C# collector is the only
implementation; the Rust collector was retired in November 2026.
Each guide is the step-by-step operator procedure for its gap, for the environments that apply (Linux packages, Windows MSI and WEC hosts, Docker, Kubernetes, AWS, Azure VMs and on-premises), and records how it was tested. Evidence covers only the build, scope and scenarios it names. Fixes made after a live run are not live-qualified unless the guide says so.
| # | Gap | Guide | Current result | Evidence | Still open |
|---|---|---|---|---|---|
| 1 | Reliable firewall delivery | Syslog delivery reliability | Qualified for the tested scope over RELP at commit 24ece92. Forced kill, TCP reset, sender reboot, collector stop with sender reboot, restart, collector reboot, blackhole, idle gaps of 5, 30 and 65 minutes, a 15-minute outage and queue-full refusal all had 0 missing events from source to Sentinel, with rsyslog 8.2610.0 as the sender. rsyslog 8.2204.0 lost 6 of 600 at collector stop plus sender reboot. The 8.2606.0 minimum comes from the rsyslog changelog and was not tested. Five RELP review fixes after 24ece92 were tested locally against librelp, not re-run live. Plain TCP/TLS syslog is not crash-lossless: 21 of 6,000 lost at forced kill, 4 of 1,200 at reset, 2 of 600 at collector stop plus sender reboot. | fw-20261101-111911, plain TLS final campaign | A real firewall appliance and the appliance → relay hop; sender power loss with an unacknowledged RELP window; collectors other than Linux EC2. |
| 2 | Duplicate (replay) handling | Replay-safe detections | Qualified live for the content at 4dabe15 (a pre-rebase commit; git diff 4dabe15 cc5e57e -- deploy/sentinel/analytics is empty): zero duplicate incidents and zero missed late events. ws2c included one live owned-window brute-force alert; ws2d validated queries with the rules disabled. Neither qualifies scheduled operation of the final content. No rule has yet run as a scheduled rule on the final content. The fixes in 206684a and e12afe5 were committed after the ws2d run and have not run live. | ws2b, ws2c, ws2d | Scheduled runs of the firewall rule, catch-up path and late-event monitor on the final content; query cost at production volume. |
| 3 | Full queues and low disk | Storage pressure | Tested live on Linux and Windows at 261f61a. Every accepted event reached Sentinel, and the drain-deadlock and stall fixes worked. 7,109 plain-TCP syslog frames were lost before acceptance when the collector restarted while its volume was full. Review fixes after 261f61a are unit-tested only. | ws3b, ws3 | Plain TCP/TLS loss at restart under backpressure (use RELP); ForwardedEvents overflow when Bower stalls. |
| 4 | Upgrades and recovery | Upgrade, backup and recovery | Tested live in one low-rate run. RPM and Docker upgrades under load, crash, forced stop/start and backup/restore lost no accepted events. The MSI kept operator settings only after the bower.env fix: the 264 → 270 upgrade dropped 12 of them. | ws4 | MSI downgrade blocking and release versioning; WEC host loss and disaster-recovery replay; Kubernetes, Azure VM and DEB live runs; post-review fixes not re-run live. |
| 5 | Credential rotation and security failures | Credential rotation and security failures | Tested live, not a production approval. Covered: expired, revoked and disabled Entra credentials; client-secret and certificate rotation; syslog mutual-TLS rotation; HMAC rotation; and an adversarial privacy sweep. Pack rejection and trusted-key rotation were unit-tested only. The Azure collectors ran on macOS, not as packaged services. Replacing the Azure ingestion credential certificate file needs a restart; syslog TLS material reloads automatically. rsyslog drops one frame per refused TLS session: 40 lost during revocation. | ws5 | A live Windows/WEC run; post-campaign fixes not re-run against the cloud; managed identity, workload identity, Key Vault and Secrets Manager live; HMAC dual-key period; pack version pinning; CRLF frame forging under LF syslog framing (use octet counting). |
| 6 | Monitoring and runbooks | Monitoring and runbooks | Drilled live on earlier rule versions. Alerts fired and recovered for silent sources, failed delivery, backlog and collector down. The storage alert did not fire for an already-full queue in that drill. #161's revised queue-storage query fired for both full collectors in the gap 3 run (ws3b, 261f61a); later query and reserve fixes are not live-validated. The TLS-expiry false positive the 24-hour soak found was fixed on main by #171: in the 72-hour re-soak (8f615b8, which includes #171, rules deployed from its monitoring.bicep), syslog-tls-expiry did not fire in 72 hours on the Linux RELP collector, including the F04 syslog TLS certificate rotation. That shows no false positive in that lab configuration; the rule was not tested against a certificate near expiry. | ws6, final campaign | Alerts marked "unverified" in the guide; changes since the drills; management API with Entra sign-in. |
| 7 | 24-hour soak and release validation | Soak and release validation | 72-hour re-soak failed on both collectors (8f615b8, 2026-11-06 to 11-09, eleven scheduled faults, all fault outcomes met). Both failed criteria 3 (growth) and 5 (one unexcused dcr-request-errors alert, cause not determined). The firewall was exact (2,693,520 at every stage). The WEC had 2,693,520 accepted and at Sentinel with 0 missing and 0 duplicates, but criterion 1 is blocked: the last native window was not exported before the hosts terminated. Latency passed on both (p99 7.9 s and 11.5 s); restarts were only the scheduled ones; no privacy-marker matches. Earlier 24-hour soak also failed on both collectors (soak of 1a4c4c2). The firewall failed criteria 3 (growth) and 5 (alerts: the TLS-expiry false positive). The WEC failed 3, 4 and 5: whole-run p99 latency was 881.8 s against an 18.16 s limit, on a CPU-credit-limited t3 host, first attributed to reader cost growing with ForwardedEvents (not profiled), and two true alerts came from retention contention. All 896,400 source events per collector reached Sentinel. Firewall identity sets matched exactly; the WEC had none missing, but accepted-stage extras are unverified. There were no process restarts and no privacy-marker matches. | resoak-20261106-092539, soak-20261102-015614 | Firewall and WEC growth slopes; the cause of the DCR alert; a harness that exports its last native window without an operator; token refresh and journald rotation, registered natural events that the re-soak did not show; the WEC on burstable hosts, not re-tested; the hourly-retention purge scenario, not repeated (defect 2); suppression replay around a bookmark update, unqualified (defect 5). Release check 7 needs a re-soak that passes all six criteria for both collectors. |
Before you release
- Work through the release checklist for the exact build you are shipping, and record its version and artifact hashes.
- Confirm every "Still open" item above is either closed by new evidence or accepted as a documented risk for your deployment.
- Re-run the parts of gaps 1–6 that your build or configuration changes, using the procedure in each guide, and keep failed attempts.
- Remove every temporary cloud resource and keep the cleanup audit.
The operations qualification record keeps the history of earlier gates and campaigns.