Skip to main content

3. Full queues and low disk space

Gap: prove backpressure, early warning and safe recovery when storage fills, without ever discarding unacknowledged events.

Scope: the scoped rollout, Windows WEF/WEC (ForwardedEvents) and firewall syslog (RFC 5424 over mutual TLS) to the C# Bower collector to Microsoft Sentinel (Logs Ingestion API). Releases and packages are unsigned at the moment (signing is deferred); Bower still refuses unsigned or untrusted packs.

Status: qualified on AWS for the scoped rollout, with the limits below. Four storage defects found on AWS (including the page-ceiling stall workstream 6 also reported) are fixed. Free-space watermarks, storage status, collector_health storage fields, the queue-storage alert and host checks are in place. A final run on 1 GiB (Linux) and 2 GiB (Windows) queue volumes reconciled every accepted sequence-numbered event in Sentinel through queue saturation, the watermarks, a completely full volume and a restart while full. Plain syslog over TCP still loses frames held in socket buffers if the collector restarts while backpressured (measured; RELP senders do not). Releases are unsigned. See how we tested this and known limits.

Contents:

  1. How Bower behaves when storage fills
  2. What changed
  3. Capacity planning
  4. Configuration reference
  5. Early-warning signals
  6. Procedures by environment
  7. WEC ForwardedEvents size and retention
  8. Recovery runbook
  9. Troubleshooting
  10. How we tested this
  11. Known limits

1. How Bower behaves when storage fills​

Bower keeps every accepted event in one SQLite database (BOWER_QUEUE_PATH, WAL mode, synchronous=FULL). An event is accepted once its queue transaction has committed; it is deleted only after the destination acknowledged it and the acknowledged-row retention has passed. Storage can run out in five places, checked in this order on every new event:

LimitWhat it boundsDefaultWhen reached
Free-space critical watermarkFree bytes on the queue volumebelow max(256 MiB, 5%)New events refused (storage-critical)
Pre-write physical budgetDatabase + WAL bytes4 × M + 16 MiB minus a half-ceiling reserveAll writes refused until the WAL resets
Intake page limitDatabase pages in use85% of (M + 4 MiB)New events refused (queue-capacity)
Logical cap BOWER_QUEUE_MAX_BYTES (M)Undelivered payload bytes10 GiBNew events refused (queue-capacity)
SQLite page ceilingDatabase file1.75 × (M + 4 MiB)Only settlement can reach it

M is BOWER_QUEUE_MAX_BYTES. The free-space and physical-budget checks can bind before the intake page limit. With enough free space and a resettable WAL, a queued row takes about 2.2 times its payload on disk once its ledger entry and indexes are counted (measured 2.6 KB per syslog event and 3.0 KB per Windows 4625 event for 1.2 KB and 1.4 KB payloads). The queue therefore refuses new events when undelivered payload is about 40–45% of BOWER_QUEUE_MAX_BYTES, not 100%, and retained acknowledged rows count against the same pages. Size the queue in events, with the formula below.

What keeps running when new events are refused​

Free-space watermark refusals apply to new events only. The candidate reserves room for leasing, delivery, acknowledgement, retries, dead-lettering and retention when the intake page limit is reached. Progress is not guaranteed if the physical volume is full or a pinned WAL exhausts the write budget; those failures require space or the reader to be released before writes can resume:

  • The free-space watermarks never refuse settlement writes.
  • Settlement may grow the database up to 1.75 × (M + 4 MiB), past the intake page limit. Acknowledging a row enlarges it, and pages filled by appends split; settling a full backlog grew the database to 140% of the intake limit in the regression test.
  • When pages in use reach 80% of the intake limit, or the volume is below its warning watermark, retention removes acknowledged rows older than BOWER_QUEUE_PRESSURE_RETENTION_MINUTES (default 60), oldest first. The 95% health warning alone does not waive that floor. For critical disk pressure, or an actual intake page refusal, eligible acknowledged rows can be reclaimed below the floor; page-refusal reclaim stops once enough space is available, checked between batches. Undelivered, retrying and dead-lettered rows are never removed. With an archive configured, only rows the archive verifiably holds are removed.
  • A write over the physical budget first attempts a wal_checkpoint(TRUNCATE) (at most every 10 seconds), so a WAL that readers kept from resetting no longer refuses acknowledgements until the daily maintenance job.

How each source reacts to a refusal​

SourceOn refusalEvent kept byLoss window
HTTP POST /v1/events, pack endpoints, relay intakeHTTP 503; nothing committedThe producer, which must retryNone if the producer retries
Syslog TCP/TLSHolds the frame, stops reading, keeps the connection, retries every second (retries counter)Kernel socket buffers, then the sender's queueFrames in kernel buffers are lost if the collector restarts or the connection resets (measured below)
Syslog RELPWithholds the acknowledgement; after 30 seconds of refusals closes the session (relpBusyCloses)The sender, which resends unacknowledged messagesNone for a RELP sender that resends
Syslog UDPDrops and counts (dropped)NobodyEvery refused datagram
Windows Event Log / WEC ForwardedEventsStops reading; the bookmark stays before the first unread event; subscription state is backpressureThe event log channelEvents overwritten when the channel wraps (gaps counter)

Disk-full errors from SQLite during intake (SQLITE_FULL, or an I/O error while the volume is critical) are reported as the same backpressure (storage-full). Before this change they closed the syslog connection, losing buffered frames.

Filesystem cases​

CaseWhat happens nowData at risk
Volume below the warning watermarkWarning state, /v1/status and log 1501; acknowledged rows past the floor are reclaimed so new events reuse pagesNone
Volume below the critical watermarkNew events refused (storage-critical); /health reports degraded; delivery and acknowledgement continueNone accepted; syslog TCP kernel buffers on restart
WAL growth (checkpoint starvation)Bounded by the pre-write budget; the next write over budget truncates the WAL once readers release itNone
Checkpoint cannot grow the database (volume full)Committed pages remain in the WAL. Writes can fail until space returns and a checkpoint succeedsPreviously accepted events remain durable subject to the underlying filesystem; in-flight syslog frames remain at risk
SQLITE_FULL during commitThe transaction rolls back; intake reports storage-full; settlement retries after 5 seconds; leases expire and are re-leasedNone (acknowledged uploads may be replayed: duplicates, not loss)
Cursor or bookmark write failsThe bookmark stays at the last committed position; events are re-read and deduplicated by event idNone
Logs fill the same diskBower writes no log files: Linux logs go to journald, Windows warnings to the Application log. Storage logs are bounded (below)Depends on what else shares the volume
Restart on a full volumeThe collector starts and serves reads; new events are refused until space returns; settlement fails until a write fitsSyslog frames in kernel buffers at the moment of restart
Retained acknowledgements fill the pagesFixed: pressure retention reclaims themNone

Bounded logging​

Storage conditions log state changes, not events. Every message is metadata only: byte counts, percentages, reasons, counts. Never paths, payloads or event ids.

Event idLevelWhen
1500ErrorVolume entered the critical watermark
1501WarningVolume entered the warning watermark
1502InformationFree space recovered
1503WarningFree space cannot be measured (watermarks not enforced; the queue limits still apply)
1504WarningRefusal summary, at most once a minute: count, latest reason, storage refusals
1505WarningStorage sampling failed (at most once a minute)
1506WarningRetention removed acknowledged rows early (trigger, page use before and after)
1002ErrorDelivery loop failure: first occurrence, then every 12th consecutive one (about one a minute)

On AWS, a 19-minute queue-full run logged 45 lines (7.4 KB) in total.


2. What changed​

ChangeWhy
Free-space watermarks (BOWER_STORAGE_*) with hysteresis, enforced on every insertIntake stops before the volume is exhausted; nothing previously looked at free disk space
Intake page limit at 85% and a 1.75× settlement ceilingDefect fixed: a backlog that filled the page ceiling could never be leased or acknowledged (SQLITE_FULL), so the queue never drained. Seen on both AWS collectors
Pressure retention of acknowledged rowsDefect fixed: acknowledged rows kept for duplicate detection (7 days by default) filled the pages and refused all intake with nothing undelivered. At 10/sec and the default 10 GiB cap this happens after about 4.5 days
Intake-blocked reclaim ignores the retention floorDefect fixed: after a full backlog drained, every page held a fresh acknowledgement and intake stayed refused for the floor duration
WAL truncate before refusing a writeDefect fixed: a WAL that checkpoints could not reset exceeded the write budget and refused acknowledgements (72.6 MB WAL on the Windows collector)
SQLITE_FULL and disk-full I/O errors on intake are backpressureSyslog keeps the frame and the connection; WEC keeps the bookmark
storage block in /v1/status and the management heartbeat; /health degraded when critical; deliveryStatus backpressure when the volume is critical or pages reach 95%Early warning, metadata only
queuePages (workstream 6's page snapshot) now reports the limit at which new events are refused, plus ceilingBytesOne page computation for status, health records, alerts and host checks
collector_health storage gains volume state, free space, thresholds, WAL size, storage refusals and pressure purges; problem codes storage-critical and storage-lowThe queue-storage alert and host checks see the volume (see monitoring)
The collector's own health record uses the settlement reserve, skips the critical watermark and may use a small logical reserve past BOWER_QUEUE_MAX_BYTES (a sixteenth of it, at most 1 MiB)Defect fixed (workstream 6): the storage alert could not see a full queue because its records were refused, including when large events filled the byte cap with pages to spare
A large event refused by the page limit triggers retention at any page useDefect fixed: an event needing more pages than remained was refused below the 80% trigger, and its source retried it forever
New queues use incremental auto-vacuum; retention returns free pages to the file system in 8 MiB steps under pressure, at most 10 seconds per run with one checkpoint at the end. It keeps a reserve of free pages for the undelivered rows to settle (about twice their payload, plus 64 pages) and releases the rest, even while a backlog waits for the destination. Without disk pressure it waits until pages in use are back within the intake limitPruning reclaims disk, not only pages (existing queues keep reusing free pages); the reserve lets a draining backlog reuse pages instead of regrowing the file. Also runs when archive coverage refuses purging
Rust queue: same intake limit and settlement ceilingEither runtime opens and drains a queue the other grew while settling
Archive-backed retention purges only the archive-verified sequencesA concurrent purge could shift a verified batch onto unverified rows
Bounded storage logs; delivery-loop failure logs throttledA full disk must not also fill the journal
Kubernetes manifest sets BOWER_QUEUE_MAX_BYTES (5 GiB), a 16Gi claim and ephemeral-storageThe 10 GiB default, and settlement growth, could exceed the old 10Gi claim

The Rust collector (bower-collector-rs, retired in November 2026) shared the page model: the same 85% intake limit and 1.75× settlement ceiling, so either runtime opened and drained a queue the other grew while settling. It had no watermarks, pressure retention, incremental vacuum or storage status; see known limits. The Sentinel schema, DCR and table are unchanged.


3. Capacity planning​

Inputs​

SymbolMeaningMeasured on AWS
RAccepted events per second (after selection)10/sec steady, 100/sec burst in the pilot envelope
TLongest destination outage the collector must ride through1 hour (pilot), 24 or 72 hours for weekend coverage
FDatabase bytes per queued row2.6 KB syslog, 3.0 KB Windows 4625 (use 3 KB)
PPayload bytes per event (undeliveredBytes / undelivered rows)1.2 KB syslog, 1.4 KB Windows 4625
T_floorBOWER_QUEUE_PRESSURE_RETENTION_MINUTES60 minutes
EForwardedEvents bytes per forwarded event2.2 KB (Security 4625, ContentFormat=Events)

Measure F and P on your own traffic: run steady load for ten minutes with the destination blocked, then divide storage.queueDatabaseBytes by the undelivered row count, and queue.undeliveredBytes by the same count.

Formula​

  1. Rows to hold: N = R × T + R × T_floor (the backlog plus the acknowledged rows pressure retention must keep).
  2. Queue cap: M ≥ N × F / 0.85 − 4 MiB. Round up. The logical rule M ≥ R × T × P is then always met because F > P.
  3. Database ceiling on disk: D = 1.75 × (M + 4 MiB).
  4. Queue volume: V ≥ max((D + 128 MiB + 1 GiB) / (1 − r), (D + 128 MiB) / (0.9 − r)) with the default watermarks (warning below max(1 GiB, 10%) of the filesystem). r is the share reserved for root, which the service cannot use: 0 for xfs and for ext4 created with -m 0, 0.05 for ext4 defaults. V is the filesystem size; a disk or claim of that size formats to about 2% less on ext4 (inode tables and journal), so request V / 0.98. The 128 MiB covers WAL bursts (a 5,000-row retention purge wrote up to 72 MB of WAL on AWS, truncated within 5 seconds; retention now purges in 1,000-row transactions, see Windows Event Log source) and shared memory. Add archive and spool space if they share the volume.
  5. ForwardedEvents: S ≥ R × E × T_stall × 1.2, where T_stall is the longest time Bower itself may stop reading (service down, queue full, volume critical). A destination outage alone does not use this budget while Bower's queue has room.
  6. Acknowledged retention: keeping BOWER_QUEUE_RETENTION_HOURS (default 168) of acknowledged rows needs R × hours × 3600 × F more. At 10/sec, 168 hours is about 18 GB. Either budget it, or lower the retention (for example 24 hours) and let pressure retention trim it under load.

Worked examples (F = 3 KB, E = 2.2 KB, 1-hour floor, xfs: r = 0)​

RateOutageEventsBOWER_QUEUE_MAX_BYTES (M)Database ceiling 1.75 × (M + 4 MiB)Queue volumeForwardedEvents for the same stall
10/sec1 hour36,000268435456 (256 MiB)455 MiB2 GiB91 MiB
10/sec24 hours864,0003221225472 (3 GiB)5.3 GiB7 GiB2.1 GiB
10/sec72 hours2,592,0009663676416 (9 GiB)15.8 GiB18 GiB6.4 GiB
100/sec1 hour360,0002684354560 (2.5 GiB)4.4 GiB6 GiB906 MiB
100/sec24 hours8,640,00032212254720 (30 GiB)52.5 GiB59 GiB21.2 GiB
100/sec72 hours25,920,00093415538688 (87 GiB)152.3 GiB170 GiB63.7 GiB

The default BOWER_QUEUE_MAX_BYTES (10 GiB) holds about 3 million rows: roughly 84 hours at 10/sec or 8 hours at 100/sec, less the acknowledged rows still retained.

ForwardedEvents above about 4 GiB is impractical on one WEC: plan shorter Bower stall budgets at 100/sec (see section 7).


4. Configuration reference​

All settings are environment variables of the collector (/etc/bower/bower.env on Linux, the service Environment value on Windows, the pod spec on Kubernetes). They apply to the C# collector only.

VariableDefaultRangeMeaning
BOWER_QUEUE_PATH/var/lib/bower/queue.db (packages)pathQueue database. Put its directory on the dedicated volume
BOWER_QUEUE_MAX_BYTES10737418240 (10 GiB)≥ 1 MiBM above. Raising it later is safe; lowering it below the current database is refused at start. The collector's own health records may exceed it by min(M/16, 1 MiB)
BOWER_QUEUE_RETENTION_HOURS1681–2160Acknowledged rows kept for duplicate detection
BOWER_QUEUE_PRESSURE_RETENTION_MINUTES601 to the retentionAcknowledged rows younger than this survive pressure retention (unless intake is blocked)
BOWER_STORAGE_WARNING_FREE_BYTES1073741824 (1 GiB)≥ critical bytesWarning when free space is below max(bytes, percent)
BOWER_STORAGE_WARNING_FREE_PERCENT100–50, ≥ critical percent
BOWER_STORAGE_CRITICAL_FREE_BYTES268435456 (256 MiB)≥ 16 MiBNew events refused below max(bytes, percent)
BOWER_STORAGE_CRITICAL_FREE_PERCENT50–50
BOWER_DELIVERY_LEASE_SECONDS / BOWER_DELIVERY_TIMEOUT_SECONDS300 / 240see queue durabilityThe pilot used 90/60

A state is left only once free space clears its threshold by max(16 MiB, 1% of the volume), so intake does not flap. Free space is measured as the service account sees it (statvfs f_bavail on Linux, GetDiskFreeSpaceEx on Windows, which follows NTFS mounted folders). Reserved root blocks (ext4 keeps 5% by default) are excluded. Non-default values append to the reported configuration hash; defaults keep existing hashes.

The thresholds suit volumes of about 10 GiB and larger. On a small dedicated volume set them explicitly (the AWS lab used 256 MiB warning and 128 MiB critical on 1 GiB, and 512/256 MiB on 2 GiB, both with percentages at 0). Keep the critical threshold large enough for a retention purge's WAL burst and settlement growth: at least 128 MiB.


5. Early-warning signals​

GET /v1/status (bearer token) reports metadata only:

  • queuePages: pageSize, pageLimitBytes (in-use bytes at which new events are refused), usedBytes, freeListBytes, usedPercent (100 = refused) and ceilingBytes (largest size settlement may reach).
  • storage: the queue volume and storage refusals (below). The management heartbeat carries the same object (older management servers ignore it).
storage fieldMeaningSuggested alert
statenormal, warning, critical, unknownPage on critical; ticket on warning or unknown for 15 minutes
intakePausedTrue while the critical watermark refuses new eventsPage
freeBytes, totalBytes, freePercentQueue volume as the service sees itTrend; complements host disk alerts
warningFreeBytes, criticalFreeBytesEffective thresholds—
stateChangedAt, warningTransitions, criticalTransitionsSince startFlapping: more than 3 transitions an hour
queueFileBytes, queueDatabaseBytes, queueWalBytesPhysical queue filesWAL above 10% of the database for 30 minutes
queueMaximumBytes, queueUndeliveredBytes, queueUtilisationPercentLogical cap and useInformational (the page limit binds first)
oldestPendingAgeSecondsAge of the oldest undelivered eventPage above 15 minutes (or your SLA)
pressurePurgedEvents, lastPressurePurgeAtAcknowledged rows removed early since startTicket on any increase: retention is undersized
storageRefusals, storageRefusedBytes, lastStorageRefusalAt, lastStorageRefusalReasonRefusals caused by free space (storage-critical, storage-full); a subset of intake.capacityRefusals plus intake.privacyAlertRefusalsPage on any increase

Existing signals that matter here: intake.capacityRefusals and lastRefusalReason (all refusals, shape shared with the Rust collector), queue.oldestUndelivered, syslog[].retries, dropped and relpBusyCloses, windowsEventLog[].state (backpressure), gaps and bookmarkRecordId, /health (degraded when critical, dead-lettered or a source is overdue), and the heartbeat deliveryStatus.

In Sentinel and on the host​

With BOWER_HEALTH_RECORD_INTERVAL_SECONDS set (monitoring), each collector_health record carries, under Attributes.storage, MiB and percent values only: queuePagesUsedPercent, queuePagesUsedMiB, queueFreeListMiB, queuePageLimitMiB, queueCeilingMiB, volumeState, intakePaused, volumeFreeMiB, volumeFreePercent, volumeWarningFreeMiB, volumeCriticalFreeMiB, queueWalMiB, and window deltas storageRefusals and pressurePurgedEvents. Problem codes storage-low (medium) and storage-critical (high) join problems[]; backpressure also applies at 95% page use or a critical volume. The record is queued from the settlement reserve, so it still arrives while new events are refused (measured below).

The queue-storage alert fires on page use, undelivered bytes, volumeState warning or critical, intakePaused, a critical record in the window, backpressure, or refusals in the window; volumeState unknown is incomplete coverage. The host checks (Linux, Windows) report queue-volume WARNING/CRITICAL from storage.state and queue-pages from queuePages.usedPercent. Runbook R5 in the monitoring guide links back to section 8.


6. Procedures by environment​

Each procedure gives the queue its own volume, sets limits, verifies, and can be rolled back. Do them during a maintenance window: moving a queue means stopping the collector, and a syslog sender connected while the collector stops loses whatever sits in the kernel socket buffers. Stop or pause the sender's forwarding first.

6.1 Linux packages (RPM or DEB, systemd)​

Purpose: queue on a dedicated filesystem at /var/lib/bower, sized from section 3, with watermarks and journald limits.

Prerequisites: root; the collector package installed (installers); a spare block device of size V.

  1. Check the queue is drained or note its size, then stop the collector and the syslog sender's forwarding.

    TOKEN=$(sudo cat /etc/bower/secrets/ingest-token) # or your BOWER_INGEST_TOKEN_FILE
    curl -s -H "Authorization: Bearer $TOKEN" http://127.0.0.1:4319/v1/status | jq '.queue'
    sudo systemctl stop rsyslog # or pause the appliance's forwarding
    sudo systemctl stop bower
  2. Create the filesystem. xfs has no reserved root blocks; for ext4 use -m 0 so the service account sees the whole volume.

    lsblk -o NAME,SIZE,TYPE,MOUNTPOINT # identify the empty disk, e.g. /dev/nvme1n1
    sudo mkfs.xfs -L bowerqueue /dev/nvme1n1
  3. Move existing state onto it, preserving ownership and mode.

    sudo mkdir -p /mnt/bowerqueue && sudo mount LABEL=bowerqueue /mnt/bowerqueue
    sudo rsync -aHAX /var/lib/bower/ /mnt/bowerqueue/
    sudo umount /mnt/bowerqueue
    echo 'LABEL=bowerqueue /var/lib/bower xfs defaults,noatime 0 2' | sudo tee -a /etc/fstab
    sudo mount /var/lib/bower
    sudo chown bower:bower /var/lib/bower && sudo chmod 0700 /var/lib/bower

    Do not add nofail without the next step: the service would write into the bare mount point on the root filesystem.

  4. Make the service require the mount.

    sudo install -d /etc/systemd/system/bower.service.d
    printf '[Unit]\nRequiresMountsFor=/var/lib/bower\n' | sudo tee /etc/systemd/system/bower.service.d/queue-volume.conf
    sudo systemctl daemon-reload
  5. Set limits in /etc/bower/bower.env (example: 10/sec, 24-hour outage, 8 GiB volume).

    sudo tee -a /etc/bower/bower.env >/dev/null <<'EOF'
    BOWER_QUEUE_MAX_BYTES=3221225472
    BOWER_QUEUE_RETENTION_HOURS=24
    BOWER_STORAGE_WARNING_FREE_BYTES=1073741824
    BOWER_STORAGE_CRITICAL_FREE_BYTES=268435456
    EOF
  6. Bound the journal so logs cannot fill the root volume, then start everything.

    sudo install -d /etc/systemd/journald.conf.d
    printf '[Journal]\nSystemMaxUse=1G\nSystemKeepFree=2G\nRateLimitIntervalSec=30s\nRateLimitBurst=10000\n' |
    sudo tee /etc/systemd/journald.conf.d/bower.conf
    sudo systemctl restart systemd-journald
    sudo systemctl start bower && sudo systemctl start rsyslog
  7. Verify.

    TOKEN=$(sudo cat /etc/bower/secrets/ingest-token)
    curl -s -H "Authorization: Bearer $TOKEN" http://127.0.0.1:4319/v1/status | jq '{volume: (.storage | {state, freeBytes, totalBytes, warningFreeBytes, criticalFreeBytes, queueMaximumBytes}), pages: .queuePages.usedPercent}'
    findmnt /var/lib/bower
    systemctl show bower -p RequiresMountsFor

    Expected: state normal; totalBytes is the dedicated volume, not the root disk; queueMaximumBytes matches; RequiresMountsFor=/var/lib/bower.

Pass: status shows the dedicated volume and normal; journalctl --disk-usage stays below SystemMaxUse; a filler test (section 8.6) produces warning then critical and recovers. Fail: totalBytes equals the root disk (wrong mount), or state is unknown.

Rollback: stop the collector, copy /var/lib/bower back to the root filesystem, remove the fstab line and the drop-in, systemctl daemon-reload, start. Never delete queue.db, queue.db-wal or queue.db-shm separately: they are one database.

6.2 Windows (MSI service, including WEC hosts)​

Purpose: queue on its own NTFS volume without changing BOWER_QUEUE_PATH. The MSI rewrites the service Environment value on repair and upgrade (installers), so a custom queue path would silently revert to %ProgramData%\Bower\state after an upgrade, leaving the old queue orphaned. Mount the volume at the state folder instead.

Prerequisites: local administrator; the MSI installed; an empty disk.

  1. Stop the collector (and, on a WEC host, nothing else: WEF keeps forwarding into ForwardedEvents, which Bower reads after the restart).

    Stop-Service Bower
  2. Initialise and format the disk with a temporary drive letter (Q: here).

    Get-Disk | Where-Object PartitionStyle -eq 'RAW' | Format-Table Number, Size
    $disk = 1 # the empty disk
    Initialize-Disk -Number $disk -PartitionStyle GPT
    New-Partition -DiskNumber $disk -UseMaximumSize -DriveLetter Q | Out-Null
    Format-Volume -DriveLetter Q -FileSystem NTFS -NewFileSystemLabel bowerqueue -Confirm:$false
  3. Give the volume root the state folder's rights. An ACL set on the mount point later applies to the reparse point, not to the volume, and the service then cannot create the WAL (in the lab it hung at start until the SCM timed out). Never combine icacls /inheritance:r with /T: it strips inherited entries from existing files and leaves them with an empty ACL.

    icacls Q:\ /grant 'NT SERVICE\Bower:(OI)(CI)M'
    icacls Q:\ /remove:g 'BUILTIN\Users' 'Everyone' 'CREATOR OWNER'
    icacls Q:\ # expect only NT SERVICE\Bower (M), Administrators and SYSTEM (F)

    Move the state folder's contents (all queue.db* files together) onto the volume and mount it at the now empty folder:

    $state = 'C:\ProgramData\Bower\state'
    Get-ChildItem $state -Force | Move-Item -Destination Q:\
    Add-PartitionAccessPath -DriveLetter Q -AccessPath "$state\"
    # Optional, once verified: Remove-PartitionAccessPath -DriveLetter Q -AccessPath 'Q:\'
  4. Set the limits. Other settings are not MSI properties and are rewritten on upgrade; re-apply this step after every MSI upgrade or repair.

    $key = 'HKLM:\SYSTEM\CurrentControlSet\Services\Bower'
    $environment = @((Get-ItemProperty $key).Environment | Where-Object { $_ -notmatch '^BOWER_(QUEUE_MAX_BYTES|QUEUE_RETENTION_HOURS|STORAGE_)' })
    $environment += 'BOWER_QUEUE_MAX_BYTES=3221225472', 'BOWER_QUEUE_RETENTION_HOURS=24',
    'BOWER_STORAGE_WARNING_FREE_BYTES=1073741824', 'BOWER_STORAGE_CRITICAL_FREE_BYTES=268435456'
    Set-ItemProperty $key -Name Environment -Type MultiString -Value $environment
    Start-Service Bower
  5. Size the event logs (section 7) and verify.

    $token = (Get-Content C:\ProgramData\Bower\secrets\ingest-token -Raw).Trim()
    (Invoke-RestMethod http://127.0.0.1:4319/v1/status -Headers @{Authorization = "Bearer $token"}).storage |
    Select-Object state, freeBytes, totalBytes, queueMaximumBytes
    Get-Partition -DiskNumber $disk | Select-Object AccessPaths

    Expected: totalBytes is the dedicated volume (the probe follows the mounted folder), state is normal.

Pass: as for Linux; after a test MSI repair, BOWER_QUEUE_PATH is unchanged and totalBytes still shows the dedicated volume. Fail: totalBytes equals C:.

Rollback: stop the service, copy the queue.db* files from the volume to a temporary folder on C:, remove the access path (Remove-PartitionAccessPath -DriveLetter Q -AccessPath 'C:\ProgramData\Bower\state\'), move the files back into the (now ordinary) state folder, reset their ACL with icacls C:\ProgramData\Bower\state\queue.db* /reset, and start.

6.3 Docker and containers​

  1. Back the queue with a volume on a dedicated filesystem, never the container's writable layer. A bind mount makes the filesystem explicit:

    sudo mkfs.xfs -L bowerqueue /dev/sdb && sudo mkdir -p /srv/bower-queue
    echo 'LABEL=bowerqueue /srv/bower-queue xfs defaults,noatime 0 2' | sudo tee -a /etc/fstab && sudo mount /srv/bower-queue
    sudo chown 1654:1654 /srv/bower-queue && sudo chmod 0700 /srv/bower-queue # the image runs as 1654:1654
    docker run -d --name bower --read-only \
    -v /srv/bower-queue:/var/lib/bower \
    -e BOWER_QUEUE_PATH=/var/lib/bower/queue.db -e BOWER_QUEUE_MAX_BYTES=3221225472 \
    --log-driver json-file --log-opt max-size=20m --log-opt max-file=5 \
    ghcr.io/jusso-dev/bower-collector:VERSION

    Named volumes (docker volume create) live under /var/lib/docker and share that filesystem with images and other containers; give /var/lib/docker its own volume if you use them. --storage-opt size= limits only the writable layer (overlay2 on xfs with pquota), not volumes.

  2. Cap container logs (max-size, max-file above, or in /etc/docker/daemon.json).

  3. The image is chiseled (no shell), so query /v1/status through the published port as in 6.1. Expected: totalBytes is the dedicated filesystem. The image declares VOLUME /var/lib/bower; without -v Docker creates an anonymous volume under /var/lib/docker.

Rollback: stop the container, copy the bind-mounted directory to the previous volume, start with the previous mount.

6.4 Kubernetes (deploy/kubernetes)​

The manifest now sets BOWER_QUEUE_MAX_BYTES=5368709120 (5 GiB) with a 16Gi claim, and an ephemeral-storage request of 64Mi and limit of 512Mi (logs only; the root filesystem is read-only). The claim follows section 3 for a typical ext4 storage class (r = 0.05): D = 1.75 × (5 GiB + 4 MiB) = 8.8 GiB needs V = 10.4 GiB, requested as about 10.7 GiB. The earlier 10Gi claim was too small: at the settlement ceiling it left less free space than the warning watermark. A claim that cannot be expanded keeps 10Gi with BOWER_QUEUE_MAX_BYTES=4294967296 (4 GiB, D = 7.0 GiB, about 8.7 GiB needed).

  1. Size the claim from section 3 (V), and BOWER_QUEUE_MAX_BYTES to match. Use a storage class that supports expansion (allowVolumeExpansion: true).

  2. Keep the collector's ephemeral-storage limit: the kubelet evicts a pod that exceeds it, and evicts pods under node nodefs/imagefs pressure (defaults nodefs.available<10%). The queue on the PVC is not ephemeral and survives eviction and rescheduling (ReadWriteOnce, strategy: Recreate).

  3. Apply and verify. The manifest listens on HTTPS: create the bower-intake-tls secret (and the bower-ingest secret and reviewed scopes) before applying, or the new pod waits in ContainerCreating after Recreate has stopped the old one. See upgrading an existing Kubernetes installation.

    kubectl --context YOUR-CONTEXT -n bower get secret bower-ingest bower-intake-tls
    kubectl --context YOUR-CONTEXT -n bower apply -f deploy/kubernetes/deployment.yaml
    kubectl --context YOUR-CONTEXT -n bower get pvc bower-data
    kubectl --context YOUR-CONTEXT -n bower port-forward deploy/bower-collector 4319:4319 &
    curl -s --cacert ca.pem --connect-to bower-collector.bower.svc:4319:127.0.0.1:4319 \
    -H "Authorization: Bearer $TOKEN" https://bower-collector.bower.svc:4319/v1/status | jq .storage
  4. To grow the queue: patch the claim (resources.requests.storage), wait for FileSystemResizePending to clear, then raise BOWER_QUEUE_MAX_BYTES and roll the deployment.

The readiness probe uses /health, which reports degraded but still answers 200; the pod stays ready while refusing intake so producers get 503 rather than connection errors. Rollback: restore the previous manifest; the PVC keeps the queue.

6.5 AWS-hosted collectors (EC2)​

  1. Add an encrypted gp3 volume of size V (3,000 IOPS and 125 MiB/s baseline are ample: the AWS lab peaked below 1 MB/s at 100/sec) and follow 6.1 or 6.2.

    aws ec2 create-volume --availability-zone ap-southeast-2b --size 8 --volume-type gp3 --encrypted \
    --tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=bower-queue}]'
    aws ec2 attach-volume --volume-id vol-XXXX --instance-id i-XXXX --device /dev/sdf
  2. Publish disk metrics with the CloudWatch agent (/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json):

    {"metrics": {"append_dimensions": {"InstanceId": "${aws:InstanceId}"},
    "metrics_collected": {"disk": {"measurement": ["used_percent", "inodes_free"],
    "resources": ["/var/lib/bower"], "metrics_collection_interval": 60}}}}

    On Windows use "LogicalDisk": {"measurement": ["% Free Space"], "resources": ["*"]}.

  3. Alarm before Bower's own warning so both layers fire in order:

    aws cloudwatch put-metric-alarm --alarm-name bower-queue-disk --namespace CWAgent \
    --metric-name disk_used_percent --dimensions Name=InstanceId,Value=i-XXXX Name=path,Value=/var/lib/bower \
    Name=device,Value=nvme1n1 Name=fstype,Value=xfs \
    --statistic Maximum --period 300 --evaluation-periods 1 --threshold 80 \
    --comparison-operator GreaterThanOrEqualToThreshold --alarm-actions arn:aws:sns:REGION:ACCOUNT:TOPIC
  4. Grow online: aws ec2 modify-volume --size N, then sudo xfs_growfs /var/lib/bower (Linux) or Resize-Partition (Windows). Raise BOWER_QUEUE_MAX_BYTES afterwards.

6.6 Azure VMs​

  1. Attach a managed disk (Premium SSD v2 or Standard SSD; size V) and follow 6.1 or 6.2.

    az vm disk attach -g RG --vm-name VM --name bower-queue --new --size-gb 8 --sku StandardSSD_LRS
  2. Collect disk metrics with the Azure Monitor Agent (data collection rule with performance counters \LogicalDisk(*)\% Free Space on Windows, Logical Disk(*)\% Free Space on Linux) or VM insights, then alert:

    az monitor scheduled-query create -g RG -n bower-queue-disk --scopes WORKSPACE_ID \
    --condition "count 'Disk' > 0" --condition-query Disk="InsightsMetrics | where Namespace == 'LogicalDisk' and Name == 'FreeSpacePercentage' | where Tags has '/var/lib/bower' | where Val < 20" \
    --evaluation-frequency 5m --window-size 5m --severity 2 --action-groups ACTION_GROUP_ID
  3. Grow: az disk update --size-gb N (detach-free for most SKUs), then grow the filesystem and raise BOWER_QUEUE_MAX_BYTES.

6.7 On-premises​

Use a dedicated LUN or partition as in 6.1 or 6.2. Monitor it with the platform's agent at 80% used, and alert on Bower's storage.state through the management heartbeat or /v1/status. Thin-provisioned datastores can fill underneath a volume that still reports free space: monitor the datastore too.


7. WEC ForwardedEvents size and retention​

ForwardedEvents is the buffer while Bower is not reading: collector stopped, queue full, or volume critical. WEF keeps delivering into it, and when it is full the oldest records are overwritten. Bower counts overwritten-before-read records as gaps; those events are lost upstream.

Measured: 64 MiB held 30,709 forwarded Security 4625 events (2,185 bytes each with ContentFormat=Events). The Windows default of 20 MiB holds about 9,500: about 16 minutes at 10/sec, under 2 minutes at 100/sec.

RateBower stall to surviveForwardedEvents maximum size
10/sec1 hour100 MiB
10/sec24 hours2.2 GiB
100/sec1 hour1 GiB
100/sec6 hours5.5 GiB (split across WEC servers instead)
wevtutil sl ForwardedEvents /ms:2361393152 /rt:false # 2.2 GiB, overwrite oldest
wevtutil gl ForwardedEvents # verify maxSize and retention
  • Keep /rt:false (overwrite). /rt:true stops the log when full, and WEF then backs up on the sources; /ab:true archives full logs to files Bower never reads.
  • Put the event log on a volume with room for it: wevtutil sl ForwardedEvents /lfn:D:\Logs\ForwardedEvents.evtx.
  • Size each source's Security log too (the pilot used 1 GiB): with source-initiated subscriptions a source keeps unforwarded events in its own log while WEC is unreachable.
  • Alert on windowsEventLog[].gaps increasing and on state = backpressure for more than a quarter of the stall budget.

8. Recovery runbook​

Never delete or edit queue.db, queue.db-wal or queue.db-shm. Never run VACUUM or SQL against a running queue.

8.1 Warning watermark​

  1. Confirm: storage.state = warning, log 1501.
  2. Find what grew: queueDatabaseBytes vs the volume. If the queue is small, another writer is filling the volume (du -xh --max-depth=2 /var/lib/bower; on Windows Get-ChildItem -Recurse | Sort-Object Length -Descending | Select-Object -First 20).
  3. Remove or move that writer's files, or grow the volume (6.5, 6.6). Intake continues.

8.2 Critical watermark (intake refused)​

  1. Confirm: storage.intakePaused = true, log 1500, /health degraded, sources show retries (syslog) or backpressure (WEC).
  2. Free space or grow the volume. Do not restart the collector: it is not needed, and a restart drops syslog frames held in socket buffers.
  3. Within one sample (5 seconds) state returns to warning or normal (log 1502) and intake resumes by itself. Expected: syslog retries stop increasing, the WEC bookmark moves again.
  4. Check delivery drains: queue.queued + retrying + uploading falls, oldestPendingAgeSeconds falls.

8.3 Queue full (page limit)​

  1. Confirm intake.lastRefusalReason = queue-capacity and inspect queuePages.usedPercent. A large event can be refused below 95% when it needs more pages than remain.
  2. Restore the destination and monitor backlog age, settlement errors and free disk space. A refused page allocation may trigger targeted reclaim of acknowledged rows (log 1506, trigger large-event) and reopen intake. With an archive, refusal-triggered reclaim is disabled; verified rows remain subject to the retention floor unless the disk is critical. A full volume, pinned WAL or insufficient settlement headroom can prevent progress; use the recovery steps below and do not assume a drain deadline without a validated measurement.
  3. If the queue is too small for your outages, raise BOWER_QUEUE_MAX_BYTES (and the volume) at the next maintenance window. That needs a restart; see 8.5 for syslog.

8.4 Collectors from before this change stuck with a full backlog​

Symptom: destination reachable, oldestUndelivered growing, nothing delivered, log 1002 with SqliteException. Upgrade the collector (the upgrade restart drains it; the AWS Windows collector drained 22,516 events within a minute). If you cannot upgrade, raise BOWER_QUEUE_MAX_BYTES by at least 50% and restart (the AWS Linux collector drained 27,374 events in 10 seconds). Read 8.5 first.

8.5 Restarting while a syslog sender is backpressured​

Plain syslog has no application acknowledgement. While Bower holds a frame the kernel keeps accepting the sender's bytes into socket buffers (megabytes with Linux autotuning); a restart resets the connection and those frames are gone, although the sender believes it sent them. On AWS a restart in this state lost 21,494 frames.

  1. Avoid the restart (8.2, 8.3 do not need one).
  2. If you must restart, first stop the sender from sending new data and wait for Bower to read what is buffered, which only happens if the queue has room. If it has none, the buffered frames will be lost: record the sender's sequence or counters before and after, and replay from the sender's source if it can.

8.6 Restart on a full volume​

The candidate retries settlement every 5 seconds; a full-volume restart and recovery have not yet been qualified for the final binaries. Free space, then verify service health, delivery and intake; restart only if needed after considering the syslog loss window in 8.5. On Linux, RequiresMountsFor keeps it from starting if the volume is missing. On Windows, if the volume behind the mounted folder is missing the folder is just an empty directory on C: and the collector would start a new, empty queue there: check that storage.totalBytes is the dedicated volume after every reboot and keep the original disk attached until you have.

8.7 WEC gaps​

gaps increased: events were overwritten in ForwardedEvents before Bower read them. They are lost to Bower. Re-forward from the sources' Security logs if they still hold them (create a temporary subscription with ReadExistingEvents=true scoped to the affected period), then enlarge ForwardedEvents (section 7).

8.8 Testing the procedure (filler)​

On a test collector, with the destination blocked by a host firewall rule scoped to the service, create a bounded filler on the queue volume and remove it afterwards:

sudo fallocate -l 700M /var/lib/bower/filler && sleep 15 &&
curl -s -H "Authorization: Bearer $TOKEN" http://127.0.0.1:4319/v1/status | jq '.storage.state'
sudo rm /var/lib/bower/filler
fsutil file createnew C:\ProgramData\Bower\state\filler 734003200; Start-Sleep 15
Remove-Item C:\ProgramData\Bower\state\filler

9. Troubleshooting​

SymptomLikely causeAction
storage.state = unknown, log 1503The queue directory's volume cannot be measured (permissions, unusual filesystem)Check the service can stat the directory; watermarks are off until it can
totalBytes is the root or C: volumeQueue not on the dedicated volume (mount missing, Windows path not mounted)Fix the mount; on Linux confirm RequiresMountsFor
Refusals with queuePages.usedPercent near 100 and low storage.queueUtilisationPercentNormal: pages bind before the logical capSize by section 3
pressurePurgedEvents keeps growingRetention or the queue is too small for the rateRaise BOWER_QUEUE_MAX_BYTES or lower BOWER_QUEUE_RETENTION_HOURS
Destination up, nothing delivered, log 1002 repeatingOld collector stuck at the page ceiling8.4
queueWalBytes large for a long timeA reader pins the WAL (backup tool, sqlite3 shell, probe)Close the reader; the next write truncates the WAL
Syslog sender shows sent, Bower missing frames after a restartSocket buffer loss (8.5)Avoid restarts while backpressured; qualify sender replay
WEC gaps > 0ForwardedEvents wrapped during a stall8.7, section 7
Service does not start after a volume changeMount missing or ACL wrongsystemctl status bower, findmnt; Windows: the state folder ACL
BOWER_QUEUE_MAX_BYTES lowered and start fails with capacityDatabase already larger than the new ceilingRestore the previous value; queues cannot shrink without migration

10. How we tested this​

Two AWS runs, both preserved. The first run found the drain deadlock, the WAL over budget, intake stuck after a drain and the syslog restart loss, and tested the fixes in place. The final run used one CI build (261f61a, unsigned) on a 1 GiB Linux volume and a 2 GiB Windows volume mounted at the MSI state folder, with sequence-numbered input reconciled in Sentinel by a separate verifier:

ScenarioLinux syslogWindows WEC
ws6 stall, retained acknowledgements fill the page limit (16 MiB cap, 60-min floor)Intake reopened 50 s after the first refusal; no restartReopened 42 s after
ws6 stall, backlog fills the page limit, destination restoredDelivery resumed 6 s after restore, intake 46 s15 s and 72 s
Watermarks: warning, critical, space freedIntake resumed 6 s after freeing1 s
Volume completely full, destination restored while full, restart while fullAnswering 6 s after restart; intake 6 s after freeingAt once; 6 s
Accepted events in Sentinel83,725 of 83,725, 0 duplicates74,415 of 74,415, 1,008 duplicate rows
Lost before acceptance7,109 frames in socket buffers at the restart while full0 (bookmark)

Health records written while intake was refused arrived after restore with storage-critical, intakePaused and backpressure, and the queue-storage alert fired for both collectors. Payloads averaged 1,208 and 1,369 bytes; ForwardedEvents at 64 MiB held 30,705 forwarded events. Not run within the time box: a ForwardedEvents wrap loss, RELP under the watermark live, UDP and long soaks.

Unit and integration tests (StoragePressureTests, plus additions to SyslogCollectorTests, WindowsEventLogCollectorTests and HeartbeatV2CollectorTests) cover watermark thresholds and hysteresis, sampling bounds, configuration validation and hashes, refusal without commit and with exact replay still answered as duplicate, settlement during critical, pressure and refusal-triggered retention, the drain-deadlock regression, WAL self-healing, page-limit refusal with crash/recovery, restart while critical, syslog frames held without dropping or closing the connection, WEC bookmarks held, status, health and heartbeat fields.

Lab coverage of later changes. The AWS runs used build 261f61a. Review fixes made afterwards were covered by unit and local tests only, not re-run on AWS: the Rust settlement ceiling and 85% intake limit, the page-based free-page release reserve, the health-record byte reserve and the bounded large-event purge (3d73c90, 547d175).


Known limits​

  • Plain TCP and TLS syslog: socket-buffer loss on restart or reset while backpressured (measured above). These transports have no application acknowledgement, so frames the kernel accepted but Bower had not queued are lost; qualify each sender's retention and replay. RELP (from #168) is different: Bower acknowledges a message only after it is durably queued, withholds the acknowledgement while intake is refused, and closes the session after 30 seconds of refusals (relpBusyCloses); the sender keeps and resends every unacknowledged message, so nothing is lost but resends can arrive as duplicates. RELP under the watermark is unit-tested only. UDP is best effort.
  • Real disk-full is qualified on AWS volumes (xfs and NTFS filled to 20 KB and 0 bytes free, with a restart while full); unit tests cover the page-limit and restart-while-critical paths but cannot produce ENOSPC portably.
  • Existing queue files created without SQLite incremental auto-vacuum do not return freed pages to the filesystem through the new incremental-vacuum path. Freed pages can be reused inside the database, but that does not recover volume free space. An offline VACUUM (as in the rollback procedure) returns it but does not switch the file to incremental auto-vacuum; use additional disk capacity until a migration is qualified.
  • Rust collector (retired in November 2026; recorded as tested): the intake limit and settlement ceiling are ported (tests open a queue grown past the intake ceiling, refuse intake at the limit while a backlog still leases and acknowledges, and refuse a file past the settlement ceiling). Watermarks, pressure retention, the large-event reclaim, incremental vacuum and the storage status are not ported: Rust has no free-space watermark, and only age retention frees pages. A C# to Rust switch with a grown queue was not run live. A Rust collector refuses new events about 15% earlier in practice: it stops at the same 85% intake limit, but without pressure retention the acknowledged rows kept for duplicate detection (BOWER_QUEUE_RETENTION_HOURS) hold that room until the hourly age purge removes them. Size Rust queues for the backlog plus the retained acknowledgements, or lower the retention.
  • Rollback to builds without the settlement ceiling (C# before this change, Rust before the port): they refuse to open a database larger than M + 4 MiB. Follow the downgrade procedure (drain, purge, VACUUM) before rolling back. Raising BOWER_QUEUE_MAX_BYTES on the older build is not a qualified recovery and changes the storage budget.
  • Management accepts the heartbeat storage object but does not store or display it yet (workstream 6).
  • Free space is sampled once a second on the insert path and every 5 seconds by the monitor; a writer that fills the volume faster than that can still cause storage-full refusals (handled as backpressure).
  • Large events below the pressure trigger: a large event can need more pages than remain below the intake limit, so it is refused well below 80%. The refusal now triggers retention, which removes the oldest acknowledged rows until the event fits (at any age, checked in batches of 50), so the rest keep detecting duplicates; unacknowledged rows are never touched. An event larger than the whole intake limit is refused permanently. Not for archived queues (next item): there the source retries until age or pressure retention frees enough verified rows. Unit-tested only.
  • Archive-backed retention: retention now deletes only the sequence set whose archive segments it re-hashed, and rechecks acknowledgement and cutoff in the write transaction. Interleaved purge and corrupted-segment regressions cover the previously shifted-batch failure. Segment verification and queue commit are separate operations: a segment can be changed or replaced after its hash is checked but before its local row is deleted. The archive writer lock coordinates cooperating writers; it does not prevent an external writer or storage mutation. Protect the archive directory against concurrent modification. Refusal-triggered reclamation remains blocked for archived queues until an immutable or version-retained archive guarantees the verified copy survives the queue commit, and adversarial mutation and crash recovery are tested. Until then retain and retry the source event where the source supports it, and restore capacity without deleting recent acknowledgements or unverified rows. UDP can drop events under backpressure.
  • Pressure and refusal-triggered retention shorten duplicate detection. Replays of events whose acknowledged rows were removed are accepted again and duplicated at the destination. Production queries should deduplicate by CollectorId, EventId as in the pilot guide.
  • The settlement ceiling raises the worst-case database size from M + 4 MiB to 1.75 × (M + 4 MiB); volumes sized for the old figure may reach the critical watermark while a large backlog is acknowledged (intake pauses; nothing is lost).
  • Power loss, filesystem corruption and backup/restore are not covered here.