Suppression, selection metrics, source silence and the archive
Four collector capabilities for running Bower at volume and proving coverage. All are off unless configured and keep the queue at schema version 5 (new tables are additive).
Suppression
A policy can cap a noisy event type per key without losing the count:
decision:
action: accept
minimumValueScore: 50
neverSample: false
suppression:
keyFields: [source.ipAddress] # 1–5 string paths or labels.<key>
windowSeconds: 60 # 1–86400, windows aligned to the Unix epoch
allowPerWindow: 20 # 1–100000 (0 or omitted for action: aggregate)
- The first
allowPerWindowevents per key in each window are queued as usual. Later ones are counted in the queue database inside the same transaction as the duplicate check, and the collector answers"suppressed": true(HTTP 202). Duplicates never consume allowance. Counts survive restarts. When the queue database is at its physical storage limit a suppressible event is refused with the same capacity backpressure as any other event and is not counted. - Counts are of accepted attempts. A suppressed event is not stored, so the duplicate check cannot recognise it: a producer that retries an event after it was counted (for example after a lost response) is counted again. Stored events still never duplicate.
- Within 5 seconds of a window ending, one
telemetry_aggregation_summaryevent (actiontelemetry.suppressed) is queued with the suppressed and admitted counts, the key fields and their redacted values, the window bounds and the first and last suppressed times. Its id derives from the window, so it is never duplicated. - The summary goes through policy like any event. A bundle that suppresses must also
accept
telemetry_aggregation_summary, or it does not load; if the summary is later refused (for example its policy was changed to require a field it lacks), windows stay open and the collector logs the failure instead of dropping counts. The same applies when the queue is at capacity: closing is retried every 5 seconds with a warning (EventId 1302) until there is room. Alert on that warning. Example policies:examples/policies/suppression. neverSamplepolicies, non-accepting policies and the summary type itself cannot suppress. Choose key fields that keep distinct actors in distinct windows: a key ofeventTypealone lets one noisy client hide another.bower policy testandbower policy diffreport the policy decision only; suppression is stateful and applies when an event is queued (the bundle comparison lists it undernotExercisedOffline). The Azure Functions collector has no durable queue and never suppresses.
Aggregate: count only
For noisy, low-value event types where individual events aren't needed, set the decision
action to aggregate. Every matching event is counted per key and window exactly as
above, and none is stored: the collector answers "suppressed": true (HTTP 202) and
queues one telemetry_aggregation_summary per key and window with admittedCount 0.
decision:
action: aggregate
minimumValueScore: 50
neverSample: false
suppression:
keyFields: [application.name, eventType]
windowSeconds: 300 # allowPerWindow is omitted (or 0)
The same rules apply as for suppression: a suppression section is required (an
aggregate policy without one fails to load), neverSample policies can't aggregate,
the bundle must accept telemetry_aggregation_summary, and counts survive restarts.
Choose keys that keep distinct actors apart (for example include actor.userId or
source.ipAddress) if the counts will be used to investigate anyone: a count per
application alone hides who did what.
Population diff
Run every sample through two policy directories before rollout:
bower policy diff --baseline policies/current --candidate policies/proposed \
samples/*.json --fail-on-change
Output lists each event whose decision or deciding policy changes, with both bundle
hashes. Metadata only; exit 3 on any change when --fail-on-change is set. Attach the
output to the policy change, as AGENTS.md requires for policy content changes.
Comparing complete bundles
A directory comparison cannot see pack, privacy or detection changes. Describe each
side as the full bundle the collector would load, with the same options as
bower policy test, prefixed --baseline- or --candidate-:
bower policy diff \
--baseline policies/current \
--baseline-pack aws-security-1.0.0.bowerpack --baseline-trusted-key pack-signing.pub.pem \
--candidate policies/current \
--candidate-pack aws-security-1.1.0.bowerpack --candidate-trusted-key pack-signing.pub.pem \
--candidate-privacy-profile privacy/proposed.yaml \
--at 2026-10-17T01:00:00Z samples/*.json --fail-on-change
| Option (per side) | Meaning |
|---|---|
--baseline DIR / --candidate DIR | Policy directory. Optional when the side has a pack; when given it must exist, even alongside packs. |
--{side}-pack FILE | Signed .bowerpack (repeatable). |
--{side}-trusted-key PUB | PEM public key trusted to sign that side's packs (repeatable). |
--{side}-privacy-profile FILE | Privacy profile (at most one per side, including pack profiles). |
--{side}-hmac-key-file FILE, --{side}-hmac-key-id ID | HMAC key for hmac actions (id defaults to k1). |
--at ISO | Evaluate every sample at this time, so results are reproducible. Defaults to now, taken once. |
--at (like archive replay --from and --to) takes only
YYYY-MM-DDTHH:MM[:SS[.FFFFFFF]] followed by Z or +HH:MM/-HH:MM (at most 14:00).
Date-only and time-only values, missing offsets, other date formats and more than 7
fractional digits are refused with exit 1, so neither CLI rounds, truncates or fills in
today's date.
Any --baseline-* or --candidate-* bundle option, or --at, selects this mode.
Without them the command is the directory comparison above, with unchanged output and
exit codes.
Both bundles load before any sample is evaluated. An unsigned pack, a pack signed by a
key not given with --{side}-trusted-key, an invalid profile or a missing file stops
the command with exit 1 and nothing on stdout. Each sample then runs alone through the
offline intake (redaction, schema and validation, policy, privacy alerts, pack
detections) at the fixed time. Nothing is queued, sent or deployed and no network call
is made.
The output keeps the directory-mode fields (baselineBundleHash, candidateBundleHash,
events, changed, changes[].file, eventId, baseline.decision,
baseline.policyId) and adds, per changed sample:
| Field | Content |
|---|---|
accepted | Decision is accept or redactandaccept. |
failureStage, failureReasons | redaction, schema, validation or policy, and the reasons, when not accepted. |
privacyActions | {detector, action}: the first action per detector, as the event's privacy metadata records it (for example secret.aws / Removed). Never values. |
privacyFindings | {detector, path, action} per finding, sorted and deduplicated; detector includes a sub-kind as id:subKind, and every field rule is field-rule. Shows a second field rule or a changed action on a later finding that privacyActions hides. Never values. |
removedCount, maskedCount | Number of distinct field paths removed, and masked, hashed, encrypted, replaced or truncated. |
privacyAlert | Whether a privacy alert event would be raised. |
detections | {ruleId, ruleVersion, ruleHash} per pack detection rule that matches an accepted event, so a rule content change is visible. Never matched values. |
changedFields | Which of the fields above differ. |
Top-level evaluatedAt, baselineBundle and candidateBundle (bundle hash, policy
count, pack id, version and hash, privacy profile id, version and hash, detection rule
count) record what was compared. notExercisedOffline lists stateful behaviour one
offline request per sample cannot show: suppression windows, source silence,
cross-event deduplication, queue capacity and backpressure, delivery and destination
validation, pack parsers (samples are already semantic events) and management privacy
assignments. A privacy-only change, such as masking instead of removing a detector's
match, is reported even when acceptance stays the same.
Exit codes: 0 when the comparison completes, 3 when --fail-on-change is set and any
sample has a changed field, and 1 for invalid arguments (an --at outside the format
above, a side with neither a directory nor a pack, a named directory that does not
exist), bundle load or trust failures, or an unreadable sample. The output JSON for
given inputs and --at is pinned by tests/fixtures/bundle-diff.
Selection metrics
GET /v1/status (token protected) includes pipeline: events and bytes received,
queued, duplicated and suppressed since start, per stage (transform, redaction,
schema, validation, policy), policy id and decision. transform counts signed
pack transform failures, quarantined before redaction. No payload content. Queued bytes
include Bower's decision context.
bower top renders these rows as per-stage rates (its total row equals eventsIn),
with the activity field (busiest 50 sources by accepted event count) for per-source
rates. bower tap shows sampled, post-redaction outcomes whose stage and action
match these rows. See tap and top.
Intake metrics
GET /v1/status also includes intake, so a full queue is distinguishable from an
idle collector. The base fields are:
"intake": {"observedSince": "2026-10-26T01:00:00+00:00",
"attemptedRequests": 42, "attemptedBytes": 51234,
"capacityRefusals": 5, "capacityRefusedBytes": 6100,
"privacyAlertRefusals": 1, "privacyAlertRefusedBytes": 1300,
"lastRefusalAt": "2026-10-26T01:20:31.5+00:00", "lastRefusalReason": "queue-capacity"}
- Boundary: one attempt per candidate submitted to processing on
/v1/eventsor a signed pack parser endpoint, after authentication, producer authorisation and (for packs) parsing. Rate-limited, unauthorised, oversized or unparseable requests are not attempts. Bytes are the UTF-8 candidate JSON, as forpipeline.bytesIn. - Both refusal kinds return HTTP 503 (
Collector queue is at capacity; retry later.). For direct HTTP intake they differ in what was committed:capacityRefusals(reasonqueue-capacity): the queue refused the event itself, whichever admission branch refused it. Nothing was committed for the attempt: no queue row, nopipelinecount, no source activity, no evidence.privacyAlertRefusals(reasonprivacy-alert-capacity): the event was decided first and its outcome is already inpipeline(and, if accepted, its row is queued and its source marked seen). Only the privacy alert for a high-risk detection was refused. This applies to accepted, quarantined (invalid) and rejected events.
- A retry is a new attempt. After a
queue-capacityrefusal it counts inpipelineonce admitted. After aprivacy-alert-capacityrefusal of an accepted event, the retry is reported as a duplicate (HTTP 200) and emits the alert; alert ids are stable, so retries never create a second alert. - For a WEC source using the collector's shared SQLite queue and cursor, the entire
refused record is rolled back, including a main event whose generated privacy alert
could not be queued. The additional
sourceRecordsRolledBackandsourceRecordRolledBackBytesfields count affected source records and their candidate bytes for capacity refusals; they are a subset of the primary refusal counts, not additional attempts. Source retries are not HTTP responses. lastRefusalReasonis one of the two bounded codes above, matchinglastRefusalAt(the latest refusal); both arenullbefore any refusal.- Counters are process lifetime: they start at zero at
observedSince(collector start) and reset only on restart. There is no reset endpoint. Updates are lock free and safe under concurrent intake: the time and reason are written before a refusal is counted, so a snapshot never shows a refusal without them. Metadata only: no payloads, event ids, credentials or request text.
Source silence
Set BOWER_SOURCE_EXPECTATIONS to a JSON file
(schema,
example):
{"apiVersion": "bower.security/v1", "kind": "SourceExpectations",
"sources": [{"application": "FinanceApi", "environment": "production", "maximumSilenceMinutes": 15}]}
- A source is seen when an event from it is queued, deduplicated or suppressed. Rejected and quarantined events do not count: they never reach the SIEM.
/healthbecomesdegradedwhile any expected source is overdue./v1/statuslists each source asactive(seen within its allowance),pending(not seen yet, still inside its first allowance),silentornever-seen(overdue). Heartbeats reportactivesources ashealthyand every other state by name, so the console counts pending and quiet sources as not healthy. A pending source is never reported as healthy: there is no evidence from it yet.- A source never seen gets its allowance from when the collector first began
expecting it. That monitoring start is stored per source in the queue database
(
source_monitoring), so restarting the collector, however often, cannot extend the deadline. A source added to the expectations file gets its own allowance from the next start; existing sources keep theirs. A source removed from the file is forgotten at the next start, and listing it again later starts a fresh allowance. Starting withoutBOWER_SOURCE_EXPECTATIONSforgets every monitoring start. Each start that forgets sources logs a warning (EventId 1321) with the count and theapplication/environmentids, so a reset is auditable. - A monitoring start recorded while the host clock ran ahead (for example before time synchronisation) is replaced by the earlier time at the next start. A start only ever moves earlier, which can shorten a deadline but never extend it.
- Last-seen times are saved every minute; after a crash up to a minute of activity is lost, which can only make a source look quieter, never healthier.
- Offline:
bower sources check --expectations sources.json --database queue.dbexits 3 when any source is overdue. It reads, never writes, the collector's monitoring starts, so a source the collector reports aspendingispendinghere too. A source the collector has never monitored gets no grace and, never seen, is reported asnever-seenat once.
Archive and replay
Set BOWER_ARCHIVE_PATH to keep every stored event beyond queue retention, for
example for a records-authority period such as AFDA Express v2.
-
Every minute the collector writes
segment-<first sequence>.jsonl.gzfiles of up to 10,000 events, each with a manifest holding the file digest, sequence range and the queue ledger chain hash at both ends. A segment is committed once its manifest is published. Files are written under unique temporary names, flushed and moved into place without replacing an existing file, and the directory is synced after each move; a segment left without a manifest by a crash is an uncommitted orphan and is rewritten. -
A writer extends only its own queue's chain: if the queue ledger hash at the archive head differs from the manifest (another collector's archive, a queue restored from an older backup) or the queue has not reached that sequence, the run fails with
lineage-mismatchand writes nothing. -
Every writer (the collector job and
bower archive create, in either runtime) holds an exclusive OS lock onarchive.lockin the archive directory for its whole run, taken before it reads the archive head. A second writer fails at once with an I/O error and writes nothing. Orphan segments and abandonedsegment-*.tmpfiles (up to 1,000 per run) are removed only by the lock owner. The lock isflockon Unix, so keep the archive on a local or lock-capable file system. Where the lock is not enforced (an unsupported file system, orDOTNET_SYSTEM_IO_DISABLEFILELOCKINGset) the writer refuses to run rather than write unlocked. -
The archive continues the queue ledger chain, so it verifies without the queue:
bower archive verify --archive /mnt/bower-archive --anchor 48213:<ledger head hash>Use a ledger head witnessed by the management console as the anchor. Without one, verification proves the archive is internally consistent from its first segment; only an anchor proves nothing was cut from the start or rewritten wholesale.
-
While an archive is configured, retention purges acknowledged events only up to verified archive coverage of this queue. Before each run it checks that every segment exists and matches its manifest digest and ledger chain, that the archive starts at or below the oldest event still stored, and that the archived chain hash equals the queue ledger hash at the highest boundary both record (so an archive written by another queue, even with overlapping sequence numbers, is refused). Otherwise retention purges nothing and logs event 1202 with a reason code (
segment-missing,segment-digest-mismatch,archive-starts-above-retained,lineage-mismatch,manifest-invalidand so on) and sequence numbers only. Undelivered, retrying and dead-lettered events are never purged. Segments already verified are not re-read while their manifest, size and write time are unchanged; any change triggers full verification. Before each purge batch, the segments holding the events about to be removed are re-hashed regardless, so silent corruption or a timestamp-preserving restore stops the purge. An archive started on an existing queue begins at the oldest event still stored. -
Bower never deletes or rewrites committed archive segments. Put the directory on write-once storage and manage retention there.
-
bower archive replay --archive DIR --collector-url URL [--from ISO] [--to ISO] [--event-type TYPE]... [--collector-id ID] [--dry-run]verifies the archive, then posts each selected event through a collector's full intake, labelledbowerReplay: archiveandbowerReplayCollectorId(the archived envelope's collector), withBOWER_INGEST_TOKEN. Replay into a collector with its own queue (the original queue reports replayed events as duplicates) and the originalBOWER_COLLECTOR_ID, so Sentinel keeps one(CollectorId, EventId)identity per event (replay-safe detections). This holds only for archived envelopes with a stampedcollector.id. An envelope without one is replayed without an origin label and accepted by any collector, which stamps its own id, so its(CollectorId, EventId)is not the original identity: treat such events as a separate migration path. The origin label is always the archived envelope's stampedcollector.id(a label the producer set is replaced). A collector with another id quarantines the event asarchive-replay-collector-mismatch; the CLI counts it ascollector-mismatch, continues and exits 2.--collector-id IDreplays only that collector's events, so an archive that spans an id change is replayed one slice per matching collector. Events older than five years failtimeGeneratedvalidation.--fromand--totake the same strict format aspolicy diff --at(YYYY-MM-DDTHH:MM[:SS[.FFFFFFF]]withZor an offset); other forms, including date-only values, exit 1. -
bower archive create --archive DIR --database queue.dbarchives a queue by hand. While a collector is writing the same archive it fails on the writer lock instead of racing it; rerun it later.