Skip to main content

Suppression, selection metrics, source silence and the archive

Four collector capabilities for running Bower at volume and proving coverage. All are off unless configured and keep the queue at schema version 5 (new tables are additive).

Suppression​

A policy can cap a noisy event type per key without losing the count:

decision:
action: accept
minimumValueScore: 50
neverSample: false
suppression:
keyFields: [source.ipAddress] # 1–5 string paths or labels.<key>
windowSeconds: 60 # 1–86400, windows aligned to the Unix epoch
allowPerWindow: 20 # 1–100000 (0 or omitted for action: aggregate)
  • The first allowPerWindow events per key in each window are queued as usual. Later ones are counted in the queue database inside the same transaction as the duplicate check, and the collector answers "suppressed": true (HTTP 202). Duplicates never consume allowance. Counts survive restarts. When the queue database is at its physical storage limit a suppressible event is refused with the same capacity backpressure as any other event and is not counted.
  • Counts are of accepted attempts. A suppressed event is not stored, so the duplicate check cannot recognise it: a producer that retries an event after it was counted (for example after a lost response) is counted again. Stored events still never duplicate.
  • Within 5 seconds of a window ending, one telemetry_aggregation_summary event (action telemetry.suppressed) is queued with the suppressed and admitted counts, the key fields and their redacted values, the window bounds and the first and last suppressed times. Its id derives from the window, so it is never duplicated.
  • The summary goes through policy like any event. A bundle that suppresses must also accept telemetry_aggregation_summary, or it does not load; if the summary is later refused (for example its policy was changed to require a field it lacks), windows stay open and the collector logs the failure instead of dropping counts. The same applies when the queue is at capacity: closing is retried every 5 seconds with a warning (EventId 1302) until there is room. Alert on that warning. Example policies: examples/policies/suppression.
  • neverSample policies, non-accepting policies and the summary type itself cannot suppress. Choose key fields that keep distinct actors in distinct windows: a key of eventType alone lets one noisy client hide another.
  • bower policy test and bower policy diff report the policy decision only; suppression is stateful and applies when an event is queued (the bundle comparison lists it under notExercisedOffline). The Azure Functions collector has no durable queue and never suppresses.

Aggregate: count only​

For noisy, low-value event types where individual events aren't needed, set the decision action to aggregate. Every matching event is counted per key and window exactly as above, and none is stored: the collector answers "suppressed": true (HTTP 202) and queues one telemetry_aggregation_summary per key and window with admittedCount 0.

decision:
action: aggregate
minimumValueScore: 50
neverSample: false
suppression:
keyFields: [application.name, eventType]
windowSeconds: 300 # allowPerWindow is omitted (or 0)

The same rules apply as for suppression: a suppression section is required (an aggregate policy without one fails to load), neverSample policies can't aggregate, the bundle must accept telemetry_aggregation_summary, and counts survive restarts. Choose keys that keep distinct actors apart (for example include actor.userId or source.ipAddress) if the counts will be used to investigate anyone: a count per application alone hides who did what.

Population diff​

Run every sample through two policy directories before rollout:

bower policy diff --baseline policies/current --candidate policies/proposed \
samples/*.json --fail-on-change

Output lists each event whose decision or deciding policy changes, with both bundle hashes. Metadata only; exit 3 on any change when --fail-on-change is set. Attach the output to the policy change, as AGENTS.md requires for policy content changes.

Comparing complete bundles​

A directory comparison cannot see pack, privacy or detection changes. Describe each side as the full bundle the collector would load, with the same options as bower policy test, prefixed --baseline- or --candidate-:

bower policy diff \
--baseline policies/current \
--baseline-pack aws-security-1.0.0.bowerpack --baseline-trusted-key pack-signing.pub.pem \
--candidate policies/current \
--candidate-pack aws-security-1.1.0.bowerpack --candidate-trusted-key pack-signing.pub.pem \
--candidate-privacy-profile privacy/proposed.yaml \
--at 2026-10-17T01:00:00Z samples/*.json --fail-on-change
Option (per side)Meaning
--baseline DIR / --candidate DIRPolicy directory. Optional when the side has a pack; when given it must exist, even alongside packs.
--{side}-pack FILESigned .bowerpack (repeatable).
--{side}-trusted-key PUBPEM public key trusted to sign that side's packs (repeatable).
--{side}-privacy-profile FILEPrivacy profile (at most one per side, including pack profiles).
--{side}-hmac-key-file FILE, --{side}-hmac-key-id IDHMAC key for hmac actions (id defaults to k1).
--at ISOEvaluate every sample at this time, so results are reproducible. Defaults to now, taken once.

--at (like archive replay --from and --to) takes only YYYY-MM-DDTHH:MM[:SS[.FFFFFFF]] followed by Z or +HH:MM/-HH:MM (at most 14:00). Date-only and time-only values, missing offsets, other date formats and more than 7 fractional digits are refused with exit 1, so neither CLI rounds, truncates or fills in today's date.

Any --baseline-* or --candidate-* bundle option, or --at, selects this mode. Without them the command is the directory comparison above, with unchanged output and exit codes.

Both bundles load before any sample is evaluated. An unsigned pack, a pack signed by a key not given with --{side}-trusted-key, an invalid profile or a missing file stops the command with exit 1 and nothing on stdout. Each sample then runs alone through the offline intake (redaction, schema and validation, policy, privacy alerts, pack detections) at the fixed time. Nothing is queued, sent or deployed and no network call is made.

The output keeps the directory-mode fields (baselineBundleHash, candidateBundleHash, events, changed, changes[].file, eventId, baseline.decision, baseline.policyId) and adds, per changed sample:

FieldContent
acceptedDecision is accept or redactandaccept.
failureStage, failureReasonsredaction, schema, validation or policy, and the reasons, when not accepted.
privacyActions{detector, action}: the first action per detector, as the event's privacy metadata records it (for example secret.aws / Removed). Never values.
privacyFindings{detector, path, action} per finding, sorted and deduplicated; detector includes a sub-kind as id:subKind, and every field rule is field-rule. Shows a second field rule or a changed action on a later finding that privacyActions hides. Never values.
removedCount, maskedCountNumber of distinct field paths removed, and masked, hashed, encrypted, replaced or truncated.
privacyAlertWhether a privacy alert event would be raised.
detections{ruleId, ruleVersion, ruleHash} per pack detection rule that matches an accepted event, so a rule content change is visible. Never matched values.
changedFieldsWhich of the fields above differ.

Top-level evaluatedAt, baselineBundle and candidateBundle (bundle hash, policy count, pack id, version and hash, privacy profile id, version and hash, detection rule count) record what was compared. notExercisedOffline lists stateful behaviour one offline request per sample cannot show: suppression windows, source silence, cross-event deduplication, queue capacity and backpressure, delivery and destination validation, pack parsers (samples are already semantic events) and management privacy assignments. A privacy-only change, such as masking instead of removing a detector's match, is reported even when acceptance stays the same.

Exit codes: 0 when the comparison completes, 3 when --fail-on-change is set and any sample has a changed field, and 1 for invalid arguments (an --at outside the format above, a side with neither a directory nor a pack, a named directory that does not exist), bundle load or trust failures, or an unreadable sample. The output JSON for given inputs and --at is pinned by tests/fixtures/bundle-diff.

Selection metrics​

GET /v1/status (token protected) includes pipeline: events and bytes received, queued, duplicated and suppressed since start, per stage (transform, redaction, schema, validation, policy), policy id and decision. transform counts signed pack transform failures, quarantined before redaction. No payload content. Queued bytes include Bower's decision context.

bower top renders these rows as per-stage rates (its total row equals eventsIn), with the activity field (busiest 50 sources by accepted event count) for per-source rates. bower tap shows sampled, post-redaction outcomes whose stage and action match these rows. See tap and top.

Intake metrics​

GET /v1/status also includes intake, so a full queue is distinguishable from an idle collector. The base fields are:

"intake": {"observedSince": "2026-10-26T01:00:00+00:00",
"attemptedRequests": 42, "attemptedBytes": 51234,
"capacityRefusals": 5, "capacityRefusedBytes": 6100,
"privacyAlertRefusals": 1, "privacyAlertRefusedBytes": 1300,
"lastRefusalAt": "2026-10-26T01:20:31.5+00:00", "lastRefusalReason": "queue-capacity"}
  • Boundary: one attempt per candidate submitted to processing on /v1/events or a signed pack parser endpoint, after authentication, producer authorisation and (for packs) parsing. Rate-limited, unauthorised, oversized or unparseable requests are not attempts. Bytes are the UTF-8 candidate JSON, as for pipeline.bytesIn.
  • Both refusal kinds return HTTP 503 (Collector queue is at capacity; retry later.). For direct HTTP intake they differ in what was committed:
    • capacityRefusals (reason queue-capacity): the queue refused the event itself, whichever admission branch refused it. Nothing was committed for the attempt: no queue row, no pipeline count, no source activity, no evidence.
    • privacyAlertRefusals (reason privacy-alert-capacity): the event was decided first and its outcome is already in pipeline (and, if accepted, its row is queued and its source marked seen). Only the privacy alert for a high-risk detection was refused. This applies to accepted, quarantined (invalid) and rejected events.
  • A retry is a new attempt. After a queue-capacity refusal it counts in pipeline once admitted. After a privacy-alert-capacity refusal of an accepted event, the retry is reported as a duplicate (HTTP 200) and emits the alert; alert ids are stable, so retries never create a second alert.
  • For a WEC source using the collector's shared SQLite queue and cursor, the entire refused record is rolled back, including a main event whose generated privacy alert could not be queued. The additional sourceRecordsRolledBack and sourceRecordRolledBackBytes fields count affected source records and their candidate bytes for capacity refusals; they are a subset of the primary refusal counts, not additional attempts. Source retries are not HTTP responses.
  • lastRefusalReason is one of the two bounded codes above, matching lastRefusalAt (the latest refusal); both are null before any refusal.
  • Counters are process lifetime: they start at zero at observedSince (collector start) and reset only on restart. There is no reset endpoint. Updates are lock free and safe under concurrent intake: the time and reason are written before a refusal is counted, so a snapshot never shows a refusal without them. Metadata only: no payloads, event ids, credentials or request text.

Source silence​

Set BOWER_SOURCE_EXPECTATIONS to a JSON file (schema, example):

{"apiVersion": "bower.security/v1", "kind": "SourceExpectations",
"sources": [{"application": "FinanceApi", "environment": "production", "maximumSilenceMinutes": 15}]}
  • A source is seen when an event from it is queued, deduplicated or suppressed. Rejected and quarantined events do not count: they never reach the SIEM.
  • /health becomes degraded while any expected source is overdue. /v1/status lists each source as active (seen within its allowance), pending (not seen yet, still inside its first allowance), silent or never-seen (overdue). Heartbeats report active sources as healthy and every other state by name, so the console counts pending and quiet sources as not healthy. A pending source is never reported as healthy: there is no evidence from it yet.
  • A source never seen gets its allowance from when the collector first began expecting it. That monitoring start is stored per source in the queue database (source_monitoring), so restarting the collector, however often, cannot extend the deadline. A source added to the expectations file gets its own allowance from the next start; existing sources keep theirs. A source removed from the file is forgotten at the next start, and listing it again later starts a fresh allowance. Starting without BOWER_SOURCE_EXPECTATIONS forgets every monitoring start. Each start that forgets sources logs a warning (EventId 1321) with the count and the application/environment ids, so a reset is auditable.
  • A monitoring start recorded while the host clock ran ahead (for example before time synchronisation) is replaced by the earlier time at the next start. A start only ever moves earlier, which can shorten a deadline but never extend it.
  • Last-seen times are saved every minute; after a crash up to a minute of activity is lost, which can only make a source look quieter, never healthier.
  • Offline: bower sources check --expectations sources.json --database queue.db exits 3 when any source is overdue. It reads, never writes, the collector's monitoring starts, so a source the collector reports as pending is pending here too. A source the collector has never monitored gets no grace and, never seen, is reported as never-seen at once.

Archive and replay​

Set BOWER_ARCHIVE_PATH to keep every stored event beyond queue retention, for example for a records-authority period such as AFDA Express v2.

  • Every minute the collector writes segment-<first sequence>.jsonl.gz files of up to 10,000 events, each with a manifest holding the file digest, sequence range and the queue ledger chain hash at both ends. A segment is committed once its manifest is published. Files are written under unique temporary names, flushed and moved into place without replacing an existing file, and the directory is synced after each move; a segment left without a manifest by a crash is an uncommitted orphan and is rewritten.

  • A writer extends only its own queue's chain: if the queue ledger hash at the archive head differs from the manifest (another collector's archive, a queue restored from an older backup) or the queue has not reached that sequence, the run fails with lineage-mismatch and writes nothing.

  • Every writer (the collector job and bower archive create, in either runtime) holds an exclusive OS lock on archive.lock in the archive directory for its whole run, taken before it reads the archive head. A second writer fails at once with an I/O error and writes nothing. Orphan segments and abandoned segment-*.tmp files (up to 1,000 per run) are removed only by the lock owner. The lock is flock on Unix, so keep the archive on a local or lock-capable file system. Where the lock is not enforced (an unsupported file system, or DOTNET_SYSTEM_IO_DISABLEFILELOCKING set) the writer refuses to run rather than write unlocked.

  • The archive continues the queue ledger chain, so it verifies without the queue:

    bower archive verify --archive /mnt/bower-archive --anchor 48213:<ledger head hash>

    Use a ledger head witnessed by the management console as the anchor. Without one, verification proves the archive is internally consistent from its first segment; only an anchor proves nothing was cut from the start or rewritten wholesale.

  • While an archive is configured, retention purges acknowledged events only up to verified archive coverage of this queue. Before each run it checks that every segment exists and matches its manifest digest and ledger chain, that the archive starts at or below the oldest event still stored, and that the archived chain hash equals the queue ledger hash at the highest boundary both record (so an archive written by another queue, even with overlapping sequence numbers, is refused). Otherwise retention purges nothing and logs event 1202 with a reason code (segment-missing, segment-digest-mismatch, archive-starts-above-retained, lineage-mismatch, manifest-invalid and so on) and sequence numbers only. Undelivered, retrying and dead-lettered events are never purged. Segments already verified are not re-read while their manifest, size and write time are unchanged; any change triggers full verification. Before each purge batch, the segments holding the events about to be removed are re-hashed regardless, so silent corruption or a timestamp-preserving restore stops the purge. An archive started on an existing queue begins at the oldest event still stored.

  • Bower never deletes or rewrites committed archive segments. Put the directory on write-once storage and manage retention there.

  • bower archive replay --archive DIR --collector-url URL [--from ISO] [--to ISO] [--event-type TYPE]... [--collector-id ID] [--dry-run] verifies the archive, then posts each selected event through a collector's full intake, labelled bowerReplay: archive and bowerReplayCollectorId (the archived envelope's collector), with BOWER_INGEST_TOKEN. Replay into a collector with its own queue (the original queue reports replayed events as duplicates) and the original BOWER_COLLECTOR_ID, so Sentinel keeps one (CollectorId, EventId) identity per event (replay-safe detections). This holds only for archived envelopes with a stamped collector.id. An envelope without one is replayed without an origin label and accepted by any collector, which stamps its own id, so its (CollectorId, EventId) is not the original identity: treat such events as a separate migration path. The origin label is always the archived envelope's stamped collector.id (a label the producer set is replaced). A collector with another id quarantines the event as archive-replay-collector-mismatch; the CLI counts it as collector-mismatch, continues and exits 2. --collector-id ID replays only that collector's events, so an archive that spans an id change is replayed one slice per matching collector. Events older than five years fail timeGenerated validation. --from and --to take the same strict format as policy diff --at (YYYY-MM-DDTHH:MM[:SS[.FFFFFFF]] with Z or an offset); other forms, including date-only values, exit 1.

  • bower archive create --archive DIR --database queue.db archives a queue by hand. While a collector is writing the same archive it fails on the writer lock instead of racing it; rerun it later.