Skip to main content

Queue v5 durability and SQL cursor v2

This change addresses DUR-02 through DUR-07 and OPS-01. It changes persisted queue state and settlement APIs. No Sentinel query or live SQL Server validation is implied by the local fixtures.

Identity and replay​

sha256:v2: fingerprints use UTF-8 byte-length framing over the identity version, collector ID, source adapter, application name/environment/tenant/instance, source hostname, identity kind and stable original source ID. Without an original ID, use the producer's stable event ID. Keep these source identifiers stable across restarts and failover. Source IP, correlation ID, outcome and timestamps are facts, not deduplication keys. Two attempts with different stable IDs persist independently. An original ID must identify one immutable fact within its source namespace.

Both event-ID and fingerprint uniqueness remain enforced. A collision is only an identical replay when both the stable fingerprint and normalized source content match. Reusing an event ID across source namespaces is an identity conflict. Object key order is normalized recursively; unknown source fields and number text are retained. Collector-owned collector, security, timeObserved and top-level attributes.bowerDetections decoration is excluded. Pack upgrades can change detection metadata without making a source replay conflict; retained evidence is not rewritten. Other source attributes remain part of strict content comparison. When eventOriginalId is present, reconstructed eventId is excluded from content comparison. Changed source content raises EventIdentityConflictException. The processor returns a quarantine result with that code, never duplicate success. Original data remains. This comparison operates on redacted content; removed original secrets are never stored or compared.

Deduplication lasts while the acknowledged row remains retained. After retention purges that row, the same stable identity may be accepted as a new incarnation. Unacknowledged, retrying and dead-lettered rows are never removed by retention.

Upgrade, rollback and mixed writers​

Stop old writers and take a consistent SQLite backup, including committed WAL state, before upgrading (bower queue backup; see Upgrades, backup and recovery). Builds from this release check schema_history before any DDL and refuse a newer schema without changing the file; earlier builds did not. Startup transactionally renames the writable table to queue_events_v5, adds enqueue_sequence and lease_generation, creates retention checkpoint storage and records schema version 5. The legacy queue_events name becomes a read-only view. Old binaries cannot insert, lease, settle, replay or purge through it; their old index initialization also fails rather than silently writing.

Existing valid envelope fingerprints are recomputed without changing payload or ledger bytes. Opaque legacy queue clients retain their explicit fingerprint. A new identity collision, missing live ledger entry or corrupt chain aborts the migration. No accepted row is discarded to resolve a collision. Investigate the source identity configuration and restore the pre-upgrade backup for old-version operation. Do not drop the view or rename tables to force a downgrade.

Before upgrading either runtime, run the standalone C# identity preflight against the stopped v4 queue or a consistent backup. It uses the canonical C# EventIdentity implementation and does not invoke either runtime's queue startup:

dotnet run --project tools/queue-migration-check --configuration Release -- --database /absolute/path/bower.db

Build the tool before the maintenance window. Exit 0 means the bounded identity scan completed without collisions; 2 means collisions; 1 means refused or incomplete (including limits, unsupported schema or unreadable database). JSON contains completion/status, scanned/opaque row counts and colliding event-ID pairs only. No payload, original source ID, fingerprint or parser error is printed. The check opens SQLite read-only, uses a single read transaction, and never initializes, migrates, checkpoints or changes DB/WAL/schema contents. SQLite may use its normal shared-memory read locks; this is not a forensic no-filesystem- activity tool. Never use SQLite immutable mode to skip a committed WAL.

Limits are 100,000 rows, 1 MiB per payload, 64 MiB total payload and 100 reported pairs. Hitting a limit is an incomplete result, never a clean bill of health. Only known v4 schema history is supported; future, v5, older and unknown schemas are refused. Opaque rows retain their explicit fingerprints, matching migration. Pairs include conflicts in ordered re-keying, including retained opaque keys. This is an identity check, not a ledger verification or guarantee that the full upgrade will succeed. Keep writers stopped between the check and upgrade.

If collisions appear, preserve the backup and keep the v4 queue intact. Investigate the reported IDs in the authorized source system: distinguish a resend of one fact from producers reusing source identities. Record the reconciliation decision and evidence. Drain/acknowledge through the supported old runtime and normal retention where appropriate, or seek a separately reviewed data-preserving reconciliation plan. Do not delete accepted rows, rewrite payloads/ledger entries, or remove uniqueness constraints to force upgrade. Repeat the preflight on the reconciled consistent backup before upgrading. No automatic reconciliation or deletion is provided.

Legacy DDL/backfill and v5 migration are transactional. Restart after an aborted transaction sees the prior committed state and can retry. Tests inject rollback at schema-history insertion on populated old databases and verify restart, data preservation and old-writer refusal. They also exercise failed settlement rollback. These are SQLite interruption simulations, not power-loss certification.

The C# collector also opens and drains a committed v5 database written by the retired Rust collector (tests/fixtures/persistence-vectors/rust-queue-v5.db). Concurrent v5 workers must use the same queue limits and source identity configuration. Older writers and future unknown schema versions are unsupported and refused. There is no in-place rollback to v4; use the preserved backup and reconcile all events accepted since that backup before resuming old software.

Lease ownership​

Each lease returns LeaseToken: enqueue_sequence:lease_generation. Generation increases on every acquisition and is not reset by dead-letter replay. A post-retention enqueue gets a new sequence. Settlement predicates require both uploading state and the exact token supplied with the acquired lease.

Single-record settlement now requires a token before the cancellation argument. Delivered batches take an event-ID/token map. Failure settlements carry the token themselves. Missing or stale claims cannot mutate state; batch APIs return those IDs, single APIs reject the transition. Workers must use claims from their original batch, never read current claims at settlement time.

C# delivery deadlines and restart recovery​

BOWER_DELIVERY_LEASE_SECONDS defaults to 300 and accepts 31–3600 seconds. BOWER_DELIVERY_TIMEOUT_SECONDS defaults to 240 and must leave at least 30 seconds between the output cancellation deadline and lease expiry for queue settlement. Invalid combinations fail collector startup, including when delivery is disabled. Nondefault delivery profiles append both values to the reported configuration hash; the default profile preserves existing configuration hashes. Configuration identity does not identify software behavior: record the binary version and artifact hash separately. No database migration or lease-token change is involved.

The output receives a token linked to the deadline and host shutdown. A deadline cancellation becomes destination-output-timeout: the batch stays durable, the destination circuit pauses delivery for 30 seconds, and the poison-record attempt budget is preserved. Settlement uses the host token, so an expired output deadline does not prevent committing the retry. Host shutdown still leaves outstanding leases intact for recovery. Metadata-only warnings never include the batch body.

If an AMA file or an owned Azure request completed before a later operation was cancelled, the adapter returns those known acknowledgements and marks only the unfinished events with destination-ama-cancelled or destination-azure-cancelled. The worker commits those acknowledgements instead of replaying confirmed output. Restart regressions cover both partial-output paths. With no acknowledged output, the adapter still propagates cancellation to the worker's deadline handling.

Cancellation is cooperative. The worker awaits the adapter rather than abandoning an upload task and launching overlapping attempts. Adapters must honor cancellation; a blocked filesystem or an adapter that ignores the token cannot be made responsive by this setting. A request may reach the destination before its response is lost or cancelled: replay remains at least once, not exactly once. Reconcile by event identity and measure duplicates at the destination; lease fencing protects queue state, not external side effects. Sentinel queries and rules should use the replay-safe BowerSecurityEvents parser and rules.

For qualification of the 120-second crash-recovery target, the candidate profile is a 90-second lease and 60-second output timeout. Set both environment variables. Existing leases keep their original expiry after a configuration change. Shorter deadlines can interrupt valid large uploads on slow links, so benchmark batch sizes and destination latency before selecting this profile. No startup path resets all leases or steals another worker's live claim.

DeliveryDeadlineTests covers timed cancellation, retained retry state across restarts, destination-circuit delay, preserved poison budget, successful slow acknowledgement, host cancellation and live-lease exclusion. Its clock-driven restart checks are not a process-kill, Windows reboot or real Azure delivery test. The failed AWS recovery scorecard remains unresolved until the exact candidate artifact passes those external checks; see production readiness.

Throttling and pauses​

The Logs Ingestion output adapts request concurrency inside one delivery and never holds events outside the queue: a leased batch either settles or expires and is reclaimed. After a Retry-After, batches of that delivery that were not sent fail as destination-azure-paused with the server deadline (capped at one hour). Like every destination-* failure, they do not spend delivery attempts and open the durable destination circuit, so nothing is leased for the stream until the deadline, across restarts. Leases stay valid while a delivery waits for a slot (bounded by the batch and the lease duration). Acknowledgement, dead-letter and ledger semantics are unchanged. See adaptive request concurrency.

Retention and physical storage​

Purging and checkpoint compaction commit together. Consecutive purged ledger rows collapse into one checkpoint retaining start/end sequence, previous/end chain hashes and a boundary hash. Adjacent checkpoints merge. At most one purged run exists between each pair of live ledger rows; an unresolved early event therefore cannot pin all later history. Live payload verification is keyed by enqueue sequence, not a reusable event ID. Chain/checkpoint corruption aborts compaction. The witnessed ledger head stays stable across compaction.

Checkpoints are local hash-bound evidence with the same trust boundary as the old local anchor, not independently signed attestations. A writer controlling the whole database can rewrite local evidence. Continue independent external head witnessing; do not infer destination delivery from local acknowledgement or hashes.

For configured logical payload limit M, the physical DB+WAL ceiling is 4 * M + 16 MiB. Every write reserves two capped database sizes plus WAL frame/header allowance before starting mutations: one for the WAL transaction and one for database growth during checkpoint. Cache spill is disabled, WAL autocheckpoint runs after each commit and retained WAL size is zero after a successful reset. Accepted data is never deleted to satisfy the limit. A pinned reader can prevent checkpointing; writers then report capacity/backpressure until the reader closes. In the C# collector the next write over budget truncates the WAL itself (at most every 10 seconds) instead of waiting for maintenance. A database already larger than its page allowance is refused; configure an adequate limit before migration. Database free pages are reused.

Both collectors: new events stop once database pages in use reach 85% of one quarter of that ceiling (M + 4 MiB), while leasing, acknowledgement, retry and dead-letter writes may grow the file to 1.75 × (M + 4 MiB) (the SQLite max_page_count). Settlement enlarges rows and splits packed pages; without that room a backlog that filled the pages could never be acknowledged (seen on AWS, see storage pressure). A queued row takes about 2.2 times its payload, so the page limit binds before M. Builds before this change keep the single M + 4 MiB page ceiling and refuse a database grown past it; see the rollback procedure.

The physical cap includes retained delivered payloads, ledger/checkpoint pages and indexes; it can reject new events before the logical undelivered payload limit. The C# collector's retention job reclaims acknowledged rows early under page or disk pressure (BOWER_QUEUE_PRESSURE_RETENTION_MINUTES), and of any age once new events are refused; undelivered, retrying and dead-lettered rows are never removed. SQLite FULL is surfaced as capacity, not duplicate success. Exact replay still wins over normal admission backpressure. A suppressed event commits a counter change, so it takes the same physical check after the duplicate lookup: when the budget is spent it is refused with capacity, its window counts stay unchanged and it is not reported as suppressed. This limit covers the queue DB and WAL, not unrelated disk users, output spool directories or filesystem failure; the C# collector's free-space watermarks (BOWER_STORAGE_*) cover the volume itself. SQLite's small shared-memory file and transient memory are additional overhead. All writers must use the supported APIs and matching limits; raw administrative SQL is outside this enforcement boundary.

Unclean shutdown and temporary files​

Every queue connection runs with PRAGMA synchronous = FULL in WAL mode (C# and Rust), so a committed transaction, and therefore every 202, survives a process kill; the cost is one flush per commit, which is why intake throughput follows the disk's flush latency. A power cut is covered by SQLite's WAL recovery only to the extent the disk honours flushes; this is not power-loss certified (#136 tracks the lab test).

The C# collector writes collector.running beside the queue at start and removes it when the host stops cleanly. If the file is still there at the next start, the previous run was killed, crashed or lost power: the collector verifies the queue ledger before serving, logs warning 1600 (error 1601 if the ledger is broken; run bower queue verify) and reports the result in /v1/status as recovery (uncleanShutdown, previousStartedAt, ledgerIntact, ledgerIssues, temporaryFilesRemoved, checkedAt). It never deletes or rewrites queue rows because of it.

At start and hourly, the collector removes its own abandoned atomic-write temporaries (.NAME.<32 hex>.tmp, from the collector id, management credential, command ledger and privacy profile cache writers) older than one hour, from the queue directory and its managed-privacy, management-credential and management-commands directories only, not recursively, at most 1,000 per run. The AMA spool and the archive already clean their own temporaries when they open. No other file is touched.

SQL Server timestamp cursor v3 and allocation bounds​

Timestamp cursor v3 keeps forward (eventTime, sequence) progress separate from a bounded overlap sweep. Forward and replay pages alternate, each limited to BatchSize, even when every forward page is full. Both streams use strict keyset comparison, including the sequence tie-breaker. Zero overlap uses forward pages only. An already pending sweep finishes even if overlap is changed to zero.

The first forward checkpoint atomically saves its new watermark and a replay lower bound derived from the pre-page watermark minus TimestampOverlap. The sweep's upper (eventTime, sequence) bound is fixed at that page's new watermark. Replay position, upper bound and next-page selection survive restart; subsequent forward pages cannot move the active sweep's bounds. After a short replay page completes the sweep, its upper bound remains saved. The next sweep starts at that saved upper timestamp minus overlap and ends at the new forward watermark. This covers forward progress made during the previous sweep without a gap. A full page at the upper bound needs a later empty replay page to confirm completion.

This is a finite event-time lateness contract, not a wall-clock commit-delay guarantee. A row within a pending sweep is recovered if it is visible before that sweep's keyset passes it, and it remains readable until fetched. A row becoming visible behind the replay position needs a subsequent sweep whose lower bound still includes it. Advancing forward pages cannot discard a pending interval, but arbitrarily delayed or backdated commits can fall outside subsequent intervals. Choose overlap for the source's timestamp/commit behavior and polling throughput; use an operator-controlled replay for older rows. Timestamp and sequence must identify immutable, deterministically ordered facts. Overlap intentionally repeats rows; the sink must durably deduplicate stable fingerprints.

Version 1 and 2 checkpoints upgrade in memory when read; subsequent checkpoints are v3. A v2 active sweep retains its exact replay position and old forward watermark as its fixed upper bound. A legacy forward-only checkpoint starts the new overlap at its stored watermark; migration cannot recover rows already skipped outside that horizon. Reconcile such historical gaps with an operator replay. Old adapters reject v3. Stop old SQL pollers before upgrading, back up the cursor store consistently, and do not run mixed cursor versions. Downgrade requires the pre-upgrade cursor backup plus reconciliation of subsequently accepted rows; do not edit the schema version to force old readers to accept new state.

Optimistic checkpoint versions reject concurrent stale commits. Commit a checkpoint only after every selected row has been durably accepted by the sink, including checkpoints on empty pages that change replay state. A crash before commit repeats the page; it never skips ahead. No persistence queue or cursor-store API changes are required.

EF Core LINQ projects bounded fields on SQL Server before ToListAsync. DATALENGTH (including trailing spaces), summed as bigint, bounds combined identity columns and combined value columns independently to MaximumRecordBytes each. Oversized identity columns return only a marker and stop polling without advancing. Oversized value columns return null plus a marker: OmitValues emits the marked bounded audit fact, while Fail stops without advancing. The final serialized row limit still applies, including escaping. Row count times configured record size is at most 64 MiB; the projected string data is at most twice that bound, plus CLR and serialization overhead. No unrestricted value is first materialized then discarded.

Tests cover the actual EF-generated SQL, in-memory keyset pages, persisted restart, full forward backlogs with late commits, interleaved tied-timestamp replay, crashes before checkpoint commit, stale commits, legacy migration, malformed bounds and rejection markers. A live SQL Server allocation/plan test remains required in the integration environment.