Logs Ingestion API mode
Configure HTTPS DCE endpoint, immutable DCR ID and stream name. Choose exactly
one credential source with BOWER_AZURE_CREDENTIAL (managed-identity default,
workload-identity, environment, azure-cli for development). Set
BOWER_AZURE_CLIENT_ID for a user-assigned managed identity. Assign only the
Monitoring Metrics Publisher (DCR data sender) role on the required rule.
BOWER_OUTPUT=azure-logs-ingestion
BOWER_DCE_ENDPOINT=https://<name>.<region>.ingest.monitor.azure.com
BOWER_DCR_ID=dcr-<immutable-id>
BOWER_STREAM_NAME=Custom-BowerSecurity
Bower builds bounded requests and tracks the event IDs belonging to each request;
concurrent completion cannot attach another batch's failure to those IDs. Bower uses
the supported Azure SDK protocol upload API. Successful
responses yield azure-logs-ingestion:accepted:owned-batches. This records API
acceptance only, never Sentinel queryability.
The Log Analytics query API can cache responses. Bower's evidence client requests
fresh results and rejects partial query responses (see
evidence bundles); custom
reconciliation tools should likewise send Cache-Control: no-cache and fail closed
on partial errors.
Local malformed/mismatched payloads and failed schema preflight are individual
permanent failures. A credential the identity service rejects (any 4xx answer other than 408/429, even
after a retried network error) or a missing or unusable credential configuration fails
as retryable destination-azure-credential-failed. A token service that cannot be
reached (DNS, proxy, TLS, timeout) or answers 5xx/408/429 is
destination-azure-unavailable, like any other outage. A missing role is
destination-azure-http-403.
Remote authentication, DCR/stream configuration, throttling,
transport and server failures pause delivery through a durable destination circuit.
Retry-After survives terminal SDK retries and collector restart. Configuration
outages do not consume per-record poison budgets. Ambiguous HTTP 400 batches use
bounded isolation; unresolved singletons remain pending for operator investigation.
Queue/storage limits apply producer backpressure rather than deleting pending
records. See outage recovery and historical reconciliation.
Adaptive request concurrency
Each output (one DCR stream) runs an AIMD controller over its upload requests.
MaximumConcurrency (1 to 32, default
4) is the ceiling and the starting limit, so the healthy case is never slower than
the old fixed setting. The minimum is 1. The collector uses one controller for the
whole output (ConcurrencyScope.Output), because its single delivery loop is the only
caller. The Azure Functions host delivers up to httpPerInstanceConcurrency caller
requests at once through one output, so it uses ConcurrencyScope.Delivery: each
request gets its own controller, bounded as the fixed setting was.
- Decrease: halve the limit on 429, 5xx, 408, network timeout or transport failure; cut it by a quarter on a latency spike (RTT above twice the EWMA and at least 250 ms above it, judged after four samples; EWMA weight 0.25). At most one decrease per window: attempts started before the last decrease do not count.
- Increase: one slot after
limitconsecutive healthy attempts in the current window, up to the ceiling. - Every attempt, including Azure.Core / pipeline retries, is observed after the token is obtained. Other 4xx responses and caller cancellation are neutral.
Retry-After(andretry-after-ms) on any failed attempt pauses the stream: no further batch of that delivery starts before the deadline. That includes short delays (60 s or less) that the SDK / retry loop waits out in memory before retrying the same batch. Unsent batches fail as retryabledestination-azure-pausedwith that deadline, which opens the durable destination circuit (see queue v5 durability).- Back-pressure is a wait for a request slot over the already-leased batch, in batch order. Nothing is buffered beyond it; queue byte caps still return 503 to producers. No change to acknowledgement, dead-letter or ledger semantics.
The controller is deterministic: it takes RTT and time from its caller
(TimeProvider), with no randomness. /v1/status (bearer token) adds a delivery
block, omitted for outputs without adaptive concurrency:
"delivery": {"outputId": "azure-logs-ingestion", "limit": 2, "minimumLimit": 1,
"maximumLimit": 4, "inFlight": 1, "latencyEwmaMilliseconds": 163, "requests": 9,
"throttles": 3, "serverErrors": 1, "timeouts": 0, "latencySpikes": 1,
"decreases": 2, "increases": 1, "pauses": 1,
"processLocalPausedUntil": "2026-07-26T02:16:19+00:00"}
Counters are since start and metadata only: no payloads, event IDs or response
bodies. The whole block is process-local and resets on restart.
processLocalPausedUntil is this process's latest Retry-After deadline, null once
it passes. It is not the durable pause: after a restart the queue's destination
circuit still holds the stream (retrying destination-* rows, see
/v1/status queue.retrying) while this field reads null. The Functions host
reports no block because it keeps no output-wide controller. Tests: a controller
trace (AdaptiveConcurrencyTests), and a fake
ingestion endpoint injecting 429, 503, latency and Retry-After, through the real
SQLite queue and delivery worker (no loss, no premature acknowledgement, no
dead-lettering of retryable events, cancellation releases every slot). Not
verified against a live Azure endpoint.
Rotate and revoke ingestion credentials with the credential rotation runbook: secrets, certificates and environment variables are read when the process starts (restart to apply), while a cached access token keeps working until it expires (measured 62–67 minutes after issue).
Use the Entra ingestion planner for exact-DCR sender RBAC and optional managed identity/federation. An existing Log Analytics workspace and permission to create the reviewed table/DCE/DCR are prerequisites. Keep ingest identity separate from the read-only query verifier. Never persist access tokens or put credentials in arguments/source-controlled configuration.
Current templates and audience defaults target Azure public cloud. DNS and TLS access to the selected ingestion endpoint and Entra token service are required; private endpoint deployments must provide their own approved DNS/network route. Do not point public-cloud credentials at a sovereign-cloud endpoint and claim it supported: authority, ingestion audience, query/ARM hosts and deployment APIs need a separately tested cloud configuration. No network or identity proof is implied by template compilation.