Skip to main content

Syslog source

The collector can listen for syslog from a few high-value senders (network appliances, legacy Linux services) and turn selected messages into Bower events. It is a privacy gateway in front of those senders, not a syslog server for the estate: broad syslog fan-in stays with the Azure Monitor Agent. This page is about Bower receiving syslog; for Bower writing its events to local syslog for Azure Monitor Agent, see the syslog output.

sender ──RELP or TLS (or loopback TCP/UDP)──▶ listener ──▶ parse ──▶ route ──▶ signed pack parser ──▶ redact + policy ──▶ durable queue
allow-list RFC 5424 host/app match filter (default deny) (acknowledgement)
limits RFC 3164

The parser, limits, failure codes, record shape and event ids are pinned by test vectors (tests/fixtures/syslog-vectors/).

What is accepted​

Accepting a sender never means accepting its events. A message is queued only when:

  1. the sender's address is in the listener's allowedSenders;
  2. the frame is within the size limit and parses as RFC 5424 or RFC 3164;
  3. a route matches its hostname and app-name (first match wins);
  4. the routed signed pack parser's match filter selects the record; and
  5. redaction succeeds and the policy set accepts the resulting event.

Everything else is dropped and counted. No raw message, record or sender payload is logged, stored or shown; only the fields the pack parser maps reach the envelope, and they are redacted before persistence.

Configure​

Set BOWER_SYSLOG_CONFIG to a JSON file (at most 256 KiB). Invalid files, unknown properties, or a route that does not name an active parser of a loaded signed pack stop start-up.

{
"apiVersion": "bower.security/v1",
"kind": "SyslogSource",
"listeners": [
{
"id": "edge-tls",
"protocol": "tls",
"address": "0.0.0.0",
"port": 6514,
"allowedSenders": ["10.20.0.0/24", "2001:db8:20::/64"],
"tls": {
"certificateFile": "/etc/bower/syslog/server.pem",
"privateKeyFile": "/etc/bower/syslog/server-key.pem",
"clientCertificateAuthorityFile": "/etc/bower/syslog/senders-ca.pem"
},
"routes": [
{ "appName": "sshd", "pack": "linux-ssh-gateway", "parser": 0 }
]
},
{
"id": "local",
"protocol": "udp",
"port": 5514,
"allowedSenders": ["127.0.0.1"],
"routes": [{ "hostname": "fw01", "pack": "edge-firewall", "parser": 0 }]
}
]
}
SettingDefaultBounds
idrequired1–63 lowercase letters, digits, hyphens; unique
protocolrequiredtcp, tls, udp or relp (see RELP)
address127.0.0.1an IP address (canonical text)
portrequired0–65535 (0: chosen by the OS, for tests)
allowedSendersrequired1–64 addresses or CIDR ranges (host bits zero)
maximumMessageBytes65536480–65536
maximumConnections641–1024 (TCP, TLS and RELP)
idleTimeoutSeconds00–3600; 0 does not close an established healthy idle connection
handshakeStartTimeoutSeconds601–3600; TLS connection must send its first handshake byte in this time; a RELP session must also send open in this time
frameTimeoutSeconds601–3600; TLS handshake after its first byte, and one frame after its first byte
tlsnonerequired for tls, optional for relp (RELP over TLS), refused otherwise; PEM files: certificateFile, privateKeyFile, optional clientCertificateAuthorityFile and clientCertificateRevocationListFile (PEM or DER CRLs; requires the CA file)
allowInsecureNonLoopbackfalseallow tcp/udp beyond loopback
routesrequired1–64; pack, parser (index), optional hostname, appName, allowedSenders (1–64 addresses or CIDR ranges)

A non-loopback address needs protocol: tls (or relp with tls) unless allowInsecureNonLoopback is set; use the override only on an isolated network you control. With clientCertificateAuthorityFile, senders must present a client certificate issued by one of those CAs (mutual TLS); a sender may present intermediates between its certificate and those CAs. TLS 1.2 and 1.3 only.

With clientCertificateRevocationListFile, each client certificate is judged by the lists that name its issuer, are signed by the key of the CA that actually issued it (so a re-keyed CA keeping its name works while both keys are in use) and whose issuing distribution point covers it. The handshake is refused when no such list exists, when one lists the certificate, or when none is current (nextUpdate not passed) (end-entity check). A critical issuing distribution point is supported: a list naming distribution points covers only certificates whose CRL distribution points name one of them; a list limited to CA or attribute certificates covers no client. Lists limited to some reasons, indirect lists, distribution points relative to the issuer, delta lists, other critical extensions and signatures other than RSA PKCS#1 v1.5 or ECDSA (SHA-256/384/512) are refused at load with a message naming the feature. A client chain that can only end at an expired configured CA is refused.

Rotation without restart. A background task reads the certificate, key, CA bundle and revocation list once a second and reloads changed material for new handshakes; open connections are kept. Each file is read once per check, so the content checked is the content loaded. A change that cannot be loaded (a half-written file, a key that does not match the certificate, a path that is not a regular file such as a FIFO) is counted once in tlsReloadFailures, retried, and the last good material stays in use. Replace each file by an atomic rename, key before certificate. After a reload, or once a certificate in its chain or a revocation list expires, an open mutual-TLS session is validated again before its next frame is used; a session whose client is no longer trusted (CA removed, certificate revoked or expired) is closed and counted in tlsSessionsClosed. Revocation is checked before each sink attempt; it cannot retroactively cancel a sink submission already in progress, and detection depends on the material reload interval. Step-by-step rotation for every environment is in credential rotation and security failures.

Addresses are canonical IP text: no brackets, ports, IPv6 scopes or whitespace. Property names are case-sensitive and a repeated property fails the file.

The hostname and app-name a route matches are claimed by the sender, not verified: any allowed sender can claim hostname: fw01. Give a route its own allowedSenders to pin it to the addresses that may use it:

{ "hostname": "fw01", "allowedSenders": ["10.20.0.5"], "pack": "edge-firewall", "parser": 0 }

Senders should use RFC 5425-style TLS (octet-counted frames) on port 6514, RFC 5424 format, and send the security-relevant values as structured data. Prefer octet counting: with LF framing, a CR/LF that reaches the sender's message text starts a second frame (the live adversarial run queued such a forged frame); octet counting carries embedded newlines inside one frame.

The record a pack parser sees​

Each message becomes a flat JSON object of string values (at most 64 fields; values longer than 2048 characters are left out because a pack parser cannot read them):

FieldValue
idsyslog- + 32 hex of SHA-256 over listener, sender and the exact frame
listener, protocol, senderAddresswhere it arrived; senderAddress is IPv4 for IPv4-mapped IPv6 and never carries an IPv6 scope (fe80::1, not fe80::1%2)
receivedAtreceive time, UTC, microseconds
formatrfc5424 or rfc3164
facility, severitydecimal
timestampthe message time in UTC, or receivedAt when absent
hostname, appName, procId, msgId, messagewhen present
sd.<SD-ID>.<PARAM-NAME>each structured-data parameter

Map eventId to id and timeGenerated to timestamp. With a message timestamp, the same frame from the same listener and sender has the same id: a retransmission can deduplicate, but distinct byte-identical events can also collapse while the id remains in the queue's deduplication window. Without a timestamp the receive time is part of the id, so a retransmission at a different time can produce a duplicate. Plain syslog supplies no reliable way to tell these cases apart; qualify the actual sender and monitor its counts. RELP uses the same rule. A RELP resend is byte for byte the original, so with a message timestamp it is the same id and deduplicates. Without a timestamp (a - NILVALUE, or a template without one) the receive time is in the id: two distinct identical messages are both kept, but a resend after a reconnect becomes a second event (a duplicate row, never a lost event). Configure RELP senders with RFC 5424 high-precision timestamps (rsyslog RSYSLOG_SyslogProtocol23Format) so resends deduplicate. A pack parser for RFC 5424 sshd structured data:

{
"version": "1.0",
"format": "Json",
"fields": [
{ "sourceName": "id", "type": "Text", "ocsfPath": "metadata.uid", "asimField": "EventOriginalUid", "sensitive": false },
{ "sourceName": "timestamp", "type": "DateTime", "ocsfPath": "time", "asimField": "TimeGenerated", "sensitive": false },
{ "sourceName": "appName", "type": "Text", "ocsfPath": "metadata.product.name", "asimField": "EventProduct", "sensitive": false },
{ "sourceName": "sd.auth@32473.user", "type": "Text", "ocsfPath": "actor.user.name", "asimField": "TargetUsername", "sensitive": true },
{ "sourceName": "sd.auth@32473.src", "type": "IpAddress", "ocsfPath": "src_endpoint.ip", "asimField": "SrcIpAddr", "sensitive": false },
{ "sourceName": "sd.auth@32473.result", "type": "Text", "ocsfPath": "status", "asimField": "EventResult", "sensitive": false }
],
"runtime": {
"applicationName": "ssh-gateway", "environment": "production",
"eventCategory": "authentication", "eventType": "authentication_failure",
"eventAction": "authentication.attempt", "eventResult": "failure",
"match": { "appName": "sshd", "sd.auth@32473.result": "failure" },
"fields": {
"eventId": "id", "timeGenerated": "timestamp",
"actor.username": "sd.auth@32473.user", "source.ipAddress": "sd.auth@32473.src"
}
}
}

Free-text extraction from message (for example classic sshd lines) belongs in the routed parser's transforms (transform language), not in this source. A run-time transform failure quarantines the record without persisting it, as /v1/packs/.../events does: it counts under rejected and the pipeline transform stage, and a TCP connection keeps reading. CEF and LEEF are not parsed.

Parsing rules​

Strict and fail closed, with stable codes:

  • The whole frame must be valid UTF-8 (syslog-invalid-utf8); a leading BOM in MSG is removed.
  • PRI is 1–3 digits, no leading zero, 0–191 (syslog-priority-invalid).
  • RFC 5424 when PRI is followed by 1 ; other versions are refused (syslog-version-unsupported). Timestamps: uppercase T and Z, at most six fraction digits, no leap second, offset within ±14:00 (syslog-timestamp-invalid). Header fields are printable ASCII within RFC lengths (syslog-header-invalid). Duplicate SD-IDs or parameter names are refused (syslog-structured-data-duplicate); malformed structured data is syslog-structured-data-invalid.
  • Otherwise RFC 3164: Mmm dd hh:mm:ss (read as UTC; the year is the receive year, or the previous year when the date would be more than a day ahead) or an RFC 3339 timestamp, then an optional hostname and TAG[pid]:. A message without a timestamp is refused.

Framing, limits and backpressure​

  • TCP and TLS accept RFC 6587 octet counting (MSG-LEN SP MSG) and LF-terminated frames (a trailing CR is removed), chosen per frame. An oversized length, a bad count or any other framing error closes the connection; framing cannot be resynchronised safely. UDP is one message per datagram.
  • Once the first byte of a frame arrives, the whole frame must arrive within frameTimeoutSeconds, so senders that trickle bytes (slow loris) are closed. The TLS handshake has the same bound.
  • Before a TLS handshake begins, handshakeStartTimeoutSeconds limits the wait for its first byte. Once it starts, frameTimeoutSeconds bounds the handshake. A sender that opens TCP but delays TLS longer than this limit will be disconnected; test lazy TLS senders and reconnect behavior before rollout. Older collectors reject unknown fields: omit handshakeStartTimeoutSeconds when using its 60-second default. Setting a non-default value requires an updated collector that supports the field; guided setup omits it at the default. Older collectors retain their existing pre-handshake behavior when the field is absent.
  • Time between frames on an established connection is governed by idleTimeoutSeconds. The default, 0, never closes a healthy idle connection: a sender that never reads its socket (rsyslog) cannot see the close and loses what it writes next. With a non-zero value Bower half-closes at expiry (TCP FIN, no TLS close_notify) and keeps reading frames the sender already sent until it closes or is quiet for 5 seconds.
  • Accepted connections use TCP keepalive (first probe after 120 s idle, every 30 s, 30 probes): dead peers are closed after about 17 minutes, and keepalive traffic keeps load balancer and firewall flow state alive.
  • Connections beyond maximumConnections, and senders outside allowedSenders, are closed immediately. At most 16 submissions run at once across all listeners.
  • Acknowledgement point (TCP and TLS). A frame counts as accepted only once it is durably enqueued. Each connection reads its next frame only after the current one is enqueued or definitively dropped. When the queue is full (or a management-mode collector is waiting for its privacy profile) the connection stops reading and retries the same frame every second, so TCP flow control pushes back on the sender. A queue volume below its critical free-space watermark, or SQLite reporting the volume full, is the same backpressure (the frame is held and the connection kept); any other storage error closes the connection. Until the window closes, the kernel socket buffers keep accepting frames Bower has not read: on AWS about 21,500 frames (about 4 MB) were held this way and were lost when the collector restarted while backpressured. See storage pressure.
  • UDP is best effort. Datagrams wait in a buffer of 256 per listener; when it is full, or the queue is at capacity, they are dropped and counted. There is no way to push back on a UDP sender.
  • On stop. Bower stops accepting and half-closes connections while attempting to drain in-flight frames. The default per-connection quiet drain is 5 seconds and the graceful drain budget is 20 seconds. Neither guarantees delivery of every frame already sent. The final wait requires sinks and listeners to honour cancellation and can exceed the budget indefinitely. Size host grace periods for the workload and verify cancellation in the deployment.
  • On crash. Plain syslog has no application acknowledgement. Frames already in the kernel socket buffer or Bower's read buffer, and a frame waiting for queue space, are lost unless the sender retransmits; everything enqueued before the crash is durable. TCP/TLS with a sender-side disk queue reduces exposure but does not prove end-to-end lossless delivery. Qualify the actual sender with the delivery-reliability guide.
  • TLS 1.3 session tickets. .NET on OpenSSL sends two TLS 1.3 session tickets after every handshake and offers no setting to stop them. rsyslog never reads them, which hides Bower's half-close from rsyslog's pre-send check. Give the service OPENSSL_CONF with NumTickets = 0, or pin rsyslog to TLS 1.2; see the guide.

RELP (acknowledged delivery)​

protocol: relp speaks the Reliable Event Logging Protocol as rsyslog omrelp and librelp do, optionally over TLS with the same certificate, client CA, revocation-list and hot reload settings as tls. It closes the gap plain syslog leaves: Bower answers each syslog transaction 200 OK only after a final durable decision, so the sender keeps every unanswered message in its window and resends it after a reconnect.

  • Answered 200 OK: durably enqueued, already in the queue (duplicate), suppressed, not selected, rejected by policy, not routed, malformed, or oversized. These are final: the same bytes would get the same decision.
  • Never answered: a full queue or gated intake (Unavailable), a storage error, or a session whose client certificate stopped being trusted. A storage error or lost trust ends the session; the sender resends after it reconnects.
  • Bounded backpressure. A message the queue keeps refusing is retried every second for up to 30 seconds (RelpBusyTimeout, shorter than omrelp's 90-second response timeout); then Bower sends serverclose and ends the session without answering. Before closing it reads away what the sender still sends (bounded: until the sender closes or is quiet for 5 seconds, frameTimeoutSeconds, or 16 MiB), so the close is a FIN rather than a reset over unread bytes. The sender keeps the message and its window and reconnects; relpBusyCloses counts this, and each refused attempt also counts in retries. Bower never answers a refused message with an error code, because librelp drops a message from its resend list when it receives a non-200 response.
  • In order, one at a time. The next frame is read only after the current one is answered, so the sender's window (rsyslog default 128) bounds what is in flight.
  • Strict protocol. TXNR and DATALEN are 1–9 digits, the command 1–32 letters, each followed by exactly one space (anything else, such as 1Xopen or 2:syslog, ends the session), and a zero DATALEN is followed directly by LF. The first frame must be open (transaction 1, offering relp_version and commands=syslog; offers are bounded to 4 KiB, 16 offers of 16 values); transactions must follow 1, 2, 3 … and wrap after 999,999,999. Any other command, a second open, a client rsp, a bad trailer or an out-of-sequence number ends the session without an answer (framingErrors). An open without syslog is refused with 500 and the session ends.
  • Oversized messages. A syslog frame whose data exceeds maximumMessageBytes + 2 (a trailing CR LF is removed) is read past without buffering (any declared length up to 999,999,999 bytes; memory stays one fixed buffer and the whole frame must arrive within frameTimeoutSeconds), answered 200 OK and counted in oversized, so a sender is never stuck resending it. An oversized non-syslog frame ends the session. Keep the sender's maxMessageSize at or below maximumMessageBytes so nothing is dropped this way.
  • Stop and idle. On stop (and at a non-zero idle timeout) Bower sends the RELP serverclose hint, half-closes, then keeps reading and enqueuing frames the sender wrote before it saw the hint (unanswered; the sender resends them later and they deduplicate) until the sender closes or is quiet for 5 seconds. While closing, a frame the queue refuses gets one attempt only, then Bower stops processing and closes (the sender resends it). A client close is answered once every earlier transaction is answered.
  • Crash. Frames written but not committed when Bower is killed are unanswered, so the sender resends them; committed but unanswered frames come back as duplicates.

A RELP sender still loses its in-memory window if the sender is killed or loses power before resending; a disk-assisted action queue holds only messages not yet handed to the RELP window. See delivery reliability for measured results, sender configuration and deployment.

Monitoring​

GET /v1/status (token required) has a syslog array, one entry per listener, with counters since start: activeConnections, connectionsAccepted, connectionsRejected, sendersDenied, tlsHandshakeFailures, framesReceived, bytesReceived, queued, duplicates, suppressed, notRouted, notSelected, rejected, parserErrors, malformed, oversized, framingErrors, timeouts, dropped (UDP and backpressure), retries (TCP waits for queue space), submitFailures, receiveErrors (UDP socket receive errors; the listener backs off 100 ms to 1 s and keeps listening), halfCloses (connections Bower half-closed at an idle timeout or stop), drainedFrames (frames received after that half-close), peerResets (observed resets, including a reset before connection setup, which still frees its connection slot; best effort), relpAcknowledged (RELP transactions answered 200 OK), relpServerCloses (RELP sessions Bower ended with serverclose: stop, idle or busy), relpBusyCloses (RELP sessions ended because the queue kept refusing) and lastFrameAt. TLS and platform error reporting can prevent reset classification. Do not treat a zero count as proof of no resets. timeouts counts idle-timeout expiries and frames that missed frameTimeoutSeconds. TLS listeners add tlsReloads, tlsReloadFailures, tlsRevoked (handshakes refused by revocation), tlsSessionsClosed (open sessions closed after re-validation), serverCertificateNotAfter, clientCertificateAuthorityNotAfter (earliest trusted CA) and clientRevocationListNextUpdate (earliest list); the dates are null for other listeners. They are metadata only and counters reset on restart. Selected syslog events also appear in the pipeline and intake counters like any other source.

Silence detection works through the pack's application: list the routed parser's applicationName/environment in BOWER_SOURCE_EXPECTATIONS and a silent syslog sender degrades /health once its allowance passes.

Residual risks​

  • UDP senders can be spoofed; the allow-list checks only the source address. Prefer TLS with client certificates beyond loopback.
  • Hostname and app-name are claimed by the sender. Without a per-route allowedSenders, any allowed sender can reach any route (and its pack parser) by claiming its hostname or app-name; the parser filter, redaction and policy still apply.
  • An expired server certificate fails every new handshake until a valid one is in place (it is picked up without a restart). Alert on serverCertificateNotAfter.
  • A refused client certificate loses frames at the sender: rsyslog 8.2204 (ossl, TLS 1.2 or 1.3) writes one frame into each refused session and drops it. Rotate client CAs and certificates with overlap, and alert on tlsHandshakeFailures, tlsRevoked and clientRevocationListNextUpdate.
  • Plain TCP beyond loopback (with the override) is unauthenticated and unencrypted.
  • A sender allowed to connect can occupy up to maximumConnections slots. With the default idleTimeoutSeconds: 0 a healthy idle connection keeps its slot; keepalive closes dead ones after about 17 minutes. Set a non-zero idle timeout only for senders that read their sockets (for example syslog-ng).
  • Lost frames on crash, as above.

Not verified here​

Interoperability tests against a real sender daemon in a container are not part of the automated suite; the RFC example vectors, framing and TLS tests run against in-process clients.

The credential rotation campaign ran rsyslog 8.2204 over mutual TLS on Amazon Linux 2023: hot server-certificate rotation lost no frames and caused no reconnect; a collector restart under load lost one frame.

The manual native AWS pilot uses nftables and rsyslog over mutual TLS. It retains source-loss findings from both idle connection closure and per-event reconnect configurations. The delivery-reliability guide documents an interrupted follow-up campaign and a proposed sender and collector configuration; its results and evidence export are incomplete. A passing measured traffic window does not establish durable application acknowledgements or qualify another appliance's buffering and replay behavior.