Syslog source
The collector can listen for syslog from a few high-value senders (network appliances, legacy Linux services) and turn selected messages into Bower events. It is a privacy gateway in front of those senders, not a syslog server for the estate: broad syslog fan-in stays with the Azure Monitor Agent. This page is about Bower receiving syslog; for Bower writing its events to local syslog for Azure Monitor Agent, see the syslog output.
sender ──RELP or TLS (or loopback TCP/UDP)──▶ listener ──▶ parse ──▶ route ──▶ signed pack parser ──▶ redact + policy ──▶ durable queue
allow-list RFC 5424 host/app match filter (default deny) (acknowledgement)
limits RFC 3164
The parser, limits, failure codes, record shape and event ids are pinned by test vectors
(tests/fixtures/syslog-vectors/).
What is accepted
Accepting a sender never means accepting its events. A message is queued only when:
- the sender's address is in the listener's
allowedSenders; - the frame is within the size limit and parses as RFC 5424 or RFC 3164;
- a route matches its hostname and app-name (first match wins);
- the routed signed pack parser's
matchfilter selects the record; and - redaction succeeds and the policy set accepts the resulting event.
Everything else is dropped and counted. No raw message, record or sender payload is logged, stored or shown; only the fields the pack parser maps reach the envelope, and they are redacted before persistence.
Configure
Set BOWER_SYSLOG_CONFIG to a JSON file (at most 256 KiB). Invalid files, unknown
properties, or a route that does not name an active parser of a loaded signed pack
stop start-up.
{
"apiVersion": "bower.security/v1",
"kind": "SyslogSource",
"listeners": [
{
"id": "edge-tls",
"protocol": "tls",
"address": "0.0.0.0",
"port": 6514,
"allowedSenders": ["10.20.0.0/24", "2001:db8:20::/64"],
"tls": {
"certificateFile": "/etc/bower/syslog/server.pem",
"privateKeyFile": "/etc/bower/syslog/server-key.pem",
"clientCertificateAuthorityFile": "/etc/bower/syslog/senders-ca.pem"
},
"routes": [
{ "appName": "sshd", "pack": "linux-ssh-gateway", "parser": 0 }
]
},
{
"id": "local",
"protocol": "udp",
"port": 5514,
"allowedSenders": ["127.0.0.1"],
"routes": [{ "hostname": "fw01", "pack": "edge-firewall", "parser": 0 }]
}
]
}
| Setting | Default | Bounds |
|---|---|---|
id | required | 1–63 lowercase letters, digits, hyphens; unique |
protocol | required | tcp, tls, udp or relp (see RELP) |
address | 127.0.0.1 | an IP address (canonical text) |
port | required | 0–65535 (0: chosen by the OS, for tests) |
allowedSenders | required | 1–64 addresses or CIDR ranges (host bits zero) |
maximumMessageBytes | 65536 | 480–65536 |
maximumConnections | 64 | 1–1024 (TCP, TLS and RELP) |
idleTimeoutSeconds | 0 | 0–3600; 0 does not close an established healthy idle connection |
handshakeStartTimeoutSeconds | 60 | 1–3600; TLS connection must send its first handshake byte in this time; a RELP session must also send open in this time |
frameTimeoutSeconds | 60 | 1–3600; TLS handshake after its first byte, and one frame after its first byte |
tls | none | required for tls, optional for relp (RELP over TLS), refused otherwise; PEM files: certificateFile, privateKeyFile, optional clientCertificateAuthorityFile and clientCertificateRevocationListFile (PEM or DER CRLs; requires the CA file) |
allowInsecureNonLoopback | false | allow tcp/udp beyond loopback |
routes | required | 1–64; pack, parser (index), optional hostname, appName, allowedSenders (1–64 addresses or CIDR ranges) |
A non-loopback address needs protocol: tls (or relp with tls) unless allowInsecureNonLoopback is set;
use the override only on an isolated network you control. With
clientCertificateAuthorityFile, senders must present a client certificate issued by
one of those CAs (mutual TLS); a sender may present intermediates between its
certificate and those CAs. TLS 1.2 and 1.3 only.
With clientCertificateRevocationListFile, each client certificate is judged by the
lists that name its issuer, are signed by the key of the CA that actually issued it
(so a re-keyed CA keeping its name works while both keys are in use) and whose issuing
distribution point covers it. The handshake is refused when no such list exists, when
one lists the certificate, or when none is current (nextUpdate not passed)
(end-entity check). A critical issuing distribution point is supported: a list naming
distribution points covers only certificates whose CRL distribution points name one of
them; a list limited to CA or attribute certificates covers no client. Lists limited to
some reasons, indirect lists, distribution points relative to the issuer, delta lists,
other critical extensions and signatures other than RSA PKCS#1 v1.5 or ECDSA
(SHA-256/384/512) are refused at load with a message naming the feature. A client chain
that can only end at an expired configured CA is refused.
Rotation without restart. A background task reads the certificate, key, CA bundle
and revocation list once a second and reloads changed material for new handshakes;
open connections are kept. Each file is read once per check, so the content checked is
the content loaded. A change that cannot be loaded (a half-written file, a key that does
not match the certificate, a path that is not a regular file such as a FIFO) is counted
once in tlsReloadFailures, retried, and the last good material stays in use. Replace each file by an
atomic rename, key before certificate. After a reload, or once a certificate in its
chain or a revocation list expires, an open mutual-TLS session is validated again
before its next frame is used; a session whose client is no longer trusted (CA
removed, certificate revoked or expired) is closed and counted in tlsSessionsClosed.
Revocation is checked before each sink attempt; it cannot retroactively cancel a sink
submission already in progress, and detection depends on the material reload interval.
Step-by-step rotation for every environment is in
credential rotation and security failures.
Addresses are canonical IP text: no brackets, ports, IPv6 scopes or whitespace. Property names are case-sensitive and a repeated property fails the file.
The hostname and app-name a route matches are claimed by the sender, not verified:
any allowed sender can claim hostname: fw01. Give a route its own
allowedSenders to pin it to the addresses that may use it:
{ "hostname": "fw01", "allowedSenders": ["10.20.0.5"], "pack": "edge-firewall", "parser": 0 }
Senders should use RFC 5425-style TLS (octet-counted frames) on port 6514, RFC 5424 format, and send the security-relevant values as structured data. Prefer octet counting: with LF framing, a CR/LF that reaches the sender's message text starts a second frame (the live adversarial run queued such a forged frame); octet counting carries embedded newlines inside one frame.
The record a pack parser sees
Each message becomes a flat JSON object of string values (at most 64 fields; values longer than 2048 characters are left out because a pack parser cannot read them):
| Field | Value |
|---|---|
id | syslog- + 32 hex of SHA-256 over listener, sender and the exact frame |
listener, protocol, senderAddress | where it arrived; senderAddress is IPv4 for IPv4-mapped IPv6 and never carries an IPv6 scope (fe80::1, not fe80::1%2) |
receivedAt | receive time, UTC, microseconds |
format | rfc5424 or rfc3164 |
facility, severity | decimal |
timestamp | the message time in UTC, or receivedAt when absent |
hostname, appName, procId, msgId, message | when present |
sd.<SD-ID>.<PARAM-NAME> | each structured-data parameter |
Map eventId to id and timeGenerated to timestamp. With a message timestamp,
the same frame from the same listener and sender has the same id: a retransmission can
deduplicate, but distinct byte-identical events can also collapse while the id remains in
the queue's deduplication window. Without a timestamp the receive time is part of the id,
so a retransmission at a different time can produce a duplicate. Plain syslog supplies no
reliable way to tell these cases apart; qualify the actual sender and monitor its counts.
RELP uses the same rule. A RELP resend is byte for byte the original, so with a message
timestamp it is the same id and deduplicates. Without a timestamp (a - NILVALUE, or a
template without one) the receive time is in the id: two distinct identical messages are
both kept, but a resend after a reconnect becomes a second event (a duplicate row, never a
lost event). Configure RELP senders with RFC 5424 high-precision timestamps (rsyslog
RSYSLOG_SyslogProtocol23Format) so resends deduplicate.
A pack parser for RFC 5424 sshd
structured data:
{
"version": "1.0",
"format": "Json",
"fields": [
{ "sourceName": "id", "type": "Text", "ocsfPath": "metadata.uid", "asimField": "EventOriginalUid", "sensitive": false },
{ "sourceName": "timestamp", "type": "DateTime", "ocsfPath": "time", "asimField": "TimeGenerated", "sensitive": false },
{ "sourceName": "appName", "type": "Text", "ocsfPath": "metadata.product.name", "asimField": "EventProduct", "sensitive": false },
{ "sourceName": "sd.auth@32473.user", "type": "Text", "ocsfPath": "actor.user.name", "asimField": "TargetUsername", "sensitive": true },
{ "sourceName": "sd.auth@32473.src", "type": "IpAddress", "ocsfPath": "src_endpoint.ip", "asimField": "SrcIpAddr", "sensitive": false },
{ "sourceName": "sd.auth@32473.result", "type": "Text", "ocsfPath": "status", "asimField": "EventResult", "sensitive": false }
],
"runtime": {
"applicationName": "ssh-gateway", "environment": "production",
"eventCategory": "authentication", "eventType": "authentication_failure",
"eventAction": "authentication.attempt", "eventResult": "failure",
"match": { "appName": "sshd", "sd.auth@32473.result": "failure" },
"fields": {
"eventId": "id", "timeGenerated": "timestamp",
"actor.username": "sd.auth@32473.user", "source.ipAddress": "sd.auth@32473.src"
}
}
}
Free-text extraction from message (for example classic sshd lines) belongs in the
routed parser's transforms
(transform language), not in this
source. A run-time transform failure quarantines the record without persisting it, as
/v1/packs/.../events does: it counts under rejected and the pipeline transform
stage, and a TCP connection keeps reading. CEF and LEEF are not parsed.
Parsing rules
Strict and fail closed, with stable codes:
- The whole frame must be valid UTF-8 (
syslog-invalid-utf8); a leading BOM in MSG is removed. - PRI is 1–3 digits, no leading zero, 0–191 (
syslog-priority-invalid). - RFC 5424 when PRI is followed by
1; other versions are refused (syslog-version-unsupported). Timestamps: uppercaseTandZ, at most six fraction digits, no leap second, offset within ±14:00 (syslog-timestamp-invalid). Header fields are printable ASCII within RFC lengths (syslog-header-invalid). Duplicate SD-IDs or parameter names are refused (syslog-structured-data-duplicate); malformed structured data issyslog-structured-data-invalid. - Otherwise RFC 3164:
Mmm dd hh:mm:ss(read as UTC; the year is the receive year, or the previous year when the date would be more than a day ahead) or an RFC 3339 timestamp, then an optional hostname andTAG[pid]:. A message without a timestamp is refused.
Framing, limits and backpressure
- TCP and TLS accept RFC 6587 octet counting (
MSG-LEN SP MSG) and LF-terminated frames (a trailing CR is removed), chosen per frame. An oversized length, a bad count or any other framing error closes the connection; framing cannot be resynchronised safely. UDP is one message per datagram. - Once the first byte of a frame arrives, the whole frame must arrive within
frameTimeoutSeconds, so senders that trickle bytes (slow loris) are closed. The TLS handshake has the same bound. - Before a TLS handshake begins,
handshakeStartTimeoutSecondslimits the wait for its first byte. Once it starts,frameTimeoutSecondsbounds the handshake. A sender that opens TCP but delays TLS longer than this limit will be disconnected; test lazy TLS senders and reconnect behavior before rollout. Older collectors reject unknown fields: omithandshakeStartTimeoutSecondswhen using its 60-second default. Setting a non-default value requires an updated collector that supports the field; guided setup omits it at the default. Older collectors retain their existing pre-handshake behavior when the field is absent. - Time between frames on an established connection is governed by
idleTimeoutSeconds. The default, 0, never closes a healthy idle connection: a sender that never reads its socket (rsyslog) cannot see the close and loses what it writes next. With a non-zero value Bower half-closes at expiry (TCP FIN, no TLSclose_notify) and keeps reading frames the sender already sent until it closes or is quiet for 5 seconds. - Accepted connections use TCP keepalive (first probe after 120 s idle, every 30 s, 30 probes): dead peers are closed after about 17 minutes, and keepalive traffic keeps load balancer and firewall flow state alive.
- Connections beyond
maximumConnections, and senders outsideallowedSenders, are closed immediately. At most 16 submissions run at once across all listeners. - Acknowledgement point (TCP and TLS). A frame counts as accepted only once it is durably enqueued. Each connection reads its next frame only after the current one is enqueued or definitively dropped. When the queue is full (or a management-mode collector is waiting for its privacy profile) the connection stops reading and retries the same frame every second, so TCP flow control pushes back on the sender. A queue volume below its critical free-space watermark, or SQLite reporting the volume full, is the same backpressure (the frame is held and the connection kept); any other storage error closes the connection. Until the window closes, the kernel socket buffers keep accepting frames Bower has not read: on AWS about 21,500 frames (about 4 MB) were held this way and were lost when the collector restarted while backpressured. See storage pressure.
- UDP is best effort. Datagrams wait in a buffer of 256 per listener; when it is full, or the queue is at capacity, they are dropped and counted. There is no way to push back on a UDP sender.
- On stop. Bower stops accepting and half-closes connections while attempting to drain in-flight frames. The default per-connection quiet drain is 5 seconds and the graceful drain budget is 20 seconds. Neither guarantees delivery of every frame already sent. The final wait requires sinks and listeners to honour cancellation and can exceed the budget indefinitely. Size host grace periods for the workload and verify cancellation in the deployment.
- On crash. Plain syslog has no application acknowledgement. Frames already in the kernel socket buffer or Bower's read buffer, and a frame waiting for queue space, are lost unless the sender retransmits; everything enqueued before the crash is durable. TCP/TLS with a sender-side disk queue reduces exposure but does not prove end-to-end lossless delivery. Qualify the actual sender with the delivery-reliability guide.
- TLS 1.3 session tickets. .NET on OpenSSL sends two TLS 1.3 session
tickets after every handshake and offers no setting to stop them. rsyslog never reads
them, which hides Bower's half-close from rsyslog's pre-send check. Give the service
OPENSSL_CONFwithNumTickets = 0, or pin rsyslog to TLS 1.2; see the guide.
RELP (acknowledged delivery)
protocol: relp speaks the Reliable Event Logging Protocol as rsyslog omrelp and librelp
do, optionally over TLS with the same certificate, client CA, revocation-list and hot
reload settings as tls. It closes the gap plain syslog leaves: Bower answers each
syslog transaction 200 OK only after a final durable decision, so the sender keeps
every unanswered message in its window and resends it after a reconnect.
- Answered
200 OK: durably enqueued, already in the queue (duplicate), suppressed, not selected, rejected by policy, not routed, malformed, or oversized. These are final: the same bytes would get the same decision. - Never answered: a full queue or gated intake (
Unavailable), a storage error, or a session whose client certificate stopped being trusted. A storage error or lost trust ends the session; the sender resends after it reconnects. - Bounded backpressure. A message the queue keeps refusing is retried every second for
up to 30 seconds (
RelpBusyTimeout, shorter thanomrelp's 90-second response timeout); then Bower sendsservercloseand ends the session without answering. Before closing it reads away what the sender still sends (bounded: until the sender closes or is quiet for 5 seconds,frameTimeoutSeconds, or 16 MiB), so the close is a FIN rather than a reset over unread bytes. The sender keeps the message and its window and reconnects;relpBusyClosescounts this, and each refused attempt also counts inretries. Bower never answers a refused message with an error code, because librelp drops a message from its resend list when it receives a non-200 response. - In order, one at a time. The next frame is read only after the current one is answered, so the sender's window (rsyslog default 128) bounds what is in flight.
- Strict protocol. TXNR and DATALEN are 1–9 digits, the command 1–32 letters, each
followed by exactly one space (anything else, such as
1Xopenor2:syslog, ends the session), and a zero DATALEN is followed directly by LF. The first frame must beopen(transaction 1, offeringrelp_versionandcommands=syslog; offers are bounded to 4 KiB, 16 offers of 16 values); transactions must follow 1, 2, 3 … and wrap after 999,999,999. Any other command, a secondopen, a clientrsp, a bad trailer or an out-of-sequence number ends the session without an answer (framingErrors). Anopenwithoutsyslogis refused with500and the session ends. - Oversized messages. A
syslogframe whose data exceedsmaximumMessageBytes+ 2 (a trailing CR LF is removed) is read past without buffering (any declared length up to 999,999,999 bytes; memory stays one fixed buffer and the whole frame must arrive withinframeTimeoutSeconds), answered200 OKand counted inoversized, so a sender is never stuck resending it. An oversized non-syslogframe ends the session. Keep the sender'smaxMessageSizeat or belowmaximumMessageBytesso nothing is dropped this way. - Stop and idle. On stop (and at a non-zero idle timeout) Bower sends the RELP
serverclosehint, half-closes, then keeps reading and enqueuing frames the sender wrote before it saw the hint (unanswered; the sender resends them later and they deduplicate) until the sender closes or is quiet for 5 seconds. While closing, a frame the queue refuses gets one attempt only, then Bower stops processing and closes (the sender resends it). A clientcloseis answered once every earlier transaction is answered. - Crash. Frames written but not committed when Bower is killed are unanswered, so the sender resends them; committed but unanswered frames come back as duplicates.
A RELP sender still loses its in-memory window if the sender is killed or loses power before resending; a disk-assisted action queue holds only messages not yet handed to the RELP window. See delivery reliability for measured results, sender configuration and deployment.
Monitoring
GET /v1/status (token required) has a syslog array, one entry per listener, with
counters since start: activeConnections, connectionsAccepted,
connectionsRejected, sendersDenied, tlsHandshakeFailures, framesReceived,
bytesReceived, queued, duplicates, suppressed, notRouted, notSelected,
rejected, parserErrors, malformed, oversized, framingErrors, timeouts,
dropped (UDP and backpressure), retries (TCP waits for queue space),
submitFailures, receiveErrors (UDP socket receive errors; the listener backs off
100 ms to 1 s and keeps listening), halfCloses (connections Bower half-closed at an
idle timeout or stop), drainedFrames (frames received after that half-close),
peerResets (observed resets, including a reset before connection setup, which still frees
its connection slot; best effort), relpAcknowledged (RELP transactions answered
200 OK), relpServerCloses (RELP sessions Bower ended with serverclose: stop, idle or
busy), relpBusyCloses (RELP sessions ended because the queue kept refusing) and lastFrameAt. TLS and platform error reporting can prevent reset
classification. Do not treat a zero count as proof of no resets.
timeouts
counts idle-timeout expiries and frames that missed frameTimeoutSeconds. TLS listeners
add tlsReloads, tlsReloadFailures, tlsRevoked (handshakes refused by revocation),
tlsSessionsClosed (open sessions closed after re-validation),
serverCertificateNotAfter, clientCertificateAuthorityNotAfter (earliest trusted
CA) and clientRevocationListNextUpdate (earliest list); the dates are null for other
listeners. They are metadata only and counters reset on restart.
Selected syslog events also appear in the pipeline and intake counters like any other
source.
Silence detection works through the pack's application: list the routed parser's
applicationName/environment in BOWER_SOURCE_EXPECTATIONS and a silent syslog
sender degrades /health once its allowance passes.
Residual risks
- UDP senders can be spoofed; the allow-list checks only the source address. Prefer TLS with client certificates beyond loopback.
- Hostname and app-name are claimed by the sender. Without a per-route
allowedSenders, any allowed sender can reach any route (and its pack parser) by claiming its hostname or app-name; the parser filter, redaction and policy still apply. - An expired server certificate fails every new handshake until a valid one is in
place (it is picked up without a restart). Alert on
serverCertificateNotAfter. - A refused client certificate loses frames at the sender: rsyslog 8.2204 (
ossl, TLS 1.2 or 1.3) writes one frame into each refused session and drops it. Rotate client CAs and certificates with overlap, and alert ontlsHandshakeFailures,tlsRevokedandclientRevocationListNextUpdate. - Plain TCP beyond loopback (with the override) is unauthenticated and unencrypted.
- A sender allowed to connect can occupy up to
maximumConnectionsslots. With the defaultidleTimeoutSeconds: 0a healthy idle connection keeps its slot; keepalive closes dead ones after about 17 minutes. Set a non-zero idle timeout only for senders that read their sockets (for example syslog-ng). - Lost frames on crash, as above.
Not verified here
Interoperability tests against a real sender daemon in a container are not part of the automated suite; the RFC example vectors, framing and TLS tests run against in-process clients.
The credential rotation campaign ran rsyslog 8.2204 over mutual TLS on Amazon Linux 2023: hot server-certificate rotation lost no frames and caused no reconnect; a collector restart under load lost one frame.
The manual native AWS pilot uses nftables and rsyslog over mutual TLS. It retains source-loss findings from both idle connection closure and per-event reconnect configurations. The delivery-reliability guide documents an interrupted follow-up campaign and a proposed sender and collector configuration; its results and evidence export are incomplete. A passing measured traffic window does not establish durable application acknowledgements or qualify another appliance's buffering and replay behavior.