Quick start: AWS with the CloudWatch agent
Use this when your servers already run the Amazon CloudWatch agent. Each server writes Bower security events to a local file, the agent ships that file to CloudWatch Logs, and one Bower deployment in the account forwards the events to Microsoft Sentinel. You don't install Bower on every server.
The forwarding half (subscription filter, Firehose, S3, SQS and the bower-cloud agent) is
the existing AWS security signal stack. This page covers the part on
each server.
Before you begin
- A Bower collector delivering to Sentinel, and the
bower-cloudagent able to reach it. See AWS security signal: run the agent. - The CloudWatch agent installed on each server, with an instance profile (EC2) or
credentials profile (on-premises) that can write to the log group. AWS's
CloudWatchAgentServerPolicyworks, but it allows writing to any log group; the least-privilege policy below limits the agent to/bower/app-events.
Each block below says where it runs: on each server, on an admin workstation (with
AWS CLI credentials for the account) or on the bower-cloud host.
Get the files
Download the examples from the Bower release you deploy, not from main, so the files match
the collector and agent you run. On each server:
BOWER_REF=vX.Y.Z # the Bower release tag you deploy
base="https://raw.githubusercontent.com/jusso-dev/Bower/$BOWER_REF/examples/cloud-agents"
curl -fsSL "$base/emit_event.py" -o /tmp/emit_event.py
curl -fsSL "$base/aws/amazon-cloudwatch-agent.json" -o /tmp/bower-events.json
curl -fsSL "$base/aws/logrotate-bower-events" -o /tmp/logrotate-bower-events
On the admin workstation you also need the stack template:
curl -fsSLO "https://raw.githubusercontent.com/jusso-dev/Bower/$BOWER_REF/deploy/aws/bower-aws-security.yaml"
1. Write events to a file
Each line is one complete Bower event envelope: UTF-8 JSON, with no line breaks inside it, written with a single append. The minimum is:
{"schemaVersion":"1.0.0","eventId":"7f9c2b1e-5d4a-4c3b-9a8e-1f2d3c4b5a69","timeGenerated":"2026-11-10T01:02:03.456Z","eventCategory":"authentication","eventType":"authentication_failure","eventAction":"authentication.login","eventResult":"failure","eventSeverity":"medium","application":{"name":"orders-api","environment":"production"},"actor":{"type":"human","username":"alice@example.com"},"source":{"ipAddress":"203.0.113.24"}}
eventIdmust be unique per event and stay the same if the line is written again.- Write each line with one
writecall on a file opened for append (O_APPEND). On a local Linux file system, writers then never interleave inside a line, whatever its length. A buffered writer (a defaultprintto a file, most logging handlers) can split a line, and NFS doesn't give the same guarantee. - A short write (for example a full disk) leaves a line with no line break, and the next
event is glued to it: both are then invalid and the forwarder drops them. Before writing,
check whether the file ends with a line break and start with one if it doesn't, as
emit_event.pydoes. That limits the damage to the fragment. - Keep each line well under 1 MB. CloudWatch Logs takes log events of up to 1 MB (CloudWatch Logs quotas, "Event size"), and the forwarder drops any line over 1 MiB (1,048,576 bytes). AWS doesn't document what the CloudWatch agent does with a longer line; whether it cuts it short or drops it, the event never reaches Sentinel. A security event of a few KB is normal.
- Only lines that contain
schemaVersionleave CloudWatch Logs, so ordinary application logging can share the file, but a separate file is easier to reason about. - To build events in an application, use the framework helpers and append the result instead of posting it.
Give the application a group that can write the file, and keep everyone else out. On each
server, replacing orders-api with the account your application runs as:
sudo groupadd --system bower-events
sudo usermod -a -G bower-events orders-api
sudo install -d -m 2770 -o root -g bower-events /var/log/bower
# touch never truncates, so this is safe to run again on a file that already holds events.
sudo touch /var/log/bower/events.jsonl
sudo chown root:bower-events /var/log/bower/events.jsonl
sudo chmod 0660 /var/log/bower/events.jsonl
The setgid directory keeps new files in the bower-events group. The CloudWatch agent runs
as root by default and can read the file. If your agent configuration sets run_as_user
(for example cwagent), add that account to bower-events too. A running process keeps the
groups it started with, so restart the application, and the agent if you added its account
(sudo systemctl restart amazon-cloudwatch-agent).
Rotate the file
Install the logrotate file
(logrotate-bower-events):
sudo install -m 0644 -o root -g root /tmp/logrotate-bower-events /etc/logrotate.d/bower-events
sudo logrotate -d /etc/logrotate.d/bower-events # dry run: shows what it would do
It renames the file to events.jsonl.1 and creates a new events.jsonl with the same owner,
group and mode. Its su root bower-events line is required: without it logrotate refuses to
rotate a file in a group-writable directory ("parent directory has insecure permissions") and
the file grows without limit.
What your application must do at rotation depends on how it writes:
- Opens the file for each event (as
emit_event.pydoes): nothing. The next event goes to the new file. - Keeps the file open: it goes on appending to the renamed
events.jsonl.1until it reopens the path. Make it reopen on a signal and send that signal from apostrotateblock in the logrotate file (the example shows where). Don't switch tocopytruncate: lines written between the copy and the truncate are lost.
On the agent side, AWS documents that a rotation which changes the file's inode (rename and create, as here) makes the agent read the new file from the beginning, and that the agent keeps its read position in a state file so a restart doesn't send lines again (Log duplicate prevention and the state file).
Measured with a live agent (CloudWatch agent 1.300073.2 on Amazon Linux 2023, 11-11-2026):
- The new file is picked up and read from the start: 20 of 20 lines written after rotation arrived, none twice.
- Lines written to the renamed
events.jsonl.1after the rename are never shipped (0 of 10). The agent logs "Stopping tail as file no longer exists" for the path the moment it is renamed and only follows the path from then on. - The last lines before a rotation can be lost too. With 10 lines written and the file
rotated straight away, 8 arrived. The agent had not read the last 2 when it stopped
following the renamed file.
delaycompresskeeps that file on disk, but the agent never goes back to it. With the same 10 lines and 30 seconds of quiet before the rotation, 10 of 10 arrived.
So any line in the renamed file that the agent hasn't read yet is lost, with no error. To keep the window small:
- Writers that keep the file open must reopen on rotation (the
postrotatesignal above), as fast as they can. - Rotate when writes are quiet, and not more often than you need (
dailywithmaxsize, as in the example). - Watch for the gap. The rows that arrive in Sentinel carry
Origin.logStream, anddelivery-sequence-gaps.kqlcovers losses after the collector. A loss here happens before Bower sees the line, so only a count kept by the application can show it. - If a line is ever shipped twice, the collector deduplicates it by
eventId.
examples/cloud-agents/aws/test_logrotate.sh checks the logrotate side in a container
(Ubuntu 24.04 and Amazon Linux 2023), and CI runs it.
Several applications on one server
Give each application its own group and directory. Members of one shared group can delete,
replace or append to each other's files, so a compromised application could write events in
another one's name. The sticky bit (3770) stops deleting and replacing, but not appending,
so it isn't enough. test_logrotate.sh checks both layouts. On each server:
sudo install -d -m 0755 -o root -g root /var/log/bower
for app in orders-api billing-api; do
sudo groupadd --system "bower-$app"
sudo usermod -a -G "bower-$app" "$app"
sudo install -d -m 2770 -o root -g "bower-$app" "/var/log/bower/$app"
sudo touch "/var/log/bower/$app/events.jsonl"
sudo chown "root:bower-$app" "/var/log/bower/$app/events.jsonl"
sudo chmod 0660 "/var/log/bower/$app/events.jsonl"
sed -e "s|/var/log/bower/events.jsonl|/var/log/bower/$app/events.jsonl|" \
-e "s|bower-events|bower-$app|g" /tmp/logrotate-bower-events |
sudo tee "/etc/logrotate.d/bower-$app" >/dev/null
done
Then add one collect_list entry per file in step 2.
In this path CloudWatch Logs and the S3 bucket hold the event as the server wrote it.
Bower's privacy profile runs later, in the collector. Don't write passwords, tokens or
session identifiers into the file; the framework helpers already drop secret-named fields.
Keep the log group and bucket retention short (the stack's AppLogRetentionDays sets the
bucket expiry), and limit who can read the copies:
- Grant
logs:GetLogEvents,logs:FilterLogEvents,logs:StartQueryandlogs:StartLiveTailon/bower/app-eventsonly to the people who investigate these events. - Grant
s3:GetObjecton the app log bucket only to thebower-cloudagent (the stack'sConsumerPolicy) and the same investigators. The stack encrypts the bucket with S3-managed keys (AES256), so anyone withs3:GetObjectcan read the objects. - Consider a KMS key for the log group
(Encrypt log data in CloudWatch Logs using AWS KMS),
so reading the events also needs
kms:Decrypt.
2. Configure the CloudWatch agent
On the admin workstation, create the log group once, with a short retention, in the Region where you deploy the Bower stack in step 3 (Sydney here):
export AWS_REGION=ap-southeast-2
aws logs create-log-group --region "$AWS_REGION" --log-group-name /bower/app-events
aws logs put-retention-policy --region "$AWS_REGION" --log-group-name /bower/app-events --retention-in-days 7
If the group already exists, create-log-group fails with ResourceAlreadyExistsException;
that's harmless, and put-retention-policy still sets the retention.
The agent on each server must send to the same Region. On EC2 it uses the instance's Region
by default. On a server outside AWS it uses the region in the AmazonCloudWatchAgent
profile of its AWS config file.
Add the Bower file to the agent configuration. If the agent already has a configuration,
merge this collect_list entry into it rather than replacing the file:
{
"logs": {
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "/var/log/bower/events.jsonl",
"log_group_name": "/bower/app-events",
"log_stream_name": "{instance_id}",
"encoding": "utf-8",
"filters": [
{
"type": "include",
"expression": "\"schemaVersion\""
}
]
}
]
}
}
}
}
-
Stream name.
{instance_id}is the EC2 instance ID, so each instance writes its own stream. AWS doesn't document what it becomes on a server outside EC2, so on-premises set"log_stream_name": "{local_hostname}"(the host name from the server's own network configuration;{hostname}comes from EC2 metadata), and make sure host names are unique, or two servers share one stream (agent configuration file reference). -
Several applications. Add one entry per file, each with its own stream name:
{"file_path": "/var/log/bower/billing-api/events.jsonl","log_group_name": "/bower/app-events","log_stream_name": "{instance_id}/billing-api","encoding": "utf-8","filters": [{ "type": "include", "expression": "\"schemaVersion\"" }]} -
Event time. Without
timestamp_formatthe agent records the time it read the line ("If you omit this field, the current time is used"), so the CloudWatch console shows read time. Bower and Sentinel usetimeGeneratedfrom the event itself.
Apply it on each server with append-config, which keeps the agent's existing sources
(fetch-config would replace them). Use -m ec2 on EC2 and -m onPremise on servers outside
EC2.
Linux (amazon-cloudwatch-agent.json):
sudo install -m 0644 /tmp/bower-events.json /opt/aws/amazon-cloudwatch-agent/etc/bower-events.json
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
-a append-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/bower-events.json
Windows
CI runs the commands in this section as written on Windows Server 2025
(test_windows_event_folder.ps1).
It signs in as a second standard user, who is refused both reading the file and creating
one in the folder, and it runs the rotation task as SYSTEM. It doesn't run the
CloudWatch agent itself. Run the permission check below on your own servers too.
The event file is C:\ProgramData\BowerEvents\events.jsonl. A folder made under
C:\ProgramData inherits an ACL that lets every local user create files (the Bower installer
breaks inheritance on its own folder for the same reason), so lock the folder down before the
application writes to it. Without that, any local user could read the unredacted events, or
create events.jsonl first and own it.
In an elevated PowerShell on each server, replacing the application account (a domain
account, NT SERVICE\<service> or IIS AppPool\<pool>):
$dir = 'C:\ProgramData\BowerEvents'
$app = 'CONTOSO\svc-orders' # the account the application runs as
New-Item -ItemType Directory -Force -Path $dir | Out-Null
# Remove inherited entries, then grant SYSTEM (S-1-5-18) and Administrators (S-1-5-32-544)
# full control and the application Modify, for the folder and everything created in it.
icacls $dir /inheritance:r /grant:r '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' "${app}:(OI)(CI)M"
# Create the file now, so no one else can create it first. It inherits the folder's entries.
if (-not (Test-Path "$dir\events.jsonl")) { New-Item -ItemType File -Path "$dir\events.jsonl" | Out-Null }
icacls $dir
The last command should list only NT AUTHORITY\SYSTEM, BUILTIN\Administrators and the
application account. If the folder already existed with explicit entries (for example a
grant to Users), they survive /inheritance:r: remove them with icacls $dir /remove:g <name>.
Then add the file to the agent (amazon-cloudwatch-agent.windows.json), with -m ec2 on EC2
or -m onPremise outside EC2:
$base = "https://raw.githubusercontent.com/jusso-dev/Bower/$env:BOWER_REF/examples/cloud-agents/aws"
Invoke-WebRequest "$base/amazon-cloudwatch-agent.windows.json" -OutFile "$dir\bower-events.json"
& "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" `
-a append-config -m ec2 -s -c "file:$dir\bower-events.json"
Set $env:BOWER_REF to the release tag first. The agent runs as Local System, so it can read
the file.
Check the permissions while signed in as a standard user who isn't the application account. Both commands must fail with "Access to the path is denied":
Get-Content C:\ProgramData\BowerEvents\events.jsonl
New-Item -ItemType File -Path C:\ProgramData\BowerEvents\other.jsonl
Rotate the file. Windows has no logrotate.
rotate-bower-events.ps1
renames events.jsonl to events.jsonl.1 (keeping seven), and creates a new empty file,
which inherits the folder's entries. Run it daily as SYSTEM from a scheduled task. Keep the
script in a folder only administrators can write to, because SYSTEM runs it:
$tool = 'C:\Program Files\BowerEvents'
New-Item -ItemType Directory -Force -Path $tool | Out-Null
Invoke-WebRequest "$base/rotate-bower-events.ps1" -OutFile "$tool\rotate-bower-events.ps1"
$action = New-ScheduledTaskAction -Execute 'powershell.exe' `
-Argument "-NoProfile -File `"$tool\rotate-bower-events.ps1`""
$trigger = New-ScheduledTaskTrigger -Daily -At 00:05
Register-ScheduledTask -TaskName 'Bower event file rotation' -Action $action -Trigger $trigger `
-User 'NT AUTHORITY\SYSTEM' -RunLevel Highest
The application must open the file for each event, or reopen it after rotation. If a process holds events.jsonl open without delete sharing, the rename
fails and the script stops without changing anything; it tries again the next day. If the
task history shows it failing every day, this approach doesn't suit your application, and
the file keeps growing until you rotate it another way.
Measured with a live agent on Windows Server 2022 (CloudWatch agent from AWS's current MSI, 11-11-2026), using this script and the guide's config:
- After rotation the agent follows the new file: 20 of 20 lines arrived, none twice.
- Lines written to the renamed
events.jsonl.1after the rename are never shipped (0 of 10), as on Linux. - 10 lines then an immediate second rotation: 10 of 10 arrived. So on Windows too, any line written to the old file after the rename is lost. The application must not keep the file open across the rotation, which the script's held-open check already enforces by refusing to rename.
Least-privilege policy for the agent
Attach this to the instance profile role (EC2) or the credentials the agent uses
(on-premises) instead of CloudWatchAgentServerPolicy, with your Region and account. It
covers the Bower file only; if the same agent ships other logs or metrics, keep what those
need. CreateLogStream and PutLogEvents act on the log stream, DescribeLogStreams on the
group
(actions, resources and condition keys for CloudWatch Logs).
Tested with a live agent (CloudWatch agent 1.300073.2, Amazon Linux 2023, 11-11-2026): with only
this policy on the instance role, the agent created the instance's stream and shipped every
event.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "FindBowerStreams",
"Effect": "Allow",
"Action": "logs:DescribeLogStreams",
"Resource": "arn:aws:logs:ap-southeast-2:111122223333:log-group:/bower/app-events"
},
{
"Sid": "WriteBowerEvents",
"Effect": "Allow",
"Action": ["logs:CreateLogStream", "logs:PutLogEvents"],
"Resource": "arn:aws:logs:ap-southeast-2:111122223333:log-group:/bower/app-events:log-stream:*"
}
]
}
3. Forward the log group to Bower
On the admin workstation, deploy (or update) the AWS stack with the log group name. It adds
the subscription filter that only sends lines containing schemaVersion:
aws cloudformation deploy --region "$AWS_REGION" \
--template-file bower-aws-security.yaml \
--stack-name bower-aws-security \
--capabilities CAPABILITY_IAM \
--parameter-overrides AppLogBucketName=acme-bower-app-logs AppLogGroupName=/bower/app-events
Then, on the bower-cloud host, allow the new bucket in the agent. It ignores S3
notifications from any bucket not listed in BOWER_AWS_S3_BUCKETS and deletes them as
unsupported, so without this the events reach the queue but never Sentinel. Set the complete
list (buckets already allowed, plus the new one) where Compose reads it, then recreate the
agent, from the directory where you deployed it:
# deploy/docker/compose.cloud.yaml reads BOWER_AWS_S3_BUCKETS from the environment or an .env file.
export BOWER_AWS_S3_BUCKETS=existing-bucket,acme-bower-app-logs
docker compose -f deploy/docker/compose.cloud.yaml up -d
docker compose -f deploy/docker/compose.cloud.yaml exec bower-cloud printenv BOWER_AWS_S3_BUCKETS
A plain BOWER_AWS_S3_BUCKETS=... line in your shell isn't passed to Compose unless it's
exported; an empty value would make the agent drop every app-log notification.
The stack takes one log group. To collect from several applications, point their agents at
the same group with different log_stream_name values, as in step 2.
Trust boundary
The collector sees every event on this path as coming from one producer: the bower-cloud
agent's ingest token, so producer scopes can't tell one
server from another here. That means any server or account that can write to the log group
can send events that claim any application identity, and the collector accepts them. A
compromised low-value server, or any local account in its bower-events group, could write
events in the name of a different application. Policy and redaction still run on them.
What Bower does record is where each event came from. The agent passes on the log group and
stream of every line, and the row's Origin column holds them
(Origin.kind == "cloudwatch-logs", Origin.logGroup, Origin.logStream). With the stream
named after the instance (the example's {instance_id}), each row names the server whose
stream it was written to. The application cannot set this. To turn it on, give the agent its
own entry in the collector's BOWER_INGEST_TOKENS_FILE and list that producer in
BOWER_ORIGIN_PRODUCERS, for example BOWER_ORIGIN_PRODUCERS=bower-cloud. Without it the
column stays empty. See BowerSecurity_CL columns.
Origin names the stream, not the account that wrote to it, so it identifies the server only
as far as each server is limited to its own stream. To keep that blast radius small:
-
Use one log group, and one Bower stack, per trust level, so servers you trust less can't write where your most important applications write.
-
Limit what each server can write. The confirmed facts, from AWS's actions, resources and condition keys for CloudWatch Logs:
logs:CreateLogStreamandlogs:PutLogEventsact on the log-stream resource (arn:aws:logs:<region>:<account>:log-group:<group>:log-stream:<stream>), and the only condition key CloudWatch Logs offers on log groups and streams isaws:ResourceTag/${TagKey}. So the limit goes in the policy'sResource. The simplest form that needs no further checking is one role per server or per group of equally trusted servers, each allowed only its own stream names, for example...:log-stream:orders-api-*, with the agent'slog_stream_nameset to match. -
One shared role can't pin each instance to its own stream. Tested live with one instance role (11-11-2026):
- A policy naming the instance's own stream exactly
(
...:log-stream:i-0abc123...) allowed that stream and denied writes and stream creation for any other. - None of the policy variables that could make that one shared policy worked:
${ec2:InstanceID}and${ec2:SourceInstanceARN}in the stream ARN, and anaws:useridcondition against${logs:LogStreamName}, all denied even the instance's own stream. ${aws:userid}resolves to<role-id>:<instance-id>, and a stream name can't contain:(CloudWatch refused it as an invalid parameter).
So the limit has to be per role: one role, or one policy, per server or per group of equally trusted servers, each naming its own streams. Treat every server that shares a role as able to write any stream that role allows. AWS documents the two values (IAM policy variables, condition keys).
- A policy naming the instance's own stream exactly
(
Limits
-
Firehose buffer. The stack's Firehose stream writes an S3 object every 60 seconds or 5 MB, whichever comes first (
BufferingHintsinbower-aws-security.yaml). -
One S3 object per SQS message. The
bower-cloudagent refuses an object over 32 MB (BOWER_AWS_S3_MAX_OBJECT_MB), over 64 MiB once decompressed, or holding more than 50,000 Bower events. It treats such a message as malformed: it releases it at once, and afterMaxReceiveCountdeliveries (10 by default) SQS moves it to the dead-letter queue, with every event in that object. At a 5 MB buffer this needs an unusual burst, but a server that writes a flood of events can cause it. Fix the cause, then use SQS dead-letter queue redrive; the objects stay in S3 forAppLogRetentionDays. -
Dropped lines. A log event that isn't a usable Bower envelope (not JSON, over 1 MiB, or missing
schemaVersion,eventIdoreventType) is dropped. The subscription filter passes it because it containsschemaVersion. The agent counts these drops in its pass summary (... log line(s) skipped as not a Bower envelope.) and logs a warning per object with the bucket, key and count, never the content:S3 object acme-bower-app-logs/bower/2026/11/10/01/...: 1 of 40 log event(s) skipped as not a usable Bower envelope (not JSON, over 1 MiB, or missing schemaVersion, eventId or eventType); those events are not forwarded.Look on the server for a line that's too long, was written in two pieces or follows a partial line.
4. Check it
Send the test event only now, once the agent and the forwarding are in place, so the check
doesn't depend on whether the agent reads a file it has just been given from the start or
from the end. On a server (--application and --environment set the event's
application, so you can tell servers apart):
sudo -u orders-api python3 /tmp/emit_event.py /var/log/bower/events.jsonl \
--application orders-api --environment production
On Windows, run python emit_event.py C:\ProgramData\BowerEvents\events.jsonl as the
application account (for example with runas /user:CONTOSO\svc-orders). It prints the
eventId. Then:
- On the server:
sudo tail -1 /var/log/bower/events.jsonlshows the event. - In CloudWatch Logs: the event appears in
/bower/app-events. - On the collector:
bower tap --stage decidedshows it once Firehose has delivered (up to about a minute, the stream's buffer interval) and the agent has read the queue. - In Sentinel:
BowerSecurity_CL | where EventId == "<your eventId>".
If step 2 works and step 3 doesn't, check the SQS dead-letter queue and the agent's log; see AWS security signal. Search the log for:
-
S3 notification for bucket {Bucket} ignored; add it to BOWER_AWS_S3_BUCKETS to read it.(warning): the bucket isn't inBOWER_AWS_S3_BUCKETS. The agent deleted the notification, so fixing the list doesn't bring those events back by itself. The objects are still in the bucket underbower/forAppLogRetentionDays. To replay them, after fixing the list, copy each object onto itself so S3 sends a new notification (this is a write to the bucket; run it with an administrator's credentials, not the agent's):aws s3 cp --recursive "s3://acme-bower-app-logs/bower/2026/11/10/" "s3://acme-bower-app-logs/bower/2026/11/10/" \--metadata-directive REPLACE --metadata replayed=trueThe collector deduplicates by
eventId, so replaying an object twice is harmless. -
log event(s) skipped as not a usable Bower envelope(warning): see Limits.