How to Detect Suspicious Activity in AWS CloudTrail Logs

How AWS detection works in plain terms

Think of AWS detection like a building security system. CloudTrail is the security camera that records every door opening and closing. CloudWatch metric filters are motion sensors you place on specific doors (the server room, the CEO’s office). GuardDuty is the trained security guard watching all the camera feeds at once, spotting suspicious behavior you might not think to watch for. And Security Hub is the central monitor room where all the alerts converge.

Here is the actual pipeline those pieces form:

1. CloudTrail records activity. Every API call (creating a resource, changing a security group, assuming a role) becomes a log event with a timestamp, identity, source IP, and outcome.
2. Logs flow to CloudWatch and S3. When you send CloudTrail logs to CloudWatch Logs, you can apply metric filters and run real-time queries. Logs in S3 give you long-term archival and Athena querying.
3. Metric filters and EventBridge catch known-bad patterns. You define the exact events you care about (root login, CloudTrail disabled, IAM policy change) and CloudWatch alarms fire when they appear.
4. GuardDuty detects suspicious patterns automatically. It analyzes CloudTrail, VPC Flow Logs, and DNS logs using machine learning and threat intelligence. It catches things metric filters cannot, like access keys used from a TOR exit node or an unusual burst of API calls.
5. Security Hub aggregates everything. Security Hub pulls findings from GuardDuty, Config, Inspector, and your custom integrations into one dashboard.

Why this matters

AWS accounts are compromised regularly through leaked access keys, misconfigured roles, phished developer credentials, or supply chain attacks. The time between initial compromise and significant damage is often hours, not days. Without detection, you may not discover a breach until you receive the bill for thousands of cryptocurrency miners.

The good news: attackers must make API calls to accomplish anything. Creating resources, accessing data, setting up backdoor users, all of it goes through the AWS API and all of it lands in CloudTrail. The patterns are predictable: privilege escalation, log tampering, resource creation in unusual regions, and risky network changes. With the right alerts, these patterns trigger a notification before real damage occurs.

Before reading further, make sure you understand how CloudTrail logs API calls and what types of events it captures.

What to watch for

Not all API calls are equally suspicious. Here are the highest-priority events, grouped by attack pattern.

Authentication and identity events

• Root account console login. The root user should almost never be used. Any login by root is a high-priority alert.
• Console login without MFA. Indicates MFA is not enforced or was bypassed.
• Multiple failed login attempts. Credential stuffing or brute force attempt.
• Console login from unusual IP or country. Possible credential compromise. Compare against your team’s normal locations.

IAM privilege changes

Privilege escalation is one of the first things an attacker does after gaining initial access. These events deserve immediate attention:

• iam:CreateUser. New user created, especially outside normal hours or outside your CI/CD pipeline.
• iam:CreateAccessKey. New access key created for any user. Attackers create keys for persistence.
• iam:AttachRolePolicy or iam:AttachUserPolicy with AdministratorAccess. Direct privilege escalation. Roles and users should follow least privilege.
• iam:UpdateAssumeRolePolicy. Trust policy modification that could enable unauthorized role assumption from an external account.

Logging and security service tampering

• cloudtrail:DeleteTrail or cloudtrail:StopLogging. Attacker disabling the audit trail.
• guardduty:DeleteDetector or guardduty:StopMonitoringMembers. Disabling threat detection.
• config:DeleteDeliveryChannel. Disabling AWS Config recording.

Network exposure changes

• ec2:AuthorizeSecurityGroupIngress with large CIDR blocks (e.g., 0.0.0.0/0). Opens services to the entire internet.
• ec2:CreateVpc or ec2:CreateInternetGateway in unexpected regions. Could be staging infrastructure for data exfiltration.
• ec2:CreateKeyPair. New SSH key pair created, possibly for persistent access.

Unusual region activity

• EC2 instances, Lambda functions, or databases created in regions you do not use. A strong indicator of crypto mining or staging infrastructure.
• Large EC2 instance types created (p3, p4, g5 families). These GPU instances are expensive and are commonly spun up for unauthorized mining.

CloudWatch metric filters and alarms

Think of CloudWatch metric filters as tripwires. You place them across specific doorways in your CloudTrail log stream, and every time a matching event crosses that wire, a counter increments. When the counter crosses a threshold, a CloudWatch alarm fires and notifies you via SNS. This approach works best for explicit known-bad event patterns you can define in advance and never want to miss.

Example: alert on root account usage

Root account activity is almost always unexpected. This filter catches any API call made by the root user:

# Step 1: Create a metric filter for root usage
aws logs put-metric-filter \
  --log-group-name CloudTrail/DefaultLogGroup \
  --filter-name RootAccountUsage \
  --filter-pattern '{ $.userIdentity.type = "Root" && $.userIdentity.invokedBy NOT EXISTS && $.eventType != "AwsServiceEvent" }' \
  --metric-transformations \
    metricName=RootAccountUsageCount,metricNamespace=CloudTrailAlerts,metricValue=1

# Step 2: Create an alarm that triggers on any root usage
aws cloudwatch put-metric-alarm \
  --alarm-name RootAccountUsageAlarm \
  --alarm-description "Root account was used" \
  --metric-name RootAccountUsageCount \
  --namespace CloudTrailAlerts \
  --statistic Sum \
  --period 300 \
  --threshold 1 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --evaluation-periods 1 \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:SecurityAlerts

Example: alert on IAM policy changes

This filter catches any IAM policy being created, deleted, attached, or detached. These are the core events in a privilege escalation attempt:

aws logs put-metric-filter \
  --log-group-name CloudTrail/DefaultLogGroup \
  --filter-name IAMPolicyChanges \
  --filter-pattern '{($.eventName=DeleteGroupPolicy)||($.eventName=DeleteRolePolicy)||($.eventName=DeleteUserPolicy)||($.eventName=PutGroupPolicy)||($.eventName=PutRolePolicy)||($.eventName=PutUserPolicy)||($.eventName=CreatePolicy)||($.eventName=DeletePolicy)||($.eventName=CreatePolicyVersion)||($.eventName=DeletePolicyVersion)||($.eventName=SetDefaultPolicyVersion)||($.eventName=AttachRolePolicy)||($.eventName=DetachRolePolicy)||($.eventName=AttachUserPolicy)||($.eventName=DetachUserPolicy)||($.eventName=AttachGroupPolicy)||($.eventName=DetachGroupPolicy)}' \
  --metric-transformations \
    metricName=IAMPolicyChangeCount,metricNamespace=CloudTrailAlerts,metricValue=1

Example: alert on CloudTrail being disabled

If someone modifies or disables CloudTrail itself, you need to know immediately. This is the anti-tamper alarm for your audit trail:

aws logs put-metric-filter \
  --log-group-name CloudTrail/DefaultLogGroup \
  --filter-name CloudTrailChanges \
  --filter-pattern '{($.eventName=CreateTrail)||($.eventName=UpdateTrail)||($.eventName=DeleteTrail)||($.eventName=StartLogging)||($.eventName=StopLogging)}' \
  --metric-transformations \
    metricName=CloudTrailChangeCount,metricNamespace=CloudTrailAlerts,metricValue=1

For more on how metric filters work beyond security use cases, see the log-based metrics guide.

CloudTrail Insights

CloudTrail Insights detects unusual patterns in your management event activity automatically. Instead of you defining the exact event to watch for (like metric filters do), Insights builds a baseline of your account’s normal API call rates and error rates over time, then flags deviations.

What Insights detects

• Unusual API call rate. A sudden spike in RunInstances, CreateUser, or any management API that normally has low volume. This can catch automated attacks or runaway scripts.
• Unusual API error rate. A burst of AccessDenied errors on sensitive APIs like AssumeRole or GetSecretValue. This often indicates an attacker probing your account’s permissions.

How Insights differs from metric filters

Metric filters match specific patterns you define in advance. If the attacker uses an API call you did not create a filter for, you miss it. Insights catches anomalies in volume and error rate across all management APIs, including ones you did not anticipate. The tradeoff is that Insights requires enough baseline data to establish “normal,” and it may generate false positives during legitimate traffic spikes like deployments.

You can enable Insights events on any trail in the CloudTrail console or via CLI. Insights events appear alongside your management events and can also be sent to EventBridge for automated routing.

AWS GuardDuty

AWS GuardDuty is a managed threat detection service that continuously analyzes CloudTrail events, VPC Flow Logs, DNS logs, and (optionally) S3 access logs. It uses machine learning and AWS threat intelligence to identify suspicious patterns automatically, with no log parsing or filter configuration required.

GuardDuty catches things that metric filters cannot: access keys used from a known malicious IP, API calls originating from a TOR exit node, credential use patterns that match known attack toolkits, and data exfiltration patterns visible only in network flow data.

Finding categories

• Credential Access. Access keys used from TOR exit nodes, unusual geographic locations, or rapid API call bursts.
• Privilege Escalation. IAM user attaching AdministratorAccess to themselves.
• Persistence. New IAM user or access key created, trust policy modified.
• Defense Evasion. CloudTrail logging disabled, GuardDuty disabling attempts.
• Discovery. Unusual volume of DescribeInstances or ListBuckets calls (reconnaissance).
• Exfiltration. Large data transfers to unusual IP addresses.

Enable GuardDuty

aws guardduty create-detector \
  --enable \
  --finding-publishing-frequency FIFTEEN_MINUTES

GuardDuty is a regional service, so enable it in every region you use. For multi-account setups, enable it at the organization level via a delegated administrator account, which automatically covers all member accounts.

Routing GuardDuty findings

GuardDuty publishes findings to EventBridge automatically. You can create EventBridge rules that route high-severity findings to SNS (for paging), Lambda (for automated response), or Security Hub (for aggregation). This means GuardDuty findings can feed into the same alert pipeline as your CloudWatch alarms.

When to use which approach

Each detection tool solves a different problem. Here is when to reach for each one.

CloudWatch metric filters

Use metric filters when you know exactly what event you want to catch and need a near-real-time alarm on it. Metric filters are best for high-signal, well-defined events: root login, CloudTrail disabled, IAM policy attached. You write the filter pattern, and you control the alarm threshold and notification target.

CloudTrail Insights

Use Insights when you want anomaly detection without defining every pattern in advance. Insights catches unexpected spikes: a sudden surge of RunInstances calls or a burst of AccessDenied errors that your predefined filters would miss. It complements metric filters rather than replacing them.

GuardDuty

Use GuardDuty when you want managed threat intelligence and behavioral analysis. GuardDuty detects threats you cannot easily write filters for, like access keys used from a known malicious IP or unusual geographic patterns. It also analyzes VPC Flow Logs and DNS data, which metric filters and Insights do not touch. Enable GuardDuty in all regions regardless of what other detection you have.

Using more than one

Use all three. They are complementary, not competing. A real compromise often produces signals across multiple layers: a metric filter catches the CreateUser event, GuardDuty flags the unusual source IP, and Insights detects the anomalous API call rate. Layered detection reduces the chance that any single attacker technique goes unnoticed.

First setup checklist

If you are starting from scratch, follow this sequence:

1. Create a CloudTrail trail that sends management events to S3 and CloudWatch Logs. Enable it as an organization trail if you have multiple accounts. See the CloudTrail overview for setup details.
2. Enable GuardDuty in every region you use. In a multi-account setup, use a delegated administrator.
3. Add three high-signal metric filters: root account usage, CloudTrail changes, and IAM policy changes (examples above).
4. Create an SNS topic for security alerts and subscribe your team’s email or PagerDuty integration.
5. Route GuardDuty findings to EventBridge and forward high-severity findings to the same SNS topic.
6. Enable CloudTrail Insights for unusual API call rates and error rates.
7. Test your alert pipeline by triggering a low-severity alarm deliberately. Verify the notification arrives. Repeat this monthly.

This setup takes less than an hour and gives you meaningful coverage for the most common attack patterns.

Example CloudWatch Logs Insights queries

Once CloudTrail is sending logs to CloudWatch Logs, you can use Logs Insights for ad-hoc investigation. These queries are useful during triage when you need to quickly understand what happened.

Find all actions by a specific user in the last 24 hours

Use this when investigating a potentially compromised user account or access key:

fields eventTime, eventName, sourceIPAddress, errorCode
| filter userIdentity.userName = "alice"
| sort eventTime desc
| limit 100

Find all failed API calls

A burst of failed calls, especially AccessDenied errors, can indicate an attacker probing for permissions:

fields eventTime, userIdentity.arn, eventName, errorCode, errorMessage
| filter ispresent(errorCode)
| stats count(*) as errorCount by eventName, errorCode
| sort errorCount desc

Find security group changes

Use this to investigate whether network exposure changed during a suspected incident:

fields eventTime, userIdentity.arn, eventName, requestParameters.groupId
| filter eventName in ["AuthorizeSecurityGroupIngress", "RevokeSecurityGroupIngress", "CreateSecurityGroup", "DeleteSecurityGroup"]
| sort eventTime desc

These queries run in the CloudWatch console under Logs Insights, or via CLI using aws logs start-query. For deeper historical analysis beyond your CloudWatch Logs retention period, use Athena queries against the CloudTrail S3 data.

What to do after an alert

An alert is only useful if you have a clear next step. When a detection fires, work through these questions:

1. Who? Identify the IAM principal (user, role, or temporary credentials). Check the userIdentity field in CloudTrail.
2. When? Note the event timestamp and check for related events in the same time window.
3. From where? Check the sourceIPAddress. Is this a known team IP, a CI/CD system, or an unexpected location?
4. What was affected? Identify the resource and the specific API action.
5. Did it succeed? Check the errorCode field. A failed attempt is still worth investigating but changes the urgency.
6. Should credentials be disabled? If an access key or role is compromised, disable or rotate the key immediately. Do not delete it because you may need it for forensic correlation.
7. Preserve evidence. Do not delete CloudTrail logs, terminate suspect instances, or remove IAM users before capturing the relevant log data and resource state.

For a complete incident response process including isolation and post-mortem, see the incident response guide.