AWS CloudWatch vs CloudTrail
Overview
Two critical factors are critical for any business system. They are monitoring and auditing. A DevOps engineer and a system admin will monitor the production system and will act if any incident occurs. The Security Team will audit the access of every user's authenticity for their system to achieve industry compliance and standards. The two services used for auditing and monitoring systems deployed on the AWS Cloud are AWS Cloudtrail and Amazon Cloudwatch.
What is CloudWatch?
AWS provides infrastructure as a service for customers. So, the customer can deploy their application on AWS EC2 servers, RDS, Lambda, etc. Modern applications use multiple services or components in their architecture to achieve scalability and operational excellence. AWS Cloudwatch helps customers monitor the performance of applications and servers deployed on AWS and on-premise.
Amazon CloudWatch also delivers data and actionable insights for monitoring applications, analyzing and responding to changes in system-wide performance, optimizing resource use, and obtaining a consolidated view of operation health.
How Does It Work?
According to the AWS documentation, Amazon Cloudwatch is a metric repository. i.e., we can configure available metrics for AWS resources such as EC2 and RDS (Relational Database service) and we can get a statistical view of those metrics.
Based on that statistical data, customers view their operational performance for their applications or services on a dashboard. But Amazon Cloudwatch does more than provide a metric repository. As we discussed earlier, nearly 70 AWS services are publishing logs to Cloudwatch logs.
We can get more details by querying this log using log insights features.
Benefits of CloudWatch
- Cloudwatch is natively integrated with the AWS SNS Service for seamless communication. i.e., so we can easily communicate about any occurrence in our infrastructure in real-time by using Cloudwatch Alarm.
- Cloudwatch alarms are used for configuring auto-scaling capabilities for computer resources such as Amazon EC2, ECS, and DynamoDB services.
- In addition to SNS and Auto Scaling capabilities, Cloudwatch metrics are also used to create an incident in System Manager.
- We can use Cloudwatch to set high-resolution alarms to visualize log metrics side by side and take automated actions, troubleshoot issues, and discover insights to optimize our applications and ensure they are running smoothly.
- We can create a custom dashboard of our own with selected metrics of our own choice for our application/project wise in the Amazon Cloudwatch dashboard.
- Container insights, Lambda insights, contributed insights, and application insights made it easy for developers to query data from the ingested logs in Cloudwatch.
- Amazon CloudWatch Logs is also PCI and FedRamp compliant. Data is encrypted at rest and in transit.
- We can also monitor application metrics across AWS accounts.
Examples and Use Cases for Cloudwatch
The common use case for Cloudwatch-EC2 is
By default, Amazon Cloudwatch has a CPU Utilization metric available for all the EC2 instances.
Let us assume the customer production environment runs on one EC2 instance. Due to unexpected traffic, the CPU utilization becomes high. In this case, how can customers know the utilization of production EC2 instances?
It is not possible to check the Amazon Cloudwatch console often. The best practice is that we can create one alarm for the CPU utilization metrics for that particular instance. When the threshold value of 80 percent CPU Utilization is reached, the alarm will be triggered, and messages will be sent to the subscribed SNS Topics associated with the cloudwatch alarm.
Creating an alarm with SNS Topic automates the process of communication of production resource CPU utilization. Similarly, we can create alarms for disc utilization metrics, memory utilization metrics, etc. Likewise, for EC2, we can create an alarm for RDS with the below metrics as well. Free storage space. Freeable memory CPU Utilization
The common use case for Cloudwatch-EC2 with actions
There are two status checks that need to be in the OK state for an Amazon EC2 instance to perform as expected. They are system status checks and instance status checks.
| No | Status check | Description |
|---|---|---|
| 1 | System status check | Underlying hardware system check |
| 2 | Instance status check | Instance layer system check |
For some reason, an Amazon EC2 instance will fall into a 1/2 status check with an instance system check failure. There are many reasons associated with that failure. During that failure, we should reboot the server to get back to the 2/2 status check and be ready to use.
How to automate that process?
The answer is Amazon cloudwatch alarm actions.
For instance, we can create one alarm for system check failure metrics. Whenever this alarm triggers, we can choose one of the default options, such as rebooting the instance. So, whenever an EC2 instance status check failure occurs, it can be automatically recovered by an Amazon Cloudwatch alarm action.
Pricing of CloudWatch
The pricing of Cloudwatch is categorized into two tiers. There is a free tier and a paid tier.
We can use the Amazon Cloud Watch for free. Most AWS services, such as Amazon EC2 and S3, send metrics automatically to Cloudwatch for free.
By default, AWS provides the below-mentioned limits in cloudwatch as a free tier. Some small applications or resources can operate within these free tiers and save costs.
If the quota for the free tier is reached, customers will fall into the paid tier.
| Components | Free Tier | Paid Tier |
|---|---|---|
| Metrics | Basic Monitoring Metrics (at 5-minute frequency),10 Detailed Monitoring Metrics (at 1-minute frequency),1 Million API requests | First 10,000 metrics-0.10,Next 750,000 metrics-0.02 |
| Dashboard | 3 Dashboards for up to 50 metrics per month | $3.00 per dashboard per month |
| Alarms | 10 Alarm metrics | Standard Resolution Metric Alarm-0.30 per alarm metric, Composite Alarm-0.50 per composite alarm |
| Logs | 5GB Data | Collect (Data Ingestion)-0.03 per GB, Analyze (Logs Insights queries)-$0.0067 per GB of data scanned |
| Events | All events except custom events are included Text | Custom Events-1.00 per million events |
| Contributor Insights | 1 Contributor Insights rule per month | Contributor Insights Rule-0.027 per one million log events that match the rule per month |
| Synthetics | 100 canary runs per month | $0.0017 per canary run |
| Evidently | 3 million Evidently events and 10 million Evidently analysis units per account | $5 per 1 million events |
| RUM | 1 million RUM events per account | $1 per 100k events |
What is CloudTrail?
CloudTrail helps to get the details of who accessed what. Here, "who" refers to an IAM (Identity and Access Management) user or an AWS service, and "what" refers to AWS resources and services. CloudTrail is one of the AWS services that provides auditing, governance, and compliance for our AWS account. It records all the activities and events that are happening on our AWS account in the form of logs.
How Does It Work?
AWS CloudTrail is a region-specific service. So, by default, AWS CloudTrail will be enabled on our AWS account when we create it.
We can create two types of trails. They are single-region CloudTrail and multi-region CloudTrail.
Cloudtrail in a single region
When we choose to create an AWS CloudTrail for a single region, AWS CloudTrail will record all the activities in a single region. Those recorded event logs will be sent to the Amazon S3 bucket of our choice. But we can create a single-region cloud trail by using the AWS CLI only.
Cloudtrail in all-regions
When we choose to create an AWS CloudTrail for all regions, all the activities and events that are happening will be recorded in cloud trail event log files. The log files will send the data to the S3 bucket of our choice.
AWS suggests creating an all-region cloudtrail as best practice so that, as a customer, we can have more visibility towards our entire account across all regions.
The All-region trail is the default option when we create a trail in the AWS Management console.
Note: For single region and all-region cloudtrail, we can specify an Amazon S3 bucket from any region.
Events in Cloudtrail Logs
There are three types of event logs we can create for cloudtrail logs. They are
| No | Event Type | Description |
|---|---|---|
| 1 | Data events | These events provide details about the operation occurring on an AWS resource or within a resource. These are also called data plane operations |
| 2 | Management events | This event gives information on management actions taken about AWS resources, such as NON-API Events. User login, logout, etc., as an example. It is also known as control plane operations. |
| 3 | Insights events | By examining the logs in our S3 bucket, Insight events can identify suspicious activity in our AWS account. A new folder will be created inside the S3 bucket once the insight events are enabled. It will recognize any change in usage patterns that is odd or inconsistent with the normal usage of an API request or resource generation in our account. Only write management API calls produce insight events. |
Benefits of CloudTrail
- All the events that have happened over the past 90 days will be recorded in the CloudTrail Management console.
- We can automatically monitor and create an alarm on specified events by sending log events to Amazon Cloudwatch logs.
- Using Athena, we can query logs and analyze AWS service activity.
- If our accounts are associated with an organization, we can build an organization trail. In other words, we may set up a trail that will record every occurrence for every AWS account that is a member of the organization. Organization trails apply to either one or all AWS regions.
- CloudTrail can centralize and store all of the logs from all of the regions in an S3 bucket.
Examples and Use Cases for CloudTrail
Imagine that an EC2 instance in a customer's production environment has gotten a reboot alert. How to find out who rebooted the EC2 instance?
The answer is CloudTrail.
We should copy the instance ID and paste it into the search box next to the Resource name option in the cloud trail console. The Cloudtrail will display a list of all actions taken together with all relevant information, such as which IAM user rebooted the instance and when it happened, among other things. We can also get additional essential information such as who logged in to the AWS console and what action the IAM user took on the AWS resources, etc.
Pricing of CloudTrail
The below table shows the pricing for AWS Cloudtrail
| No | Components | pricing |
|---|---|---|
| 1 | CloudTrail Insights | $0.35 per 100,000 events analyzed |
| 2 | Data events delivered to Amazon S3 | $0.10 per 100,000 data events delivered |
| 3 | Management events delivered to Amazon S3 | $2.00 per 100,000 management events delivered |
CloudWatch vs CloudTrail: Detailed Comparison
The main differences between CloudWatch vs CloudTrail are described in the below table.
| No | Components | Cloudtrail | Cloudwatch alarms | Cloudwatch metrics | Cloudwatch logs | Cloudwatch events |
|---|---|---|---|---|---|---|
| 1 | Purpose | Auditing, governance, and complaint standards | Once the threshold is reached for metrics, the alarm will trigger | Data points for measuring the performance of AWS resources and customer applications | The central location where the AWS service and application publish logs | All the AWS account operational events are recorded within AWS resources, including AWS cloudtrail, in near real-time |
| 2 | Recording time | Every 15 minutes | 10 seconds to 24 hours | Depends upon the service it varies | Near real-time | Near real-time |
| 3 | Native integrated Service | Amazon S3 | Amazon SNS, System Manager | Supported metrics for Nearly 70 AWS Service | Amazon S3, Log insights query editor | Provides same capabilities with Amazon Eventbridge |
| 4 | AWS Organization Support | Yes | Yes | Yes | Yes | Yes |
Conclusion
- Amazon Cloudwatch is used to monitor AWS resources, while AWS Cloudtrail is used for auditing, governance, and compliance.
- AWS Cloudtrail keeps records of the past 90 days' events in the console, whereas Amazon Cloudwatch log groups never expire by default.
- For security reasons, we should delegate IAM users access to Amazon Cloudwatch and AWS Cloudtrail services.
- AWS CLI, AWS SDK, and the AWS Management console can be used to access Amazon Cloudwatch and AWS Cloudtrail.
- Amazon Cloudwatch is natively integrated with the SNS Service for alarms. Similarly, AWS Cloudtrail is natively integrated with Amazon S3 for pushing logs.