S3 Event Notifications

What is AWS S3 Event Notification?

While working with AWS S3, there might occur scenarios where you may want to perform an action whenever you uploaded/modified a file in the s3 bucket. You might want to get notified every time any change or modification happens with your S3 bucket. To resolve this issue, AWS offers AWS S3 Event Notification service which helps to generate notifications that allow some AWS services directly or indirectly to make the users aware of some changes that happened with the S3 bucket and prevent any unwanted or unknown modifications if any.

The Amazon S3 Event Notifications offers a service where it receives notifications when any events happen in the S3 bucket. For enabling the notifications, you need to add a notification configuration that shall identify the events that you might want the AWS S3 to publish. Recommended to always identifies the destinations correctly while enabling the AWS S3 Event notification, where you might want to send the notifications. You can also store and capture the configuration in the notification subresource associated with a bucket. You also get an API to manage these subresources, that AWS S3 offers.

Event Notification Types and Destinations

AWS S3 offers and supports many event notification types and destinations where the notifications can be published. It's mandatory to define the event type and destination when you are configuring the event notifications.

Supported event destinations:

Listed below are the destinations where the AWS S3 can send event notification messages:

• Amazon Simple Notification Service (AWS SNS) topics
• Amazon Simple Queue Service (AWS SQS) queues
• AWS Lambda
• Amazon EventBridge

Amazon Simple Notification Service (SNS) topic:

AWS SNS can be defined as a fully managed and flexible push messaging service that is widely used for pushing messages to mobile devices or any distributed services. It is seen that the Amazon SNS console is used for creating the SNS topic so that the notifications can be sent forward. You can publish a message once, and then it delivers it more than once too. It helps to coordinate as well as manage the sending and delivering of messages to the endpoints of the subscribers or clients.

Amazon Simple Queue Service (SQS) queue:

AWS SQS can be defined as storage for the messages as they travel between the computers. It is a reliable and scalable hosted queue that makes use of the AWS SQS for transmitting any volume of data without even requiring the other services to be readily available. It is recommended that the AWS SQS queue must always be in the same AWS Region as the Amazon S3 bucket.

Lambda function:

With the AWS Lambda, you can extend other AWS services with your own defined custom logic, or can also create your custom-owned backend which operates at AWS performance, scale, and security. With Lambda it gets quite easier to create event-driven and discrete applications which run only when required. Ranging between a few requests a day to thousands a second, with AWS lambda you can use it to scale the applications automatically as well.

Amazon EventBridge:

AWS EventBridge is defined as a serverless event bus, that receives the events from the AWS services where you can set up the rules that will match the events. This way you can deliver it to the targets, like an AWS service or an HTTP endpoint. You can either enable or disable events that are yet to be delivered to EventBridge for a bucket. Once you enable the delivery, all the specified events are sent to the AWS EventBridge.

Supported event types for SQS, SNS, and Lambda:

Below are a few of the events that can be published by Amazon S3 which you need to mandatorily specify the event types notification configuration.

Supported event types for Amazon EventBridge:

You can use the AWS EventBridge, for the list of event types that Amazon S3 will send events to Amazon EventBridge whenever certain events happen in the S3 bucket. You don't need to pre-decide which event types you want to deliver. Once the AWS EventBridge is enabled, every event is sent to EventBridge here you can also make use of the EventBridge rules to route events to additional targets. Listed below are a few of the events Amazon S3 sends to EventBridge.

Do we need an explanation?

Creation of S3 Event Notification

Now we shall be learning how we can create and enable the S3 event notification using SQS, SNS, and Lambda.

Enabling the S3 event notifications is a bucket-level operation where notification configuration information is stored in the notification subresource associated with an S3 bucket. Once you make any modifications or changes or even create the bucket notification configuration, it shall take about five minutes for the changes to take effect. Once the notification is enabled, an s3:TestEvent occurs. You can enable the notification via the following methods, where the AWS S3 manages to store and capture the notification configuration as an XML in the notification subresource.

• Via the AWS S3 console:
  Via the console UI you can set a notification configuration on an S3 bucket (without the hassle of coding).
• Via the AWS SDKs programmatically:
  Either you choose the console or the SDK option both internally call the Amazon S3 REST API for managing the notification subresources associated with the S3 bucket.
• QuickNote:
  Any AWS S3 REST API calls can also be made directly from the code, where this might come as tedious as you have to write code for authenticating the requests.

Using AWS Lambda Function

With the below steps, you can create the S3 Event Notification using the AWS Lambda Function:

Start by creating an S3 bucket by opening the AWS Management Console and navigating to the AWS S3 service to start raising the S3 bucket option.

Specify the name of the S3 bucket and create the bucket as shown below. Your S3 bucket is created.

You can create the AWS Lambda function by searching for the Lambda service on the AWS Management Console.

Click on create function as sown below to select the 'Use a Blueprint' option. Now you need to search for 's3-get-object-python' and configure it.

Start by providing the name to the Lambda function, and for the execution role, you can choose the first option.

You can adjust the requirement from the available piece of code for the Lambda function and select the Create Function option.

Your Lambda function is ready to use. Now you need to go to the AWS S3 bucket created above where in the Properties section you shall see the section for Event Notification. Select the Create Event Notification as shown below

Configure the Event Notification by assigning it an event name and selecting different events according to the requirements.

Now select the Lambda function you created in the Lambda function and save the changes.

Great Job! You can see the event notifications created for the AWS S3 bucket.

Using Amazon SQS

With the below steps, you can create the S3 Event Notification using the AWS SQS:

Start by creating an S3 bucket where we shall be configuring the S3 event notifications. You can do this, by simply login into the AWS account, and searching for AWS S3 service in the AWS management console as shown below.

Once the S3 console appears, then click on create the S3 bucket as shown below.

Now you need to provide the S3 bucket a relevant name which must be universally unique as shown below.

Specify the AWS region for which the S3 bucket is valid to be placed. As we know that S3 is a global service, and accessible from any region, therefore you need to specify the correct region where the data shall be stored as shown below.

Now you can also manage settings like versioning, encryption, and public access, or can simply leave them as default. Now select the create bucket to finish the S3 bucket creation process as shown below.

We have created a new S3 bucket in the AWS account as shown below.

Let us get started on creating the target for the S3 event notifications, where you need to select the AWS SQS service. Start by simply searching for AWS simple queue service, on the top search bar of the AWS Management console as shown below.

Select the Create Queue option to create the new SQS topic as shown below.

Now you can choose the type of the SQS or can leave it as default too, i.e., the standard one as shown below.

Provide the name of the SQS topic as shown below.

Enable the S3 bucket for writing to this AWS SQS service, where you need to provide the right policy to SQS. You can also opt for advanced in the access policy section as shown below.

For simplicity, you can use the AWS provided Policy generator or can modify the policy in JSON by easily clicking on the link shown below to open the policy generator.

As seen in the policy generator start by selecting the policy type, for this case the SQS Queue Policy as shown below.

Now you need to select the effect ALLOW, the principle shall be the S3 bucket where we are configuring the S3 event notifications, hence we need to put the bucket ARN there. The resource should be the service where we need the policy to be applied to let our SQS topic ARN as shown below.

On the other hand, for the Action, we can select Send Messages, as it enables our S3 bucket to send the message to the SQS queue as shown below.

Select the add statement and select the create policy to generate the policy in json format as shown below.

Copy the policy contents and place them in the SQS policy as shown below.

In the S3 bucket, choose the properties tab as shown below.

Now we need to drag down to the S3 Event Notifications and start creating the S3 event notifications as shown below.

Start by giving the name of the S3 event notifications as shown below.

Select the event type for which the S3 event notification shall be enabled. You can choose it from the vast list provided, for now, we are choosing all object creation events notifications as shown below.

Select the destination for the S3 event notifications for which you need to choose the AWS SQS service where you need to choose the name of the AWS SQS topic.

Lastly, Select the save changes option as shown below to complete the process.

Now we are finally done with the S3 event notifications. Let us start to upload an object in this S3 bucket and validate if that triggers the SQS or not.

As shown below, simply upload an object to the S3 bucket.

You shall see that you will get the notifications for the events that we eventually enabled by navigating to the SQS and poll for messages.

We have successfully configured the S3 event notifications for the SQS topic.

Using Amazon SNS

With the below steps, you can create the S3 Event Notification using the AWS SNS:

We use the AWS command-line interface (CLI) to create the S3 Event Notification using the AWS SNS.

Start by creating the S3 buckets in the AWS account, for which you could use the following command as shown below.

To see all the available S3 buckets in the AWS account, try running the below command and you shall see something as shown below.

Create the AWS SNS topic by running the following command on CLI.

To define and specify the event type and SNS topic ARN, you will need a configuration file in JSON format. A sample is shown below.

Lastly, enable the S3 event notification by running the following command on the CLI.

You have now successfully enabled the S3 event notifications for the SNS topic.