AWS SQS (Simple Queue Service)

What is AWS SQS?

Before knowing what is AWS SQS? Let us understand what is a queue. A queue can be defined as a place where the messages are stored until they are extracted from the queue or eventually get expired. This way some information can be stored so that later on it can be consumed as per requirements. We implement the decoupling in the queue by managing the messages into the queue between the producer and consumer of that message which only gets to the consumer once a call to SQS is made. So, now we can say that Amazon Simple Queue Service(SQS) is a fully managed queue service offered by the AWS cloud.

The AWS SQS can be described as a distributed message queuing system, widely known for its message queuing functionality, that could also be scaled up and down as per requirements to automate the process of establishing and maintaining the messaging queues in addition to allowing decoupling between the components of servers so that they act independently.

The concept of decoupling can be incorporated between two microservices by introducing an SQS queue in-between them to allow them to communicate by enabling microservice 1 to send the messages to the SQS queue while microservice 2 can pull those messages from the queue depending upon the capacity and discretion. As this communication between the microservice is asynchronous hence we can enable it as decupled. By making it decoupled we make it fault-tolerant as well as when the consumer will be overloaded it can wait between receiving another message

Below shows a simple diagram of AWS SQS:

Features of SQS

Below are the various features that make AWS SQS a highly demanded aws service for message management solutions among developers:

• The Scalability: With the data growing at such a fast rate products we use must have the factor of scalability to scale it up and down as per traffic. The AWS SQS has high scalability by independently processing all the buffered message requests at once. Also, when the traffic or load spikes it is capable of scaling transparently without any prior instructions.
• The Elasticity: As seen above, as the scalability in the AWS SQS increase and decrease depending upon the number of task in the queue this enables the AWS SQS to stay elastic when the spike increase or decrease.
• The Reliability: As AWS SQS stores the message without losing any, it allows multiple producers to transfer messages simultaneously to multiple consumers. it does so by using the message-locking mechanism that it uses during its processing phase.
• The Availability: The amazon sqs prioritizes that its users must never face issues concerning availability and makes sure that the users' experience is good by increasing the availability of messages on both sides the producers and consumers. It does so, by leveraging infrastructure to generate high-speed concurrent messaging.
• The Durability: The two types of AWS SQS are, Standard Queues & FIFO Queues work on an at-least-once & exactly-once message delivery system as it relies on various servers to capture the messages safely.

Below is a simple diagram displaying the features of the AWS SQS service:

SQS Visibility Timeout

As seen from the above diagram, once the AWS SQS receives the message "msg" in its queue from the producer, it sits in the queue waiting for it to be processed. Once a component "abc" sends a “Recieve message request” for that message "msg", then that message starts to process for that component "abc" and this starts the visibility timeout (say 60 seconds) for that message. This message "msg" remains in the queue but gets marked as invisible that is, even if another component "def" sends a “Recieve message request” for that message "msg" , AWS SQS will not allow the message delivery to component "def" until the visibility timeout expires. But once the visibility timeout for the message "msg" runs out, the message again appears back in the queue for other components to be picked and processed. Sometimes we experience scenarios where the job has taken a long time to complete and by that time the visibility timeout got expired for a message, then, in that case, the message again gets appeared in the queue for other components to process it. Also, if the component "abc" completes its processing of the message "msg" in the visibility timeout period, then the message "msg" gets deleted from the queue.

Hence, it is advised to keep the visibility timeout (applied to the entire queue) at an appropriate timeout concerning the time taken by a job to finish to avoid the high probability of the message being delivered twice. Increasing the AWS SQS visibility timeout allows the messages to get processed in a longer time frame between the producers and get picked up by any queue or consumed. If we see that a message is having insufficient time to process a message then we can increase the visibility timeout by using the ChangeMessageVisibility Request which affects the timeout of that message and not the whole queue. The default AWS SQS visibility timeout is 30 seconds which can be increased depending on the task, with the maximum visibility timeout being 12 hours.

We can also select the Message retention period option, for retaining our messages with its duration ranging from 1 minute to 14 days, where the default retention period is 4 days as shown below

AWS SQS Message Security and Compliance

For any service that we use, we are often concerned about the security and compliance that it offers. With that being said, amazon sqs offers security concerning the messages that a producer sends to the consumers without facing any loss in the messages and keeping the privacy intact too.

It is recommended server-side encryption safely transfers confidential or sensitive messages (data) between two servers or applications. This is a highly appreciated step among developers. Here, we can also ensure that whenever any encryption keys are getting utilized, with the help of the AWS Key Management Service we can keep a record of it by logging the detail in the AWS CloudTrail. By doing so, we make sure the user follows compliance with data management regulations.

Basic Architecture & Types of Queues in AWS SQS

Any distributed messaging system has a basic architecture consisting of three major parts, Queue (that are distributed on your amazon sqs), Components of the distributed system, and the Messages that can be found in the Queue that allows you to combine to provide us with a message management system as well.

As seen in the below image, we see all three components in an instance of the AWS SQS:

As seen above, we see the producers that are known for sending the messages input into the queue which then gets consumed by the consumers as seen. The consumers are waiting for receiving the output messages from the queue. And we also see the third component which is the queue that consists of the messages A to E which are then stored by implementing the various amazon servers.

Now we shall introduce you to the two different types of Amazon SQS Queues:

• Standard Queue
• FIFO Queue

Let us learn about each type in detail.

Standard Queues (default)

Below is the Standard Queues diagram:

• The AWS SQS provides us with a default queue type called the Standard Queue that gives us the ability to work with approximately an unlimited number of API calls (transactions) that get fired every second per API action.
• The different types of actions can be SendMessage, ReceiveMessage, or DeleteMessage
• Standard queues work on the “at-least-once” message model for delivery of its messages which guarantees to deliver the message at least once which sometimes can be more than one copy when the delivery is out of order.
• The Standard Queues in AWS SQS emphasize the best effort of correct order delivery of the messages which means the sequence in which the messages were captured in the same sequence in which the messages will be delivered to the consumer but it does not provide the guarantee for this.

FIFO Queue (First-In-First-Out)

Below is the FIFO Queue diagram:

• As the name suggests, the FIFO queues also First-In-First-Out were built with the clear goal of solving the limitations encountered in the Standard queue.
• The FIFO queue enhances the inter-application messaging to ensure that the right order of user-entered commands is getting sent in scenarios where the order of the operations is crucial along with redundancy is not permissible. hence, we can say that it guarantees to order.
• The important features offered by the FIFO queue are that along with correct ordering the messages are processed with an "exactly-once processing" message model that is, a message gets delivered only once which then remains available until it gets processed by the consumer and deleted too. This ensures that no duplicates are getting generated in the queue which reduces ambiguity.
• It also enables the empowerment of the message groups which allows to send off of a single queue with multiple ordered message groups. Here we need to note that the FIFO queues get limited to 300 transactions per second but offer all the features of the standard queues.
• Situtaion for whihc FIFO queue could be implmenetd are:
• While displaying the accurate product price by transferring the price modifications in the correct order. Also, when no person can register for the workshop without paying the fee for it.

How Does AWS SQS Work?

For understanding how the AWS SQS works we need to see a scenario based on which we can analyze the findings better and learn how the AWS SQS works:

Suppose you want to convert your birthday video into a .gif format to upload on social media. Now you start searching on the web and find this amazing gif create and upload your birthday video and in return get your .gif file ready to be uploaded.

But what happens behind the conversion, let us understand in detail.

• The site first stored your video into their Amazon S3 and invoked a lambda function which then started to retrieve data from the video and starts adding it to the AWS SQS by calling the "SendMessage Request".
• Now each message that is getting added to the queue which can be anything (caption on the gif, the duration, or anything) has a deduplication ID attached to it.
• Now the taskmaster here is the AWS SQS (which contains all the relevant details of the tasks that are supposed to be performed) and is waiting for the consumer let's say EC2 instance.
• Now the EC2 sends a "RecieveMessage Request" to make a pull request. Now once the EC2 instance is assigned it starts to read the task and start executing it to create a gif. Finally, the created gif is stored back in the same location as amazon S3 or even a different Amazon S3.
• Once the tasks are finished from the EC2 end, it makes " RecieveMessage Request" pick another available task from the queue.
• Now as we know that the SQS can scale automatically if in our scenario we had three EC2 instances that were pulling messages from the SQS Queue, then the autoscaling group will start adding additional EC2 instances to process the messages when the number of messages increases more than 10 messages. Also when the scene arrives when there is only one message in the queue, then only 1 EC2 instance gets provisioned to process that which makes amazon sqs “ELASTIC” enough.

Below shows the How Does the AWS SQS Work as we discussed above: [Video --> S3 --> lambda >>>>"SendMessage Request" ----> SQS ---> Queuue ---> message has deduplication id ---> SQS has all the information ------> >>>>>>>>>> "RecieveMessage Request". ---->EC2(consumer) --- S3]

How to Set Up AWS SQS?

For setting up the Amazon sqs to build Standard and FIFO Queues which can offer us default values for various settings. We need to follow the below steps to set up your own AWS SQS Queue:

• First, we need to open the AWS console and type “SQS” in the search bar to reach https://console.aws.amazon.com/sqs/ or directly open the SQS from the link as shown below.

• Then we need to click the Create queue option, where we shall find the Type option where e shall see that the default setting represents the Standard queue which we can change to the FIFO queue by simply clicking on FIFO.
• We need to specify a suitable queue name and keep in mind if we are creating the FIFO queue, then its name needs to end with the suffix .fifo. Also, once the Queue name is created we cannot edit it.
• On the bottom we shall see the Visibility Timeout option where we need to input a suitable duration with units. The Visibility timeout limit is from 0 seconds to 12 hours. We can also select the Message retention period option, for retaining our messages with its duration ranging from 1 minute to 14 days as shown below.

• We can then set the Delivery delay option that ranges between 0 seconds to 15 minutes where the Maximum message size can also be set between 1 KB to 256 KB. To enable the FIFO queue, we can select Enable content-based deduplication.

• Now to enable Redrive to allow the policy, we need to select one option from the three options: Allow all, By queue and Deny all options as shown below.

• Finally, we click the Create queue, and the Amazon SQS console would have created the required queue and displayed the respective details on the page.

After the queue has been created, you can navigate to the left for the Queues option and choose from the queue that we created above. Then you can start typing the message and eventually the message can be seen below.

Advantages and Disadvantages of AWS SQS

Now let us explore the pros and cons of using the AWS SQS service offered by AWS.

• When we need message queues for the product that we are building, it will require a huge amount of advanced development effort. Even though prepackaged options are available, the maintenance and configuration require close attention and would also demand continuous investment as part of hardware and system administration resources. With that being said, the AWS SQS helps users to keep away from facing such issues that shall require additional time and resources.

• With the advanced ability to decouple the various components, the AWS SQS help to deliver high-performance standards where each component can run independently in addition to failing independently which helps increase the system's overall stability and fault tolerance.

• The amazon sqs gives its user the access to not keep its queues similar that is, setting a default delay option on a particular queue is possible. if any message is above a certain size, then by using the Amazon Simple Storage Service or Amazon DynamoDB the content of messages can be stored and can also be split into smaller sizes if required.

• One of the challenges that might be faced with AWS SQS is that to optimize output we have to previously fine-tune the message management system concerning a scale which is not possible with AWS SQS as the messaging service is automated and no option to edit is given.

• If we talk in terms of pricing, when your messages are massive it might hit your pockets as well. This is the result because data transfer costs are a part of AWS SQS fees which increases the overall pricing for larger messages.

• As the AWS SQS operates on an “exactly once” message transmission model, it does not support multiple organizations to access the same message at the same time it operates. This could be solved by integrating the AWS SQS with AWS SNS for scenarios around one-to-many deliveries.

AWS SQS Pricing

With AWS being in the picture we know that the organization saves a lot of money by paying no penny for the maintenance or buying any support tools for the services they use and paying for only what they used and also until the time they used. Adding to this service is why the Amazon SQS gained so much popularity as we don't have to pay any upfront cost to use its services in addition to no need to buy, deploy, build, or maintain any supporting infrastructure. This ensures that we save a good amount of money by not caring to rely on alternative tools with high-pricing models.

Also, the pricing structure of Amazon SQS is usage-based, which helps us save substantial cost savings compared to the always-on-pricing model tools. If we enable the AWS Free Tier then it enables the amazon sqs users to make up to 1 million Amazon SQS requests per month for free.

Below is a snippet from the official AWS pricing page that tells us about the pricing structure for Amazon SQS:

AWS SQS Compatibility

If any of the services that we use is highly compatible with any service that it becomes quite easy to integrate different services and build our architecture as well. With that being said, the Amazon SQS is highly compatible with many AWS products available. Below is the list of the most popular amazon services that are easily compatible with Amazon SQS:

• Amazon Elastic Compute Cloud or AWS EC2
• Amazon Relational Database Service or AWS RDS
• Amazon Simple Storage Service or AWS S3

AWS SQS vs MQ vs SNS: Key Differences

• The amazon sqs is widely popular for the serverless queues that provide a robust and fast messaging service that does not require us to for the infrastructure.

• Amazon MQ is getting known for the fully managed service that it provides to connect us with popular message brokers. With the help of this tool, we can migrate applications from our existing message brokers seamlessly. It is readily compatible with various APIs adds up to its effectiveness.

• On the other hand, The amazon sns is used for operating in domains of serverless topics where the notification service offered keeps us up to date with new message arrivals only when we are online. It is also possible to integrate the SNS with amazon sqs which ensures that the SNS messages are not lost or dropped if on fluctuating service as sqs will handle it effortlessly.

• Also, as amazon SQS and Amazon SNS are the queue-based facilities offered by aws also offer high scalability and useability without forcefully integrating them with message brokers.