Amazon SQS

• a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications
• SQS offers two types of message queues
  • Standard queues offer maximum throughput, best-effort ordering, and at-least-once delivery.
  • SQS FIFO queues are designed to guarantee that messages are processed exactly once, in the exact order that they are sent.

SQS Benefits

No administrative overhead

Reliable message delivery

Multiple copies of every message are stored redundantly across multiple availability zones so that they are available whenever needed.

Secure sensitive data

 using server-side encryption (SSE) to encrypt each message body.

Amazon SQS SSE integration with AWS Key Management Service (KMS) allows you to centrally manage the keys that protect SQS messages along with keys that protect your other AWS resources.

Scalability

 dynamically scale based on demand

There is no limit to the number of messages per queue, and standard queues provide nearly unlimited throughput.

With Amazon SQS, multiple consumers can’t receive the same message simultaneously. A message can only be received from a queue by one consumer that’s ready to process and then delete the message received. A message can be retained in queues for 14 days maximum. 

 

SQS Cost Metering

• Every Amazon SQS action counts as a request.
• Amazon SQS pricing is based on the number of requests made to SQS (be it any type of request – send, read, delete etc.)
• A single request can have from 1 to 10 messages, up to a maximum total payload of 256 KB.
• Each 64 KB chunk of a payload is billed as 1 request 
• Data transfer in and out refers to transfer into and out of Amazon SQS.
• Data transferred between Amazon SQS and Amazon EC2 or Amazon SQS and AWS Lambda within a single region is free of charge (in different regions is charged at Internet Data Transfer rates on both sides of the transfer)

Push vs Pull

Amazon SQS being a http based service, it does not support push based message consumption as some of the other messaging brokers support, especially those complying to the JMS specification. Thus, as a consumer, one has to poll Amazon SQS for consuming messages.

Two polling mechanisms

• Short polling
• Long Polling

Short Polling

• Default Polling mechanism
• almost instantaneously, typically within tens (or lower hundreds) of milliseconds
• Amazon SQS infrastructure being highly distributed, to maintain the quick sub-second response time, it may not be able to check all the servers, thus it typically checks a few servers based on a weighted random distribution and returns messages only from those servers.
• A direct side effect of this is that a read request may not return any messages even if there are messages in the queue (False empty responses)
• We can specify, in the read request, how many messages you would like Amazon to return as part of the response. The valid values for this parameter are 1 through 10. So, at max, we can request Amazon SQS to return 10 messages in the response. The default value is 1.
• one can run multiple consumers (using threads, multiple machines etc.) consuming messages from the same queue

Long Polling

• Do a Long poll while trying to read messages.
• So, as part of the read request, Amazon lets us specify a timeout period (in seconds) for which the read request blocks on the server if there are no messages available to consume. 
• If there are messages in the queue to consume, the read message request does not block. Essentially, the focus here is to at least return one (or more) messages as long as messages are available for consumption. If the queue is empty, then the read messages request would wait for the specific time (1 to 20 seconds as specified in the request) before returning an empty response. For example, if we set the Long Polling wait timeout value to 20 seconds, the request would block on the server for a maximum of 20 seconds, in case there are no messages to return.
• Long Polling can never result in False Empty responses since you are guaranteed to get at least one message in the response as long as the queue is non empty. Please note that Long Polling may result in empty responses if the queue is empty

Considerations

Long Polling definitely seems to have an edge above Short Polling in most cases. It not only eliminates false empty responses, but also reduces the number of empty responses, thus leading to significant cost benefits. 

If you want your messages to be processed as soon as possible or as close to the messages being sent as possible, Long Polling is the option to go with.

Short Polling can be used for cases in which the timing of message processing is not as critical.

Delivery Delay

lets you specify a specified amount of delay to be introduced while delivering messages to its consumers.

The delay is typically specified as part of sending the messages and is implemented by the messaging broker/provider transparently without the sender or the consumer having to do anything about it the delay is introduced by the broker while delivering the message, and neither is the sender blocked, nor is the message sending action delayed from a senders perspective. In other words, the message gets queued up in the broker/provider but is available for consumption (to any consumer) only after the specified delay.

 

Two mechanisms supported by Amazon SQS to implement Delivery Delays

• At Queue Level
• At Message Level

Queue Level Delay

• A queue can be optionally configured as a Delay Queue, thus implying all messages sent to that queue being delayed unless specifically overridden otherwise at a message level.
• The configuration property is named “DelaySeconds” and is specified in terms of seconds, with valid values being 0 to 900. Thus, the delivery of messages to consumers can be delayed by a maximum of 15 minutes. The default value is 0.
• A queue can be configured as Delay Queue during creation.
• Value is specified in seconds, valid values are from 0 to 900.
• The delay information is not available to consumers in the message.
• An existing queue can be configured as Delay Queue as well
  • Standard Queue – the existing messages do not get impacted
  • FIFO Queue – the configuration change is applied to existing messages as well, along with the new messages, making the configuration retroactive… Why – if this is not done, the ordering of messages could get distorted from a consuming application standpoint

Message Level Delay ( Message Timers)

• specify a delay for each message in the sendMessage request.
• Again, the delay is specified in terms of seconds, with valid values being 0 to 900. The default value is 0
• If a message with a Delay Seconds value is sent to a queue, and the queue itself is configured with Delay Queue option, the value specified in the send message request takes precedence over the queue level settings of the Delay Queue.
  • DelaySeconds=non-zero value in the message, the value supplied in the message takes precedence and overrides the queue level setting
  • DelaySeconds=0 in the message, the value supplied in the message overrides the queue level setting, thus there is no delivery delay for this message.
  • DelaySeconds not specified (or specified as null) in the message, the queue level settings (if any) take effect.

Message Life Cycle

New Message

Amazon SQS queues are modeled on point-to-point messaging model, so a message can be consumed by only a single consumer at any given point in time

Message Expiry

messages in Amazon SQS have a lifetime too and it is enforced.
SQS cannot let messages just pile up in queues, without getting consumed, or without getting drained from the queue to be more precise

MessageRetentionPeriod

A queue can be configured so that all messages sent to that queue would have a certain expiry period.Its value is specified in seconds, and can range from 60 seconds (1 minute) to 1209600 seconds (14 days).
The default value for a newly created queue is 345,600 seconds (4 days), unless modified.
There is no way to set the expiry period at a message level.

Corrupt message

you cannot stop processing the queue and bring the whole system down for one such poison message. The queue may have thousands of other messages that can be processed and are waiting to be processed.
the standard messaging pattern is to move the message to a Dead Letter Queue after a certain number of retries. The Dead Letter Queue is a separate queue that is monitored separately for such messages and can be used for alerting and notifications about such events.
SQS supports configuring a Dead Letter Queue for every SQS queue you create, and lets you configure a threshold in terms of the number of times a message is retried for processing, failing which, the message is moved to the Dead Letter Queue. This configuration itself is known as Redrive policy and the threshold setting is known as maxReceiveCount within Amazon SQS. Thus, for example, if you have configured the maxReceiveCount setting as 100, Amazon SQS would move a message to the configured Dead Letter Queue only after the message has been read 100 times but still not deleted from the original queue, indicating the consumer(s) are unable to process this message.

Message Isolation

There is no concept of acknowledgement modes within Amazon SQS, however there is a similar concept of message visibility timeout provided by SQS.
whenever a message is returned in the response to the read request made by a particular consumer, that message is hidden from other consumers for a specific period of time. The time window is called Visibility Timeout.
once that particular consumer is done processing the message, it would make a delete call to delete the message.
delete is not automatic, Amazon SQS requires you to delete the message – the consumer directly makes a delete call to SQS indicating that it is done processing.
delete should happen within that message visibility timeout period because it is during this period that the message is not available to other consumers. Once the message is visible again, it may be read by other consumers and hence reprocessed. The message is said to be Inflight while it is being read by a consumer and is within the message visibility timeout period.

All queues have a default visibility timeout of 30 seconds, can be configured at queue level.
The maximum period supported is 12 hours.Visibility timeout can be specified by consumers in the receive message request. The visibility timeout specified as part of the receive message request does not impact the overall queue level visibility timeout settings, they are just valid for messages returned in response to that particular receive message request.

Visibility timeout Considerations
Short

delete call is expected to be made within the Message Visibility Timeout window. If the delete call is not made at all or is made after the Message Visibility Timeout expires, the same message would re-appear in the queue and hence would be available for processing by any subsequent read requests from consumers

Long

fine as long as the message is successfully processed and deleted by the consumer that read it 
if for some reason, the consumer cannot process the message – the message will remain hidden and hence not visible to other consumers for processing until the visibility timeout expires

SQS APIs

five basic APIs for developers

CreateQueue

SendMessage

ReceiveMessage

ChangeMessageVisibility

DeleteMessage

With SQS APIs, serverless developers can directly implement an SQS queue with any other AWS service, including a third-party service, allowing them to work in most any programming language of their choice. Amazon SQS supports message payloads comprising close to 256KB of text in any format. However, you can manage Amazon SQS messages content larger than 256KB (up to 2 GB) using Amazon Simple Storage Service (Amazon S3) or Amazon DynamoDB. 

SQS SSE uses the AES-256 GCM algorithm which is the 256-bit Advanced Encryption Standard to keep sensitive data secure using a unique encryption key. 
AWS KMS logs all your encryption key usage to AWS CloudTrail for  regulatory and compliance requirements. 
Amazon SQS is subscribed to HIPAA Compliance Program to allow developers to build HIPAA-compliant applications. Developers can use SQS to store and transmit messages between healthcare systems (including PHI-messages).
Amazon SQS integrates well with other AWS infrastructure web services, including Redshift, DynamoDB, RDS, EC2, ECS, Lambda, and S3 to increase the flexibility and scalability of your distributed application.

SQS Lambda Best Practices

  – you should always configure a DLQ for every SQS queue.

 – SQS Lambda Visibility Timeout – 

The SQS visibility timeout should always be greater than the Lambda function’s timeout.

However, the default visibility timeout for an SQS queue is 30 seconds. If the SQS function has a higher timeout value then in-flight messages can be available again and processed more than once.

But it gets worse. SQS functions receive messages in batches, and the messages are deleted from the queue only after the function completes successfully. It means that even if only one message was slow to process, the entire batch can elapse their visibility timeout and become available again in the queue.