Partial Batch Failure and SQS Redrive will make your life simpler
Hey there! A number of bulletins have been made within the AWS Re:Invent 2021, on this article we’ll check out a few of the ones relating to the AWS SQS.
What’s SQS? From the overview web page of the service “Amazon Easy Queue Service (SQS) is a totally managed message queuing service that lets you decouple and scale microservices, distributed methods, and serverless functions.”
I’d like to elucidate the providers through a pratical instance: think about you could have an endpoint the place a consumer needs to export an enormous excel, so that you don’t need the consumer to attend for the export and also you need to despatched it by way of e-mail.
On this case your endpoint pushes a message (or a job) within the SQS Queue and a Lambda Operate will learn the message from queue and course of the export, then ship it by way of e-mail. It is a typical producer-consumer sample. In case of error, the message will be reprocessed or you’ll be able to transfer the message in a DLQ (Useless Letter Queue).
The DLQ Queue is a SQS Queue strictly related with the primary queue. Each message within the queue has a counter: you’ll be able to set a threshold worth after which the message can be moved within the DLQ. It’s actually a useless queue. You may learn the failure message for the debug and you may determine to repair the issue in your client after which reprocess it.
As acknowledged above, when a component of SQS has failed, it returns within the queue and can reprocess, however after a few tries the aspect goes within the DLQ.
Once you course of a number of parts with a Lambda Operate and a few of them fail, all parts return within the queue, or you’ll want to delete the profitable parts from queue.
However with the Batch Partial Failure, your Lambda response you’ll return the listing of the failures id. Routinely, with out the necessity to delete the weather your self, you’ll be able to differentiate success objects and failures merchandise.
From my perspective, I see the redrive of the weather from DLQ to primary queue as a key function. Earlier than its introduction, you had to do that manually with a customized implementations that learn the weather from DLQ and pushed them in the primary queue.
At high left of the SQS Console we discover the “Begin DLQ Redrive”. So, a click on will take us to the redrive configuration.
Rembember: if the SQS Queue don’t have a DLQ the button can be disabled!
With the default settings the DLQ objects will go the in SQS Supply. However, if you wish to redrive in a unique queue or setup a unique velocity you’ll be able to edit that right here.
