How AWS Lambda Communicates with AWS SQS: An Overview of designing scalable web service

In my recent project, I built a serverless application leveraging AWS Lambda and AWS SQS to handle asynchronous task processing efficiently. This blog focuses on the communication process between these two AWS services, exploring how they seamlessly integrate to build scalable and reliable applications.

This is just an overview of how you can design scalable backend service with microservice architecture. When designing this architectures the goal was to be able to save as much as we can on computing resources and make it scalable across teams.

Why Use AWS Lambda with AWS SQS?

• Serverless Computing: AWS Lambda eliminates the need for server management and automatically scales with demand.
• Asynchronous Processing: AWS SQS allows decoupling of components, enabling reliable and scalable message passing.
• Cost Efficiency: Both services charge based on usage, making them ideal for cost-conscious applications.

In this project, AWS Lambda acted as a consumer for the SQS queue, processing messages asynchronously and ensuring the system could handle high-throughput workloads.

Setting Up Backend Communication with AWS SQS

In modern distributed systems, managing communication between microservices is critical for ensuring smooth workflows and scalability. For my project, I set up an architecture using AWS SQS (Simple Queue Service) as the backbone for message-based communication between services. Here’s an overview of how I implemented it and leveraged SQS to decouple and efficiently manage tasks across microservices.

The Architecture

The above diagram illustrates the architecture I implemented. It consists of the following components:

1. Main Backend: The core service responsible for handling API requests and orchestrating tasks. It sends messages to the SQS queue for downstream processing by other services.
2. Payment Service: Handles all payment-related tasks, such as transaction processing and payment confirmations. It listens to specific SQS messages and processes them asynchronously.
3. Email Service: Manages email notifications for user actions like confirmations, receipts, and alerts. Similar to the Payment Service, it consumes messages from SQS.
4. Error Service: Acts as a fallback mechanism to handle failed or invalid messages that cannot be processed by the Payment or Email services. It subscribes to the Dead Letter Queue (DLQ) linked with the SQS queue.

Implementation Steps

Here’s how I set up the system:

1. Creating the SQS Queue:

I created an SQS Standard Queue for handling messages from multiple services. Standard Queue was preferred because of its high throughput and best-effort message ordering.

2. Configuring the Main Backend:

The backend uses NestJS to handle API requests. Whenever a task like payment processing or email notification is triggered, it sends a JSON payload to the SQS queue using the AWS SDK for JavaScript.

import * as AWS from 'aws-sdk';

const sqs = new AWS.SQS({ region: 'us-east-1' });

async function sendMessageToQueue(payload: any, queueUrl: string) {
    const params = {
        MessageBody: JSON.stringify(payload),
        QueueUrl: queueUrl,
    };
    await sqs.sendMessage(params).promise();
}

sendMessageToQueue({ type: 'payment', data: { userId: 123, amount: 50 } }, 'https://sqs.us-east-1.amazonaws

3. Listening to the Queue

• Both the Payment Service and Email Service are set up as consumers for the SQS queue. These services poll the queue to retrieve messages and process them asynchronously.

4. Dead Letter Queue (Error Service):

• Any message that cannot be processed due to errors is sent to a Dead Letter Queue (DLQ). This is configured directly in SQS as part of the queue attributes. The Error Service periodically reviews these messages for debugging and retrying failed tasks.

Key Takeaways

• Flexibility and Decoupling: SQS allows each service to operate independently, enhancing the overall system’s flexibility.
• Scalability: This setup can easily handle thousands of requests per second without bottlenecks.
• Error Handling: The Dead Letter Queue ensures no messages are lost, even in case of errors.
• Extensibility: New services can be added to consume the queue without impacting existing ones.

Future Improvements

1. Message Filtering: Using SNS and SQS subscriptions, I can filter messages at the queue level based on attributes to reduce unnecessary polling by services.
2. Monitoring and Alerts: Setting up monitoring with AWS CloudWatch to track queue metrics and trigger alerts for anomalies.
3. Retry Logic: Implementing exponential backoff for message retries in case of transient errors.

This architecture showcases the power of AWS SQS in enabling asynchronous, scalable, and fault-tolerant communication for modern backend systems. Let me know if you’ve implemented something similar or have suggestions to enhance this setup!