Reducing Response Payload size when working with AWS API gateway & Lambda

When working with AWS API gateway and Lambda based Microservice Architect we must incorporate many best practices like Exception handling, Security, and performance tuning. Apart from these some of them belongs to AWS service hard and soft limit like Payload size, timeout, and number of API request.

We have faced the similar challenge in our application when returning the response data having the size more than 6MB but less than 15MB . The API was failing because the AWS API gateway has the response size hard limit of 10MB and AWS Lambda only support up to 6MB response size, which obviously cannot be modify.

Although there are patterns to handle the same as API Pagination and S3 pre signed URL, but they require a code change in both Front end as well as Back end. To solve this challenge, we have found that API Gateway and Lambda both support to send GZIP response, which can be developed using custom response builder and can reduce the response size up to 70%. This solution significantly reduce the size of the response payload, making it faster to transmit over the network and supported by all the browser and mobile client.

Step by step Solution

To Develop a PoC, we have used AWS API-Gateway, Lambda, Nodejs, Serverless Framework, faker.js, zlib and Postman. Please configure the AWS secret key and access key as mentioned here. Please follow the steps to develop two end points one with compression and one without the compression.

1. Create a new Project

Create a new serverless project using command:

serverless create --template aws-nodejs --path node-lambda-gzip-api

2. Create API Endpoints

Open serverless.yml file, create below two GET API:

functions:
  apiResponse:
    handler: apiHandler.apiResponse
    timeout: 30
    events:
      - http:
          path: api
          method: get
          cors: true
  
  gzipResponse:
    handler: apiHandler.gzipAPIResponse
    timeout: 30
    events:
      - http:
          path: gzip
          method: get
          cors: true

3. Create a Reponses Payload

We have used the faker library to generate the random response data. Install faker package using the below command:

npm install @faker-js/faker

create a new file payload.js and build a random payload using faker package

const { faker } = require("@faker-js/faker");

exports.responsePayload = function () {
  const resultArr = [];
  var resultObj = {};

  for (let i = 0; i < 10000; i++) {
    let randomName = faker.name.fullName();
    let randomEmail = faker.internet.email();
    let id = faker.datatype.uuid();
    let avatar = faker.image.avatar();
    let birthdate = faker.date.birthdate();
    let firstName = faker.name.firstName();
    let lastName = faker.name.lastName();

    resultObj.name = randomName;
    resultObj.email = randomEmail;
    resultObj.id = id;
    resultObj.avatar = avatar;
    resultObj.birthdate = birthdate;
    resultObj.firstName = firstName;
    resultObj.lastName = lastName;

    resultArr.push(resultObj);
  }

  return resultArr;
};

4. Create a GZIP response Builder

We have used zlib module to compressed the response payload. Install zlib package using below command:

npm install zlib

Create a file, gzipResponseBuilder.js to send GZIP response:

Sending a GZIP response consists of five steps:

a. Compress the response data using gzip module.

b. Convert the GZIP response data in Base64 String.

c. Set isBase64Encoded:true in response.

d. Set Content-Encoding: gzip in response header.

e. Set the base64 data in response.

const { gzipSync } = require("zlib");


exports.buildResponse = function(statusCd,responseData){
  const data = JSON.stringify(responseData);
  const gzip = gzipSync(data);
  const base64String = gzip.toString('base64'); 
  let response = {
    isBase64Encoded: true,
    statusCode: statusCd,
    headers: {
      "Access-Control-Allow-Origin":"*",
      "Access-Control-Allow-Methods": "GET,OPTIONS,POST",
      "Access-Control-Allow-Headers": "Origin, Content-Type, X-Auth-Token",
      'Content-Type': 'application/json',
      'Content-Encoding': 'gzip',
    },
    body: base64String,
  };
   return response;
};

5. Create API Endpoints Implementation

Open apiHandler.js , write the implementation code for both endpoints with and without response compression.

const gzipResponseBuilder = require("./gzipResponseBuilder.js");
const payload = require("./payload.js");

module.exports.apiResponse = async () => {
  const data = payload.responsePayload();
  return {
    statusCode: 200,
    body: JSON.stringify(data),
  };
};

module.exports.gzipAPIResponse = async () => {
  const data = payload.responsePayload();
  return gzipResponseBuilder.buildResponse(200, data);
};

Observe that we are passing the same payload data as parameter for both functions.

6. Building and Deploying the code

Use Serverless command to deploy the code:

Serverless deploy

7 . Testing and result

a. Without compression API: response size was 2.79MB.

b. With compression API: response size was 13.33 KB.

Summary:

In this blog post we have learned the step by step approach to build the PoC using AWS API gateway and Lambda to implement the GZIP compression for the API.

By the test result, it is clear that we can leverage this response compression approach to reduce the response payload size and could achieve up to 70% compression. As mentioned earlier all browser and mobile client has built in capability to unzip the response thus require no change at the front end.

Source Code:

Please find the code here for Nodejs and Java.