Guide to Querying and Pagination with AWS DynamoDB in NestJS

Introduction

In my previous article, I provided a basic guide on initializing and using AWS DynamoDB, but in this article, we will delve deeper into QueryCommand.

In AWS DynamoDB, the Query method allows you to search for data based on the primary key (Partition Key) and filter conditions (Sort Key). To optimize performance and cost, DynamoDB supports a Pagination mechanism through the ExclusiveStartKey and LastEvaluatedKey parameters. Applying pagination not only reduces the data transfer load but also enables the application to handle large data tables smoothly, ensuring system stability.

Detail

Use AWS CDK to create the file lib/dynamodb-gsi-stack.ts

import * as cdk from "aws-cdk-lib"
import * as dynamodb from "aws-cdk-lib/aws-dynamodb"
import { Construct } from "constructs"

export class DynamoDbGsiStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props)

    const orderTable = new dynamodb.Table(this, "ProductTable", {
      tableName: "Products",
      partitionKey: { name: "id", type: dynamodb.AttributeType.STRING },
      sortKey: { name: "createdDate", type: dynamodb.AttributeType.STRING },
      billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
      removalPolicy: cdk.RemovalPolicy.DESTROY,
    })

    orderTable.addGlobalSecondaryIndex({
      indexName: "StatusIndex",
      partitionKey: { name: "status", type: dynamodb.AttributeType.STRING },
      sortKey: { name: "createdDate", type: dynamodb.AttributeType.STRING },
      projectionType: dynamodb.ProjectionType.ALL,
    })
  }
}

By default, DynamoDB only allows you to query efficiently based on id. If you want to find "All products with status 'DELIVERED'", you would have to scan the entire table (Scan), which is very costly.

Adding StatusIndex (GSI) helps filter the list of products by status (e.g., Active, Pending, Deleted) without scanning the entire database.

indexName: StatusIndex. This is the index name to use when querying.
partitionKey (of Index): status. Helps query extremely fast based on the status column.
sortKey (of Index): createdDate. Helps retrieve the list of products by status but still sorted by time (e.g., getting the latest pending orders).
projectionType: ALL. This means that when querying on this Index, DynamoDB will return all fields (attributes) of that item, just like querying the main table.

Update the file bin/aws-cdk.ts

#!/usr/bin/env node
import 'dotenv/config';
import * as cdk from "aws-cdk-lib/core"
import { DynamoDbGsiStack } from '../lib/dynamodb-gsi-stack';

const app = new cdk.App()
new DynamoDbGsiStack(app, "DynamoDbGsiStack")

In the NestJS source code, please update the file dynamodb.service.ts

import {DynamoDBClient} from '@aws-sdk/client-dynamodb'
import {
  DynamoDBDocumentClient,
  QueryCommand,
} from '@aws-sdk/lib-dynamodb'
import {Injectable} from '@nestjs/common'
import {v4 as uuidv4} from 'uuid'

@Injectable()
export class DynamoDBService {
  private readonly docClient: DynamoDBDocumentClient
  private readonly tableName = 'Products'

  constructor() {
    const client = new DynamoDBClient()
    this.docClient = DynamoDBDocumentClient.from(client)
  }

  async queryProduct(query: {id: string; startDate: string; endDate: string}) {
    const {id, startDate, endDate} = query
    const command = new QueryCommand({
      TableName: this.tableName,
      KeyConditionExpression:
        'id = :uid AND createdDate BETWEEN :start AND :end',
      ExpressionAttributeValues: {
        ':uid': id,
        ':start': startDate,
        ':end': endDate,
      },
    })
    const response = await this.docClient.send(command)
    return response.Items
  }

  async queryProductByStatus(status: string) {
    const command = new QueryCommand({
      TableName: this.tableName,
      IndexName: 'StatusIndex',
      KeyConditionExpression: '#s = :status',
      ExpressionAttributeNames: {
        '#s': 'status',
      },
      ExpressionAttributeValues: {
        ':status': status,
      },
    })
    const response = await this.docClient.send(command)
    return response.Items
  }

  async queryProductsWithPagination(query: {
    id: string
    limit: number
    lastKey: string
  }) {
    const {id, limit, lastKey} = query
    const command = new QueryCommand({
      TableName: this.tableName,
      KeyConditionExpression: 'id = :id',
      ExpressionAttributeValues: {
        ':id': id,
      },
      Limit: +limit,
      ExclusiveStartKey: lastKey
        ? JSON.parse(Buffer.from(lastKey, 'base64').toString())
        : undefined,
    })
    const response = await this.docClient.send(command)
    const nextToken = response.LastEvaluatedKey
      ? Buffer.from(JSON.stringify(response.LastEvaluatedKey)).toString(
          'base64'
        )
      : null
    return {
      items: response.Items,
      nextToken,
    }
  }
}

The code above includes 3 main functions for processing data:

queryProduct: Executes a product query by ID and a time range (BETWEEN) of the creation date.
queryProductByStatus: Uses a Global Secondary Index (GSI) named 'StatusIndex' to search by status. Note the use of ExpressionAttributeNames to avoid conflicts with the system keyword "status".
queryProductsWithPagination: Implements pagination. The function receives a 'limit' to restrict the quantity and 'lastKey' (in Base64 form). If 'lastKey' exists, it decodes it to put into ExclusiveStartKey to retrieve the next page, then encodes the new LastEvaluatedKey into 'nextToken' to return to the Client.

Update the file dynamodb.controller.ts to use DynamoDBService

import {
  Controller,
  Get,
  Query,
} from '@nestjs/common'
import {DynamoDBService} from 'src/service/dynamodb.service'

@Controller('dynamodb')
export class DynamoDBController {
  constructor(private readonly dynamoDBService: DynamoDBService) {}

  @Get('query')
  queryAll(@Query() query: {id: string; startDate: string; endDate: string}) {
    return this.dynamoDBService.queryProduct(query)
  }

  @Get('query-by-status')
  queryByStatus(@Query('status') status: string) {
    return this.dynamoDBService.queryProductByStatus(status)
  }

  @Get('query-with-pagination')
  queryWithPagination(@Query() query: {id: string; limit: number; lastKey: string}) {
    return this.dynamoDBService.queryProductsWithPagination(query)
  }
}