Creating Idempotent Lambda Handlers with DynamoDB

While AWS Serverless is a fantastic platform to develop your application, there are a few gotchas that could cause problems down the line. One such issue is that many systems pass events at least once. This means that while they guarantee you will get the message, it means you might get it more than once. If you aren't careful and mindful about this you can have corrupt or bad data as you reprocess an event and update something else a second, duplicate, time.

Ultimately you will want your Lambda event handlers to be idempotent. This isn't always a trivial change, especially when those two events could be received at nearly the same time. Two separate instances of your Lambda could be invoked. Thanks to DynamoDB this is actually pretty easy to add into any existing handler without having to change any existing data models.

A lot of the explanation is already laid out well here by Michael Wittig, take a quick look if you're unfamiliar, it's a good explanation. The only issue I saw with it was it never talks about cleaning up these records. Thanks to TTL, this is actually quite easy:

const AWS = require('aws-sdk');
const ddb = new AWS.DynamoDB.DocumentClient();
module.exports.handler = async (event, context) => {
  console.log({event, context});
  for (let record of event.Records) {
    let messageId = record.messageId;
    let params = {TableName: process.env.IDEM_CHECK_TABLE_NAME, Key: {messageId}};
    let creationTime = Math.floor(Date.now() / 1000);
    const fiveSeconds = creationTime + 5;
    try {
      await ddb.put({
        ...params,
        ConditionExpression: 'attribute_not_exists(messageId)',
        Item: {messageId, ttl: fiveSeconds, creationTime}
      }).promise();
    } catch (err) {
      console.warn(`Error when writing idemotency record, probably because you've already tried to process this message`, err);
    }
    console.log('Going to process the record now, will be reprocessable in', fiveSeconds, record);
  }
};

This is an example when using SQS as the Event Source. Generally speaking, this code is similar to what Mr. Wittig setup. This code assumes the table was created with a TTL attribute called 'ttl'. Here I set it to 5 seconds in the future. This should keep the table fairly clean and shouldn't leave orphaned records around for long.