How to handle importing CloudFormation resources when you use the CDK

CloudFormation allows you to import resources into a Stack, in case the resource was initially created outside a stack, but you'd still like to manage it using code going forward. If you use the CDK to generate your CloudFormation templates, then you may need to know how to handle this appropriately inside your code.

First, not all resources can be imported. Refer to these docs for what can.

To illustrate the process, let's start with the simplest example, using a DynamoDB Table:

export class MyStack extends Stack {
  constructor(scope: Construct, id: string, props: StackProps = {}) {
    super(scope, id, props);

    const table = new Table(this, 'JustSomeTable', {
      partitionKey: {
        name: 'PK',
        type: AttributeType.STRING,
      },
    });

    const handler = new Function(this, 'SomeFunction', {
      handler: 'handler',
      runtime: Runtime.PYTHON_3_8,
      code: Code.fromInline('cant be empty'),
      environment: {
        TABLE_NAME: table.tableName,
      },
    });
    table.grantReadData(handler);

  }
}

Here I have a Table, and a Lambda function. The only reason for the Function is to keep a reference to the Table. There will be some swapping around of resources so having this in my example helps keep me honest (if there was nothing referencing this Table the work would be a little simpler, but that's not how things normally work, you've always got references to other things).

The Use Case

Now let's do something that I've seen people asking about on the cdk.dev Slack channel and other forums, let's restore the Table from a backup. Perhaps we've got a lower environment, one where we want our Table managed by IaC, but we want its contents to be a restoration from a higher environment, like QA.

The assumption here is that we've already had this Stack created in an AWS account, so the Table and Function exist in the account. We now are going to manually restore a new Table from a backup of the existing one. Then we'll make some changes and iterations through our CDK code, applying some changes to the stack multiple times until we're finished and have a stack and related resources entirely under IaC control again.

A Clean Diff

Before we start this process we want to make sure our CDK code and our CloudFormation Stack are matching, we wouldn't want any unexpected changes. This can be done with a simple cdk diff command:

cdk diff

We should see no diff between our current code and the CloudFormation Stack.

Stack my-stack-dev
There were no differences

Pre-req's

Next we're going to make two changes that shouldn't directly affect any resources. First is setting the Table to have a retain policy on removal. This is required for the CloudFormation import process. The Table resource defaults a Retain policy, but you can override if you're using a different resource or just want to make sure:

const table = new Table(this, 'JustSomeTable', {
  partitionKey: {
    name: 'PK',
    type: AttributeType.STRING,
  },
});
table.applyRemovalPolicy(RemovalPolicy.RETAIN);

We also need to turn off the Metadata output of the CDK:

export class MyStack extends Stack {
  constructor(scope: Construct, id: string, props: StackProps = {}) {
    super(scope, id, { ...props, analyticsReporting: false });
    // ...
  }
}

Now the synthesized template won't have any changing Metadata records. It goes from this:

  CDKMetadata:
    Type: AWS::CDK::Metadata
    Properties:
      Analytics: v2:deflate64:H4sIAAAAAAAAAzWLyQrCMBRFv6X75NmIIq4FPyB1Ly+DkDYDZFAk5N9tja7O4VzuHkYYaZTAzgO+EpVqodYIqFNGuZA13at6e3RBrfGGwmpyefivNGLRCYvQr8XLbILfpr83YtBB5aE/NrZGuE6hRNnTzxvxQWmY0+7JDsBOcBzmZAyNxWfjNPDOD9ehvkirAAAA
    Metadata:
      aws:cdk:path: my-stack-dev/CDKMetadata/Default

to this:

  CDKMetadata:
    Type: AWS::CDK::Metadata
    Properties:
      Modules: aws-cdk=1.120.0

When importing resources CloudFormation requires that no other resources be changing. To ensure that the CDKMetadata resource doesn't change we must disable this analytics flag so the Analytics property, which changes frequently, doesn't cause issues.

Now go ahead and deploy these changes out:

cdk deploy

(or however you normally deploy)

Restoring the Table

We're going to now restore a table snapshot to a new table. This can be done manually through the UI, or you can make it happen through API calls. The CloudFormation Table resource (AWS::DynamoDB::Table) doesn't support providing a snapshot identifier to restore from, but other resources might, in which case you don't really need to be importing anything and you can stop reading right now.

You can refer to the AWS docs for more details on restoring a DynamoDB table. In this case, I'll just use the CLI:

aws dynamodb restore-table-from-backup \
--target-table-name MyRestoredTable \
--backup-arn arn:aws:dynamodb:us-east-1:123456789012:table/JustSomeTable/backup/01489173575360-b308cd7d 

Referencing the new Table

Now that the table is restored, you need to start using it. While this isn't exactly necessary in the import process, it's worth mentioning how you'd use it. In our use-case we just get a reference using a .fromTableArn (or similar) to the Table like we would anything else not managed by the stack:

    const table = new Table(this, 'JustSomeTable', {
  partitionKey: {
    name: 'PK',
    type: AttributeType.STRING,
  },
});
table.applyRemovalPolicy(RemovalPolicy.RETAIN);

const newTable = Table.fromTableArn(this, 'NewTable', 'arn:aws:dynamodb:us-east-1:123456789012:table/MyRestoredTable');

const handler = new Function(this, 'SomeFunction', {
  handler: 'handler',
  runtime: Runtime.PYTHON_3_8,
  code: Code.fromInline('cant be empty'),
  environment: {
    TABLE_NAME: newTable.tableName,
  },
});
newTable.grantReadData(handler);

The references in the Lambda function are updated and now we can deploy out the change and allow the Lambda to access the restored table with a cdk deploy.

Removing the Original Table

Before we can import the new Table into the Stack, we must remove the original table resource. However, we are going to want it back in a moment, so we'll just comment it for now:

    // const table = new Table(this, 'JustSomeTable', {
    //   partitionKey: {
    //     name: 'PK',
    //     type: AttributeType.STRING,
    //   },
    // });
    // table.applyRemovalPolicy(RemovalPolicy.RETAIN);

const newTable = Table.fromTableArn(this, 'NewTable', 'arn:aws:dynamodb:us-east-1:123456789012:table/MyRestoredTable');

const handler = new Function(this, 'SomeFunction', {
  handler: 'handler',
  runtime: Runtime.PYTHON_3_8,
  code: Code.fromInline('cant be empty'),
  environment: {
    TABLE_NAME: newTable.tableName,
  },
});
newTable.grantReadData(handler);

And then another deploy with cdk deploy. Keep in mind that because of the Retain the Table will not be removed from DynamoDB, only from the CloudFormation Stack. You will need to manually delete this table from DynamoDB when you're ready.

Importing the New Table

Now we can finally import the table! Start by un-commenting out the Table:

    const table = new Table(this, 'JustSomeTable', {
  partitionKey: {
    name: 'PK',
    type: AttributeType.STRING,
  },
});
table.applyRemovalPolicy(RemovalPolicy.RETAIN);

const newTable = Table.fromTableArn(this, 'NewTable', 'arn:aws:dynamodb:us-east-1:123456789012:table/MyRestoredTable');

const handler = new Function(this, 'SomeFunction', {
  handler: 'handler',
  runtime: Runtime.PYTHON_3_8,
  code: Code.fromInline('cant be empty'),
  environment: {
    TABLE_NAME: newTable.tableName,
  },
});
newTable.grantReadData(handler);

This gives us the resource we can now import. This is the only change that we'll make in the code. Again, this is a CloudFormation requirement, that when importing a resource into an existing stack you ONLY import that resource (or resources) and don't make any changes to anything else. Notice we're leaving the Lambda function referencing the Table that is from the .fromTableArn call.

We can't use the cdk deploy for this step. It doesn't have any way of doing an import routine with CloudFormation. So, we'll go back to other tools. We start by synthesizing the template with the Table added back with a cdk synth.

Once you have the template, you can either use the UI console, or the API, to run a change-set and import the Table. Refer to the AWS docs for more details.

Run the stack change-set through and the table restored from a snapshot is now tied back into your CDK code.

Cleanup

Finally, we just need to change the references in the CDK code. The handler goes back to use the 'table' reference and we remove the Table that came from the .fromTableArn lookup:

const table = new Table(this, 'JustSomeTable', {
  partitionKey: {
    name: 'PK',
    type: AttributeType.STRING,
  },
});
table.applyRemovalPolicy(RemovalPolicy.RETAIN);

const handler = new Function(this, 'SomeFunction', {
  handler: 'handler',
  runtime: Runtime.PYTHON_3_8,
  code: Code.fromInline('cant be empty'),
  environment: {
    TABLE_NAME: table.tableName,
  },
});
table.grantReadData(handler);

This change can be pushed out with cdk deploy like before.

It'd also be good to restore the analytics feature on the stack.

Wrap-up

CloudFormation resource importing is a feature you hopefully never have to use. But when you do, it's pretty easy to incorporate it into your CDK code. Just turn off Metadata temporarily and do a few manual deploys and you're all set.

UPDATE: Originally the code examples had an ARN to the restored table that wasn't consistent throughout the examples. That's been fixed now.