Easily deploy complex CloudFormation templates with external resources such as Lambdas or Nested Stacks.
Many CloudFormation templates are completely standalone – one single YAML or JSON file and that’s it. Easy to deploy. However in some cases CFN templates refer to other files, or artifacts. For example Lambda source or ZIP file, nested CloudFormation Template file, or an API definition for API Gateway may be such “artifacts”. These files have to be available in S3 before we can deploy the main CloudFormation template.
Deploying such complex stacks is a multi-stage process, usually performed using a custom shell script or a custom Ansible playbook.
- ZIP up the Lambda source and required libraries
- Upload the ZIP file to S3
- Create CloudFormation stack with the correct path to the S3
Not a rocket science but still…
Fortunately AWS-CLI provides a very convenient method for deploying CloudFormation templates that refer to other files. Read on to learn how.
Sample project structure
Our little project has these four files:
~/cfn-package-deploy $ tree -F.
1 directory, 4 files
One Cloud Formation template, one simple single-file Lambda function and one more complex Lambda that consists of multiple files.
Refer to local files in your CFN template
Traditionally we would have to zip up and upload all the lambda sources to S3 first and then in the template refer to these S3 locations. Perhaps through stack parameters.
aws cloudformation package we can refer to the local files directly. That’s much more convenient!
Have a look at this LambdaOne snippet for example – we refer to the
lambda_one.py file locally, as it’s in the same directory as the template.
Code: lambda_one.py # <<< This is a local file
Likewise with the more complex LambdaTwo that consists of two files in a subdirectory. Simply refer to the directory name
lambda_two in the template.
Code: lambda_two/ # <<< This is a local directory
Package and upload the artifacts
The next step is calling <code>aws cloudformation package</code> that does three things:
- ZIPs up the local files, one ZIP file per “artifact”.
- Upload them to a designated S3 bucket.
- Generate a new template where the local paths are replaced with the S3 URIs.
Decide on a S3 bucket
First of all we need an S3 bucket where the files will be uploaded. I tend to (ab)use the cf-templates-… buckets that AWS creates when we deploy CFN through the console. But feel free to use any bucket you want.
~/cfn-package-deploy $ aws s3 ls | grep cf-templates
2018-11-07 22:55:23 cf-templates-abcdefghjklm-ap-southeast-2
2019-02-01 10:27:46 cf-templates-abcdefghjklm-ca-central-1
2018-11-02 07:06:25 cf-templates-abcdefghjklm-us-east-1
Let’s use the first one as I’m working in the Sydney region (ap-southeast-2)
Run the package command
~/cfn-package-deploy $ aws cloudformation package \
--template-file template.yml \
--s3-bucket cf-templates-abcdefghjklm-ap-southeast-2 \
Uploading to 35f69109a3a3f87e999f028f03403efa 193 / 193.0 (100.00%)
Uploading to cca5b023ed6603eabf9421471b65d68b 352 / 352.0 (100.00%)
Successfully packaged artifacts and wrote output template to file template.packaged.yml.
Examine the generated files
Let’s have a look at the output template file first.
We will notice that the
Code attributes in LambdaOne and LambdaTwo were updated with the bucket and uploaded object name:
For completeness let’s also look what’s in the uploaded files. From the listing above we know the bucket and object name to download.
~/cfn-package-deploy $ aws s3 cp \
And we know it’s a ZIP file. Even though there is no
.zip extension we can still
~/cfn-package-deploy $ unzip -l cca5b023ed6603eabf9421471b65d68b
Length Date Time Name
--------- ---------- ----- ----
114 2019-02-19 15:55 index.py
35 2019-02-19 15:54 some_module.py
149 2 files
As expect it’s the content of the
Deploy the “packaged” template
~/cfn-package-deploy $ aws cloudformation deploy \
--template-file template.packaged.yml \
Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - cfn-package-deploy
Note that we used the packaged template
template.packaged.yml that refers to the artifacts in S3! Not the original one with local paths!!
We may also have to use
--capabilities CAPABILITY_IAM if there are any IAM Roles in the template – and that’s quite likely. Otherwise deploy fails:
An error occurred (InsufficientCapabilitiesException) when calling the CreateChangeSet operation: Requires capabilities : [CAPABILITY_IAM]
We can also set / override stack parameters with
--parameter-overrides just like when using aws cloudformation create-stack.
aws cloudformation deploy help for the available parameters.
What a convenience!
This is an easy way to create and update stacks with external resources. It works not only with Lambda sources but also with Nested Stacks, AWS::Include, and many other resources that need external files. Refer to <code>aws cludformation package help</code> for details and supported artifact types.
If you liked this article leave us a comment 🙂