AWS CDK : Structuring for re-use
CDK is a framework for modeling your infrastructure in a programming language of your choice ( from the currently supported list ). We are all familiar with the idea of using some sort of infrastructure as Code framework these days to provision cloud infrastructure ; Terraform , AWS CloudFormation, ARM templates, Pulumi to name a few.
As always, my thought process around learning or using a framework was to find the knobs I can move or squeeze to make my life easy with re-usability built in. The whole idea of an infrastructure modeling framework and a programming language mixed in gives you a lot of freedom to do what you want. This is my perspective and you may find better options which are suited for your needs.
- Language of choice : Python
- Reference libraries: https://docs.aws.amazon.com/cdk/api/latest/python/modules.html
- Reference stack : A simple serverless stack with Api Gateway, SQS, Lambda and a DynamoDb.
Some key terms:
- Construct: Constructs are the basic building blocks of AWS CDK apps. A construct represents a “cloud component” and encapsulates everything AWS CloudFormation needs to create the component.
- Stack: The unit of deployment in the AWS CDK is called a stack. All AWS resources defined within the scope of a stack, either directly or indirectly, are provisioned as a single unit.
Pre-reqs:
- You have a basic understanding of AWS CloudFormation or some other IAC framework.
- You have some understanding of the knobs available within a cloud resource.
Iteration 1: Mix all of it into a single construct or stack
I didn’t have a clue how to get started and so there it was. I started referencing the aws_cdk modules listed and added the resources one by one into a single stack.
CDK synthesizes my code and creates a valid CloudFormation json as the output.
Iteration 2: Separate the resources into constructs of their own
The keyword here is constructs. I like to think of them as a logical grouping of resources you want to use in your application stack.
You can reference these into Application stack as as python modules which you have created to give you some flexibility. You can argue with my above separation of modules or constructs about integrations between the queue and gateway and it is correct. I made some mistakes along the way thinking this would be the easier way to handle it and it turned out to be not.
So how do you reference them in a Stack ?
Iteration 3: Remove hardcoded values in constructs
Well, as I built the constructs, they were chained to some data like the name of the function, path to the function code, tags to be used and so on.
Welcome contexts. I started using contexts as a way to inject data into my constructs. ( db_context in one of the pictures above). There are a few of them in CDK and I started using “cdk.json”. How do I use it ? I may have gone down the rabbit hole on some of the customizations I wanted to do with a construct.
How do they translate back to the construct ? CDK gives you a nifty “self.node.try_get_context()” function to reference the values in your context json file.
This brings in no changes to App stack which uses this module.
Iteration 4: Make constructs generic and put them in a repo
Even with the context usage, I was still using some integrations in my constructs like ApiGateway/Lambda integration and I needed another construct to handle a standalone lambda. Keep It Simple, right ?
How do we get the integrations right once you separate them into resource based constructs? You can exposure them as properties to use from a python module.
That brings in a question: How do I handle all the integrations ?
- ApiGateway-> Lambda( some interim lambda)-> SQS -> Lambda -> DynamoDb
- I created an application integration construct which stitches them together. The input as you see below is references from each of the constructs.
The corresponding app.py
Final change: Move the constructs you build into a separate repo and make it part of your requirements.txt while starting your virtual environment.
Final Thoughts:
What I ended up with :
- The library of constructs helps me to define some of the good practices I would include when I create some of these resources.
- You can always set the constructs as python modules and make them available through some artifact store.
- There are a few experimental constructs which AWS is publishing under solution constructs name which may be useful to start here
- Python documentation and examples were sometimes incorrect and I had to refer the typescript variants to get the correct path forward.
- AWS CDK version is fast moving. The cli version has bumped up 5–6 minor versions in a two week span.