As Kickstarter moves from a monolithic application towards a service oriented architecture, we needed to develop a fast and secure way for a service to programmatically retrieve API tokens, passwords, keys, etc. We are currently managing our containers using Amazon’s ECS (Elastic Container Service), which we like for its seamless scalability of Docker containers, and really wanted to secure the application’s access to sensitive information.
We looked at a few tools and even built some ourselves, but we were instantly drawn to Hashicorp’s Vault. Not only is Vault an open-source project that is well maintained, it is also very secure: it uses AES-GCM-256, which is considered to be state of the art for data encryption and TLS 1.2 for data in transit to client. Since the project is entirely open source, it also benefits from the developer community’s scrutiny.
Automate, automate, automate…
One of the problems we faced was to figure out how a container would authenticate itself with our Vault server and automate the entire process. It was quite a relief to see that Hashicorp released a new AppRole authentication backend less than a month ago geared towards machines and services. An AppRole represents a set of login constraints, and the scope of the constraints can be completely customized for each application with specific access control lists (ACLs). We decided to go one step further and also added another constraint: the ECS task role, which can be queried from within the container using the following command:
We built an AWS Lambda whose sole job is to manage these AppRoles in Vault when a resource is created, updated, or deleted with CloudFormation. Once a Docker container starts up, we set up an ENTRYPOINT script that uses the credentials set by the Lambda function to retrieve a Vault token and access all the app-specific secrets in Vault. To make our lives even easier, Hashicorp built a nifty tool called Envconsul (initially built for Consul key/value pairs but now also with added Vault integration). Envconsul provides a convenient way to populate secrets inside the container, and with one line in our entrypoint script we securely set all our secrets inside the container:
export $(envconsul -once -config="/envconsul-config.hcl" env | xargs)
Here is a schematic representation of our current workflow:
We hope this pragmatic approach to secret management within containers is helpful for your infrastructure, and as always, please don’t be shy and feel free to email me at email@example.com with any questions!
Special Thanks to teammate Kyle Burckhard for bringing his Docker expertise into the equation.