Leveraging Functional Programming & AWS Lambda to Drive ChatOps


In Operations Engineering at Kickstarter, we aim to ensure the health and sustained development of our website. A large part of that is making sure our engineers are getting the right information in a timely matter. For this reason we have started to embrace ChatOps. In this post, I will detail how Kickstarter is leveraging our use of AWS Lambda and embracing functional programming to drive our ChatOps infrastructure, and through that infrastructure drive development through conversation.

Our developers often use AWS CloudFormation templates. We had the idea to use these templates to spin up infrastructure and then send the events as those resources are created into Slack, where team members could watch and talk about their progress. We also rely on alerts about our RDS instances. In fact, we have many different alerts we’d like to be informed about and discuss in real time. What if every time one of these alerts were triggered, they magically appeared in dedicated Slack channels where our developers could discuss them? That was the problem we set out to solve.

What is ChatOps?

ChatOps can be thought of as conversation-driven development. Chat rooms that use ChatOps can either be a combination of tools and humans talking, or they can be “bot-only” rooms that provide convenient logs of information. As a log, ChatOps can also help coworkers see what commands have been run recently that impact infrastructure and watch the result of that command together.

This is how the desired ChatOps flow for CloudFormation notifications in Slack works: a user creates, updates, or deletes a CloudFormation stack that is configured with the SNS role for our function. If the stack’s assigned policy is configured to that notification role, a function subscribed to this notification will be triggered. When the function hears the incoming event, it processes information and sends a POST request to the Slack webhook for the appropriate channel (in this case, #cloudformation-events). Then the Slack webhook starts receiving requests and posting to the channel where users now see logs of the impact of the event on their stack.

AWS Lambda & Serverless Infrastructure

So how do we achieve the magic above with as simple and flexible code as possible? We do it “serverless,” using AWS Lambda and pure functional programming (or, at least, as pure as we can get).

The serverless movement attempts to abstract users away from servers and infrastructure. Of course, serverless infrastructure is a bit of an oxymoron. The point of the movement is to focus on the compute time and logs for single functions or processes, without thinking about the server at large. While many people are hopeful for a future where all infrastructure is serverless, we are starting with a few components of our infrastructure that can be broken out into functions. There are many single-purpose services where we could apply serverless infrastructure. One of those is sending alerts through chat.

Lambda is an AWS service that launched at the end of 2014. Through Lambda, we can define functions that perform small operations and have AWS provisions and run them. When thinking about whether a certain task is fit for Lambda, a person should ask themselves if what they need is a simple one-off function that will reliably run in the same way and use the same small amount of resources every time. In the same way we use AWS S3 for asset uploads or auth management services for our credentials, we can use Lambda to manage our functions. For us, that means functions to fetch and post data. Lambdas are simple: you define a function that performs an operation, and AWS provisions and schedules that function.

Here is how we organize our function for ChatOps: first, we define the function and its dependencies in function.json (necessary for our deployment process) and package.json, respectively.

#function.json
{
"name": "slack-integration",
"description": "Pushes SNS Notifications to slack.",
"role": EXECUTION_ROLE,
"memory": 128,
"handler": "index.handler",
"defaultEnvironment": "ops",
"timeout": 9
}

We control the crux of our function in an index file in the root. You can contain all the code for your function in this file.

There are a couple ways to deploy your Lambda. We use a service called Apex, which allows us to deploy from the command line easily and offers several options for environment variables. You can also deploy your function code straight from the AWS console.

Functional Programming

Once we have our Lambda configuration setup, we can start to break up our code, test it, and make it efficient. Functional programming is a prime candidate for this. Before joining the Ops team, I had never used functional Javascript or AWS Lambda. Coming at them simultaneously worked incredibly well for my brain.

Functional programming is based on the premise of computation of data. Data comes in, data flows out, there are no side effects. This incorporates well with Lambda’s push/pull model. An event producer directly calls the function and pulls the update from the data stream to invoke the function. Pure functions and immutable variables also allow for lower complexity of programs. Here is a part of our function that formats the incoming data:

module.exports = (event) => {
return [event]
.filter(slowLogMessageFilter)
.flatMap(mapEventToWebHooks)
.map( (event) => {
return {
hostname : 'hooks.slack.com',
path : event.webhook,
method : ‘POST’,
body : subjectParsers[event.subject](event)
};
});
};

As the data flows through, we apply filters and maps to functions that format and push the data to the next function. The concept of using functional programming to process events seems natural. The definition of a pure function is a function that, given the same parameters, will always return the same result. So when we have technology that processes incoming events (either triggered or scheduled) over and over, functional programming is a perfect fit.

Dependency Injection

Testing is necessary for any function! However, when depending on calls to webhooks like Slack, it can be difficult. In order to test our various functions, we can use dependency injection. Instead of using hardcoded dependencies, we can pass the dependencies (mocked out) into our module. That makes testing much easier. For example, we don’t have to deal with this POST to the webhook repeatedly by doing this:

'use strict';
const defaultDependencies = require('./lib/defaultDependencies');
const slackManager = require('./lib/slackManager');
exports.handler = (event, context) => {
const dependencies = defaultDependencies;
// pass our dependency to postToSlack into the SlackManager
const run = slackManager(dependencies);
return run(event, context);
};

Then in defaultDependencies, we simply require the postToSlack method:

'use strict';
const postToSlack = require('./postToSlack');
module.exports = { postToSlack: postToSlack };

Now we can test the other functions without worrying about implicit execution of external code. Another benefit to this is that the postToSlack becomes agnostic to which channels or what data it is posting. We can thus send payloads with different information for different channels. Using one function, we can process CloudFormation events, RDS events, and many more without duplicating any code. It’s simple and, given the same input, guaranteed to work the same way every time.

What’s Next

There are several steps to moving forward with ChatOps. We are currently working on the end-to-end user experience for ChatOps so it is fluid with our use of Cog by Operable. For more information about what we are doing with Cog, check out Operations Manager Aaron Suggs’ conversation with Operable from last month. We are also working on adding tagged alerts so that only developers in charge of certain infrastructure get alerted in these channels. Finally, we are looking for more ways to embrace functional programming throughout our alerting systems and code bases.



Source link