Evolving the Slack API – Several People Are Coding


You know how to design a good API, but how do you evolve that API when it’s time to make changes? We’ve faced this with each major feature release here at Slack over the past few years, most recently while working on the highly anticipated Shared Channels feature. Using shared channels, two organizations connect directly in Slack and work together from the comfort of their own workspaces.

The idea seems simple, but building it wasn’t easy. From a user’s perspective, a shared channel looks and feels just like any other channel. Under the hood, shared channels fundamentally changed the relationship between workspaces, channels, and the web API.

Let’s take a look at how the backend design for channels has evolved over time, and how the release of shared channels turned into an opportunity to build the Conversations API, a unified interface for all types of channels.

The Evolution of Channels

In the beginning when channels had a 1:1 relationship with a workspace, our backend database tables had a close relationship to the web API. Each table had corresponding API endpoints, and each API endpoint had a single corresponding OAuth scope:

Diagram of database tables before Shared Channels, along with the API endpoint for each table

But Shared Channels were different. We merged them into the public channels table, because they needed a shared channel ID across workspaces. This was for two reasons:

  • Shared channels need to share resources (pins, files, reactions, and more)
  • A shared channel can be private for one team and public for another
Diagram of tables, along with the OAuth scope of data stored in each table.

Here’s where the problem arose for the web API: Using OAuth, customers had explicitly granted permission to public channels (channels:read), private channels (groups:read), or DMs (im:read). But with the merged data in the channels table, we were not able to control data access the same way. Any app installed with channels:read scope was about to be granted access to private shared channels as well.

A New API Design

When we considered what was best for both the Shared Channels feature and our developer ecosystem, we found that there was significant upside in creating a brand new unified interface for all types of channels: the Conversations API.

At the top of our minds was how we were going to maintain backwards compatibility for all the developers who had already built apps using our web API. A new suite of endpoints would allow developers to opt-in when they were ready to update their code for shared channels. A new API would also allow us to address major developer pain points that had emerged over the years — things we really wanted to fix, but couldn’t — because of backwards compatibility.

For the rest of this article, we’ll be looking at the major changes in the new API:

  • Smarter OAuth scope resolution
  • Simplified arguments
  • Better performance for large payloads
  • Strict JSON Schema validation

Smarter OAuth Scope Resolution

Previously, developers had to know whether a channel was public or private before calling channels.info or groups.info. With the new API, developers no longer have to know the type of channel before calling conversations.info for any channel type. At the beginning of a request to any Conversations API endpoint, we now compare the type of channel requested with the OAuth scope of the requester’s token. If the requested channel type is not authorized, the developer will get a missing_scope error. Developers can now remove app logic that had been built to map a fragmented model of channels to OAuth scopes and API endpoints.

Simplified Arguments

Our original channel API endpoints had been updated over time as our product evolved, and as a result, dramatic inconsistencies had developed. One endpoint might have a channel argument that required an ID, whereas another endpoint with the same argument might require a channel name instead. We fixed these in the new API so that argument and types are consistent across the whole suite.

To make the Conversations API as easy to use as possible, we also reduced the number of arguments whenever we saw a strong trend in historical requests. For example, all requests to channels.rename used the non-default argument validate=false. As a result, we made validate=false the new default behavior for conversations.rename.

In some cases, we changed the arguments to improve performance and scalability for apps. For example, channels.info used to include a nested list of members, but as larger and larger companies started to use Slack, response payloads grew, in some cases to as much as 290MB for a single channel. In addition, 72% of requests to channels.info specified no_members=true. To help apps scale to large teams, we now exclude members by default for conversations.info. To provide the channel membership data in a more optimal format to developers, we added a new conversations.members endpoint with pagination.

Historical Response Payload Sizes for channels.info since October 2015. Request timeouts also spiked in January and February 2017.

Better Performance for Large Payloads

In the early days, we didn’t anticipate that workspaces would have over 10,000 channels, or that channels would have over 150,000 members. Endpoints that were never intended to be paginated grew to unreasonable proportions! Performance bottlenecks and timeouts hindered developers’ ability to scale their apps, and timeouts for large enterprises became frequent.

Even before Slack released the Conversations API, the Platform team had already started to upgrade older endpoints to a new cursor-based pagination paradigm. However, some older endpoints were using a legacy form of pagination that required suboptimal database queries. We could never fully transition any endpoint that had used the legacy pagination, because we did not want to break any apps that were already using the older request format. For those endpoints, the old and new pagination had to exist side-by-side, which meant that we still experienced undue load on our databases and delivered suboptimal performance to a portion of app users.

In the Conversations API, we committed fully to cursor-based pagination so that apps could scale alongside workspaces — whether or not they used shared channels.

Strict JSON Schema Validation

Previously, Slack backend engineers could update core libraries and accidentally change our downstream API output. App developers would discover bugs before we did when their apps broke from API inconsistencies. Additionally, we had response fields that were assumed false or null when unset, which worked for JavaScript developers but was painful for almost everybody else.

Before Shared Channels, we had already started adding JSON schema validation using the json-schema-rspec gem in our automated RSpec tests to increase reliability in our web API. Since we wanted to make the Conversations API reliable, we designed the endpoints up front and wrote schema validation tests in parallel to feature development. These tests add reliability to every deploy. As a bonus, we used the schema to add Open API specs for these endpoints.



Source link