Analyses and A/B testing shows that increasing our Quality Signup rate causes an increase in our long-term engagement rate.
The concept of a custom true-north metric is not unique to LinkedIn. In fact, many other companies have similar measures that serve as their true-north metric for growth. For example, Facebook is known to track users who connect with 10 friends in seven days. Pinterest tracks Weekly Active Repinners. And Slack tracks teams who have sent 2,000 messages. For the remainder of this post, we’ll go through the general steps and considerations that you should think about when creating your own custom success metrics. Although this isn’t a foolproof method to solve all your growth problems, it is a great way to start moving in the right direction.
Step 1: Start with value propositions
The first step in creating a custom true-north metric is to understand how your product aligns with the value propositions your business provides. Since metrics serve to measure and quantify a product against an existing goal, they are ineffective without strong alignment to the value propositions. In Step 3, you’ll be using machine learning techniques to objectively quantify this relationship, but it’s important to begin with this solid conceptual understanding.
For Quality Signup: We thought about how member acquisition as a product specifically aligned with LinkedIn’s three core member value propositions:
Step 2: Data collection, label, and features
Once you have a good understanding of how your product aligns with your value propositions, you’ll want to quantify that with data. To start, you need to comb through historical logs to gather samples of what you’re trying to measure. These could be samples of: member actions, contract conversions, file uploads, etc. You then separate positive outcomes and negative outcomes, which will later serve as the label for our machine learning model. Choosing a label that accurately captures your business is probably the most difficult and influential decision in this entire process. When in doubt, you should always refer back to the mantra of growth and ask yourself: “Am I measuring long-term and sustainable value?”
For Quality Signup: We gathered all new members from a six-month cohort as samples. We classified new members who were still active six months after registration as positive outcomes, and those who were not as negative outcomes. By using “active” as the label, we made the assumption that members who were active were the ones who were receiving sufficient value from LinkedIn.
Along with the labeled data, you want to gather all the features of your product that might be useful for predicting whether a sample is a positive outcome or a negative outcome. A good starting point for picking features is to refer back to Step 1 and think about how your product aligns with the business goal. You should include all features that help drive your business value, along with any feature that might only be indirectly related. Each feature should also be measurable within a reasonable time frame. It’s hard to go wrong here and when in doubt, it’s always better to over-gather than to under-gather.
Step 3: Obligatory machine learning step
Once you have your features ready, it’s time to make sense of how they relate to the label in your dataset. Ultimately, you’re trying to decipher the arrangement of features that is most likely to lead to a positive outcome. In the case where you have a large number of samples, you’ll want to take advantage of existing machine learning techniques to simplify this task. Most classification methods will work here, but we usually prefer to start with the simple and powerful logistic regression with L1 regularization. The beauty of L1 regularization is that it prunes highly correlated features, as well as irrelevant features.
After training your machine learning model, you’ll want to verify the model’s performance using standard validation techniques. If the precision-recall of the model meets your standards, then we can wrap up this step and move on. Otherwise, we can continue to add features to your dataset or try more complex nonlinear models until you’re satisfied with the performance.
For Quality Signup: We began with an extensive list with dozens of features for each member in our sample. After removing all the unnecessary features that were not good indicators of long-term engagement, our final metric contained only a few of the most relevant features without loss in performance.
Step 4: Making your metric actionable (drive product strategy)
A metric purely based on a machine learning model is great for measuring long-term value, but it might not be very effective for driving growth. We’ve learned from our experience that in order to drive growth, a metric has to be easy to grasp and easy to rally behind. The challenge is finding the correct balance between the precision-recall of the model and the actionability of a simplified recipe. In most cases, you should be willing to sacrifice a moderate amount of predictive power for simplicity.
Take for example the case of Facebook’s metric: 10 friends in seven days. This is a great growth-driving metric because it perfectly fits the conditions—it’s easy to grasp and easy to rally behind. However, it should be obvious that there isn’t a sweet spot at 10 connections in seven days. A member who only has nine connections on Day 7 and makes two more on Day 8, would not be considered a success by Facebook’s metric, but might be even more valuable overall. In reality, the number of connections and the amount of time is a spectrum, and the most accurate model would take both of these factors as continuous variables. Facebook is making the trade-off between predictive power and simplicity.
In Step 3, we talked about removing spurious and correlated features. At the end of this process, if your metric still contains too many features to easily explain, then it’s time to start removing features manually. The simplest way to approach this is to stack rank the importance (predictive quality) of your features and repeat the machine learning process while gradually removing the lowest ranking features. Continue to repeat this process until you’re satisfied with the number of features or your model performance is below your requirements.
Finally, to really simplify your metric into a recipe that can be casually thrown around at the office, you’ll need to assign concrete numbers to each of your features. You can run your machine learning model with a single feature at a time to get an idea of where to begin and then test the precision-recall of various recipes. In the end, you’ll usually want to round your recipe to something that’s easy to remember.
Step 5: Validating your metric in its ecosystem
The steps up to this point have guided you towards designing a true-north metric that accurately measures long-term sustainable growth, while staying simple and actionable. At the very least, this metric is correlated with growth, but it’s also very likely to explain the drivers of growth. Distinguishing between correlation and causation using historical data is a recurring dilemma and is difficult to answer, even with the most sophisticated statistical methods. The only definitive way to prove that your metric is causal for growth is by passing it through the fires of A/B testing. A/B testing your metric not only proves its causality, but it also opens up a whole new set of strategies for driving your business through the insights surfaced by the definition of your true-north metric.
For Quality Signup: The new member acquisition team has focused its efforts on driving Quality Signups, even at the expense of some near-term wins. By focusing on the fact that Quality Signup was causal to long-term engagement, we were able to push through some initially alarming results and find material wins for long-term sustainable growth. For example, when we re-designed the LinkedIn guest homepage, we saw a decrease in signups, but no change in Quality Signups. We dug deeper and discovered that members with existing accounts were more likely to login with the new design and we were actually reducing the number of unintentional duplicate signups! Without this metric, we might have terminated our experiment prematurely.
Think about how the features in your metric integrate into your product ecosystem and diagram how the product moves your true-north metric in a funnel. Once you understand this funnel, create initiatives that can directly drive your true-north metric and verify that they also lead to long-term sustainable growth. In practice, there’s usually a long delay between seeing an impact on your true-north metric and seeing its effect on long-term growth, but that’s expected by the nature of these two different measurements. Once you’ve shown the causal connection between your true-north metric and growth, you can focus efforts on driving your true-north metric while remaining confident that it’ll lead to real growth in the future.