We have updated FollowFeed’s ranking engine to Quasar. Quasar, as part of LinkedIn’s Pro-ML technology, transforms machine learning features and inferences the machine learning model at query time. As a high-performance, multi-threaded ranking engine, Quasar not only optimizes for infrastructure system efficiency but also machine learning productivity. Such productivity improvements have enabled:
Cross-system leverage: We can easily port the latest machine learning models and transformers from the second pass layer to FollowFeed.
Training and serving consistency: At offline training time, the same codebase is used to represent the model as at serving time.
To reflect the rapid evolution of LinkedIn’s content ecosystem, machine learning models have to be constantly updated. We’ve built FollowFeed’s model deployment on top of LinkedIn’s Continuous Integration and Deployment (CICD) stack. Being a stateful system that indexes members’ past activities, FollowFeed presents a unique challenge in model deployment. We have to avoid calling external services to maintain high reliability and performance of index nodes where ranking is taking place. To optimize for such limits, we previously coupled ranking models with code, which leads to strong coupling of service deployment with model coefficient changes. To allow for easier model evolution, our solution is now a data pack-based deployment model, where we package the machine learning models in a separate code base, a.k.a. a multi-product. Those models are treated as a “data pack,” a deployable package consisting only of static files to be dropped into a specific location of production machines. Through such a design, model deployment can be easily managed by LinkedIn’s CICD system. Consequently, we’ve improved model deployment velocity from 3 days to 30 minutes.
In addition to ranking and model deployment, we designed and implemented advanced feature access and tracking in FollowFeed. As it scores thousands of documents per session, FollowFeed optimizes access to machine learning features needed by scoring models. Viewer features are passed down as part of the request without requiring an external call to be made. Features for feed updates are ingested, stored, and updated alongside these updates so that they are accessible locally on the index nodes for scoring. Given the well-known data inconsistency challenges between offline training and online scoring, we also added accurate tracking of machine learning features in FollowFeed. This guarantees data consistency between offline training data and online inference data. Aggregating these tracking events across the distributed index nodes presents a challenge. Even though results from the index nodes are aggregated in the broker layer, we do not want to gather the tracking data synchronously due to scalability and serving latency concerns. Our design overcomes these concerns by having individual index nodes and the broker node stream tracking events asynchronously to Kafka streams. A Samza job joins these Kafka streams and emits a comprehensive tracking event for the request.
Equipped with advanced machine learning capabilities, it is much easier to develop performant machine learning models for FollowFeed with accurate features. Such agility will enable better candidate feed updates to be surfaced on LinkedIn’s homepage. Through actively updating the machine learning model, we will be able to give more power to existing and newly-minted content creators in LinkedIn’s ecosystem. We will also facilitate the professional community builders’ curation of their audience.
We’ve rebuilt the machine learning model at FollowFeed to optimize multiple objectives under given infrastructure constraints. The existing candidate selection is conducted through a logistic regression (LR) model that predicts the click probability. To facilitate professional conversations in LinkedIn’s feed, we have introduced “contribution” as an additional objective in the candidate selection model. Probability of contribution captures members’ intent to share, comment, or react to a particular feed update. The model also takes into account timely feedback to content creators, which is a clear signal for cultivating and retaining audience builders on LinkedIn.
To achieve these goals, our design evaluates candidate selection through recall instead of precision, combines multiple objective functions in a gradient boosting tree model, and trims down features and manual transformations.
Given that there is a comprehensive model at the second pass layer with more sophisticated computation per update, the job of candidate selection should be focused on generating a rich set of candidates. Clarifying such goals helps us evaluate our model much more effectively. Instead of precision, we use recall as an evaluation metric because it measures the overlap between the top K candidates generated by FollowFeed and the second pass ranker. K is very large for LinkedIn’s feed. To overcome the selection bias introduced by FollowFeed, we randomize its output for a small percent of traffic so that the second pass layer can be exposed to a representative range of candidates. This technique helps us approximate the second pass ranking at the candidate selection process. This technique helps us approximate what the second pass layer’s ranking would be if the candidates were randomized and not pre-ranked by the first pass layer. Through various efforts outlined below, we have effectively doubled the recall percentage throughout the project.