Dynamic Machine Translation in the LinkedIn Feed


Polyglot-Online

The Polyglot-Online mid-tier service uses GaaP to safely send encrypted text snippets to the Translator Text. An additional advantage in this framework is the ability to customize the translation models for a specific domain (like our feed) and integrate logic for filtering translation outputs based on system confidence scores. The API supports more than 60 languages in any translation direction, all of which we can leverage once the source language locale of a piece of content has been detected. For this feed feature, we selectively translate source text into 24 target languages, to match each member’s interface locale supported by LinkedIn.

This translation service also has features like logic for protecting entities such as hashtags and name mentions from being distorted in translation, and integrated filters to block irrelevant or unprofessional content, as well as advertisements, from being translated on the LinkedIn platform. We also use an in-memory encrypted cache to reduce latency, with its lightweight maintenance nature and better cost-to-serve than centralized solutionsthe Java Play framework at LinkedIn, the service easily supported multiple thousands of QPS during our prototype ramp. 

Acknowledgements 

Many thanks to Weizhi (Sam) Meng and Chang Liu for great coding and ownership, to David Snider for initiating the project, and to Annie Lin for writing GaaP scripts.

We also want to thank Ian Fox for his work with Azure, Pradeepta Dash for engineering support for the feed, Atul Purohit for guidance with the feed API implementation, Jeremy Kao for guidance with web, Samish Kolli for client-side support, Nathan Hibner for his many contributions in tweaking the model, and Chao Zhang for the expert answers about overall backend functionality.

Additionally, we want to recognize our helpful friends at Microsoft: Ashish MakadiaAssaf Israel, and Brian Smith from the Text Analytics team, and Chris Wendt and Arul Menezes from the Translator team.

Finally, a huge thank you to Francis Tsang and Tetyana Bruevich for their endless support.

We hope our members enjoy this new feature!



Source link