By Peter Welch, Senior Principal Engineer
Today, Yahoo Mail introduced a feature that allows you to automatically sync your mobile photos to Yahoo Mail so that they’re readily available when you’re composing an email from your computer. A key technology behind this feature is a new photo and video platform called “Tripod,” which was born out of the innovations and capabilities of Flickr.
For 13 years, Flickr has served as one of the world’s largest photo-sharing communities and as a platform for millions of people who have collectively uploaded more than 13 billion photos globally. Tripod provides a great opportunity to bring some of the most-loved and useful Flickr features to the Yahoo network of products, including Yahoo Mail, Yahoo Messenger, and Yahoo Answers Now.
Tripod and its Three Services
As the name suggests, Tripod offers three services:
- The Pixel Service: for uploading, storing, resizing, editing, and serving photos and videos.
- The Enrichment Service: for enriching media metadata using image recognition algorithms. For example, the algorithms might identify and tag scenes, actions, and objects.
- The Aggregation Service: for in-application and cross-application metadata aggregation, filtering, and search.
The combination of these three services makes Tripod an end-to-end platform for smart image services. There is also an administrative console for configuring the integration of an application with Tripod, and an identity service for authentication and authorization.
The Pixel Service
Flickr has achieved a highly-scalable photo upload and resizing pipeline. Particularly in the case of large-scale ingestion of thousands of photos and videos, Flickr’s mobile and API teams tuned techniques, like resumable upload and deduplication, to create a high-quality photo-sync experience. On serving, Flickr tackled the challenge of optimizing storage without impacting photo quality, and added dynamic resizing to support more diverse client photo layouts.
Over many years at Flickr, we’ve demonstrated sustained uploads of more than 500 photos per second. The full pipeline includes the PHP Upload API endpoint, backend Java services (Image Daemon, Storage Master), hot-hot uploads across the US West and East Coasts, and five worldwide photo caches, plus a massive CDN.
In Tripod’s Pixel Service, we leverage all of this core technology infrastructure as-is, except for the API endpoint, which is now written in Java and implements a new bucket-based data model.
The Enrichment Service
In 2013, Flickr made an exciting leap. Yahoo acquired two Computer Vision technology companies, IQ Engines and LookFlow, and rolled these incredible teams into Flickr. Using their image recognition algorithms, we enhanced Flickr Search and introduced Magic View to the Flickr Camera Roll.
In Tripod, the Enrichment Service applies the image recognition technology to each photograph, resulting in rich metadata that can be used to enhance filtering, indexing, and searching. The Enrichment Service can identify places, themes, landmarks, objects, colors, text, media similarity, NSFW content, and best thumbnail. It also performs OCR text recognition and applies an aesthetic score to indicate the overall quality of the photograph.
The Aggregation Service
The Aggregation Service lets an application, such as Yahoo Mail, find media based on any criteria. For example, it can return all the photos belonging to a particular person within a particular application, all public photos, or all photos belonging to a particular person taken in a specific location during a specific time period (e.g. San Francisco between March 1, 2015 and May 31, 2015.)
Vespa, Yahoo’s internal search engine, indexes all metadata for each media item. If the Enrichment Service has been run on the media, the metadata is indexed in Vespa and is available to the Aggregation API. The result set from a call to the Aggregation Service depends on authentication and the read permissions defined by an API key.
APIs and SDKs
Each service is expressed as a set of APIs. We upgraded our API technology stack, switching from PHP to Spring MVC on a Java Jetty servlet container, and made use of the latest Spring features such as Spring Data, Spring Boot, and Spring Security with OAuth 2.0. Tripod’s API is defined and documented using Swagger. Each service is developed and deployed semi-autonomously from a separate Git repository with a separate build lifecycle to an independent micro-service container.
Swagger Editor makes it easy to auto-generate SDKs in many languages, depending on the needs of Yahoo product developers. The mobile SDKs for iOS and Android are most commonly used, as is the JS SDK for Yahoo’s web developers. The SDKs make integration with Tripod by a web or mobile application easy. For example, in the case of the Yahoo Mail photo upload feature, the Yahoo Mail mobile app includes the embedded Tripod SDK to manage the photo upload process.
Buckets and API Keys
The Tripod data model differs in some important ways from the Flickr data model. Tripod applications, buckets, and API keys introduce the notion of multi-tenancy, with a strong access control boundary. An application is simply the name of the application that is using Tripod (e.g. Yahoo Mail). Buckets are logical containers for the application’s media, and media in an application is further affected by bucket settings such as compression rate, capacity, media time-to-live, and the selection of enrichments to compute.
Beyond Tripod’s generic attributes, a bucket may also have custom organizing attributes that are defined by an application’s developers. API keys control read/write permissions on buckets and are used to generate OAuth tokens for anonymous or user-authenticated access to a bucket.
App developers at Yahoo use the Tripod Console to:
- Create the buckets and API keys that they will use with their application
- Define the bucket settings and the access control rules for each API key
Another departure from the Flickr API is that Tripod can handle media that is not user-generated content (UGC). This is critical for storing curated content, as is required by many Yahoo applications.
Architecture and Implementation
Going from a monolithic architecture to a microservices architecture has had its challenges. In particular, we’ve had to find the right internal communication process between the services. At the core of this is our Pulsar Event Bus, over which we send Avro messages backed by a strong schema registry. This lets each Tripod team move fast, without introducing incompatible changes that would break another Tripod service.
For data persistence, we’ve moved most of our high-scale multi-colo data to Yahoo’s distributed noSQL database. We’ve been experimenting with using Redis Cluster as our caching tier, and we use Vespa to drive the Aggregation service. For Enrichment, we make extensive use of Storm and HBase for real-time processing of Tripod’s Computer Vision algorithms. Finally, we run large backfills using PIG, Oozie, and Hive on Yahoo’s massive grid infrastructure.
In 2017, we expect Tripod will be at 50% of Flickr’s scale, with Tripod supporting the photo and video needs across many Yahoo applications that serve Yahoo’s 1B users across mobile and desktop.
After reading about Tripod, you might have a few questions
Did Tripod replace Flickr?!
No! Flickr is still here, better than ever. In fact, Flickr celebrated its 13th birthday last week! Over the past several years, the Flickr team has implemented significant innovations on core photo management features (such as an optimized storage footprint, dynamic resizing, Camera Roll, Magic View, and Search). We wanted to make these technology advancements available to other teams at Yahoo!
But, what about the Flickr API? Why not just use that?
Flickr APIs are being used by hundreds of thousands of third-party developers around the world. Flickr’s API was designed for interacting with Flickr Accounts, Photos, and Groups, generally on lower scale than the Flickr site itself; it was not designed for independent, highly configurable, multi-tenant core photo management at large scale.
How can I join the team?
We’re hiring and we’d love to talk to you about our open opportunities! Just email firstname.lastname@example.org to start the conversation.