Cindy Zhang | Pinterest engineer, Search
Earlier this year we introduced camera search to help Pinners find ideas to try using what they see in the real world to search–whether it’s a recipe made with new in-season produce or clothing matching a specific pattern. Today we’re taking another step in connecting the real world with the ideas on Pinterest with the launch of Pincodes.
Pincodes are custom codes you can scan to discover curated ideas and products from brands and publishers. When you see a Pincode, just tap the camera icon in the Pinterest app and point at the code to jump directly to a board or profile. With Pincodes, a brand can take any board or profile and generate a unique code to share anywhere–in stores, on packaging, in magazines, ads and more.
In this post, we’ll share how we built and implemented Pincodes in partnership with Quikkly who develops custom scannable code solutions for app developers in the form of SDKs.
Encoding a Pincode
At Pinterest, we use a unique Object ID to identify boards and Pinner profiles. Given the Object ID of the Pinner or board and a hero image, the Quikkly library generates a Pincode comprised of fifty dots in three possible sizes. These dot sizes provide a max data capacity of log₂(3⁵⁰) — around 79 bits, 64 of which are used for the Object ID and the rest for data integrity. The dots are arranged into equally-spaced groups which inform the scanner that the fifty dots on the circle belong to a Pincode.
The size and organization of the dots are the two visual attributes solely used to encode data for a Pincode. The colors of the dots are for aesthetics (and aren’t used in the encoding process). In fact, Pincodes can work even when printed in black and white.
Scanning a Pincode
Photos from a smartphone camera are often big and noisy, so they need to be normalized before they’re decoded. The Quikkly library starts off by shrinking the images so they’re small enough to be processed efficiently but large enough for the scanner to recognize the smallest dots. To remove noise, a filter is applied to smooth out the grain at low light and remove any visual artifacts like the stripes cameras capture in photos of screens. After addressing size and noise, the scanner finds the dots and their placement from the high-contrast edges in the image and constructs data objects from these patterns. These data objects, which contain the Object ID, are then decoded and used to navigate to the board or Pinner associated with the Object ID.
One of the biggest challenges for Quikkly in scanning is accurately measuring the dots. The size of the dots varies depending on camera perspective, angle and focus. Instead of using a standard error correction algorithm such as Reed-Solomon, Quikkly uses probabilistic decoding to give each dot a probability of belonging to a size bucket (small/medium/large) rather than measuring each dot to be in one of these buckets. With this, the scanner can consider multiple potential sizes for dots it’s unsure about.
Integrating Pincode scanning on Android
In the Android client, we set a
PreviewCallback to the native camera, which is invoked on every preview frame as long as the preview is active. In the fragment, which houses the camera, we declare a
PreviewListener (provided by Quikkly) and give it to our
CameraPreview, a class that extends
SurfaceView which has reference to the camera and a
SurfaceHolder. We then override the
onPreviewFrame(byte data, Camera camera) method in our
PreviewCallback to invoke the
PreviewListener and deliver an Object ID to the fragment via Quikkly’s scanning library. Once we receive the Object ID in the fragment, we decode it to decide whether to navigate to a profile or board.
Decoding the Object ID and navigating:
We use an Object ID (which identifies Pinners, boards, etc.) to encode the Pincode, because constraining the amount of data stored in each Pincode allows for greater flexibility in the design of the code. The type (Pinner, board, etc.) is encoded within the Object ID itself, so at the time of decoding, the client simply performs bitwise operations to detect whether it should navigate to a board or profile.
Exporting large Pincode images on iOS and Android
Because Pincodes are meant to be printed, the exported image of the Pincode and our call-to-action caption had to be high resolution. Instead of resizing the bitmap we received from Pincode generation, we use the Scalable Vector Graphics (SVG) from the Pincode. These SVGs, which are XML-represented vector images, can be scaled easily without losing image quality. We first scaled the SVG from the Pincode generation and our custom call-to-action SVG into large bitmaps. We then combined these two bitmaps into a single bitmap that can be saved to a phone’s camera roll or printed.
Generating a Pincode image and CTA text to a PNG file on web
To enable generating a Pincode image on the web, we added a backend API that integrates the Quikkly python SDK. The frontend then retrieves the SVG string from the backend API and inlines it with our custom call-to-action SVG. When a Pincode is downloaded, we combine the generated Pincode SVG and our custom call-to-action SVG into a large bitmap (png file). Here are detailed steps:
- Clone both SVG nodes: parent’s
- For the hero image in the generated Pincode SVG image, fetch the binary of the image by its URL, base64 encode it and then inline it in data URI.
- Transform each SVG element into an image element with the SVG content in the data URI.
- Wait until both SVG images are resolved
- Create a canvas with the desired export size and draw both the Pincode and call-to-action images onto the canvas
- Convert the canvas data into data URL (toDataURL)
- Convert data URL to Blob and then to object/blob URL. This step is necessary because data URI scheme is designed for small value and does not work with large data (see https://stackoverflow.com/a/41755526/4056191).
- Create an anchor element with src set to object URL, and then trigger a click event
- Tear down the anchor element and object URL
We’re launching Pincodes today with partners Pinners love, including Nordstrom, Kia, REAL SIMPLE, The Home Depot and Kraft Heinz, with more to come. Keep an eye out for Pincodes this holiday shopping season!
Acknowledgements: Special thanks to Pinterest engineers Kelei Xu, Mustafa Motiwala, Jack Hsu, Steven Ramkumar and Jeffrey Harris as well as Quikkly.