Imagine walking down the street and you see a poster for your favorite movie. By simply pointing your phone camera at the poster, the characters from the movie come to life, instantly immersing you in an Augmented Reality (AR) experience through your phone.

AR target tracking is the next milestone in the evolution of Facebook’s AR platform. Available in Facebook’s ARStudio, the AR Target Tracker effects make it possible for brands, publishers, artists and developers to create content that’s contextually tied to images, logos, signs and pictures in the real world. The technology, designed for the Facebook camera, allows the camera to track an image and open an immersive experience. Augmented reality can be anchored on any flat surface for the effect to show up as an overlay in the Facebook camera.

The movies A Wrinkle in Time and Ready Player One are using Facebook AR Target Tracking technology, to create immersive experiences tied to recent movie launches, with the goal of engaging fans in new ways. People can try the experience today on all of their movie posters worldwide. A toolkit for all developers is available as a feature under the plane tracker menu in ARStudio.

Moving from ephemeral to persistent AR

Last December, we launched AR Studio, enabling creators to build effects where people can place 3D objects into their surroundings and interact with them in real time. This ephemeral task allows you to place an AR object on any surface to interact and share. However, we wanted to take that to the next level, and explore use cases around persistent AR – experiences that are connected to and persist relative to places or things in the real world. AR Target Tracking is Facebook’s first instantiation of this kind of a persistent AR experience.

Our first foray into persistent AR is on 2D planar targets because they are ubiquitous. For example, the movie poster exists all over the world so anyone in front of poster can have the same experience since the AR is tied to that poster. This allows us to transcend time and space and to provide a consistent experience to our Facebook community.

Facebook’s approach focuses on quality and access

While others have developed similar technology, there are two things that are unique about our approach. Our goal is to make the technology available for the widest array of devices at the highest level of quality possible, maximizing accessibility to the widest variety of phones so anyone with an iPhone 5s and above or an Android 2012 device and above can use it. We are able to provide this experience for for high-end market phones all they way down to older, lower end phones, to ensure that people in any market can experience it. We achieved this coverage by heavily optimizing our tracking algorithm to perform really well on the low-end devices.

We also want to maintain the highest level of quality for the AR experience. To maintain a high bar, we test our algorithms under different lighting conditions, at night, at different angles, and with partial occlusion.

Making an AR Target Tracker effect

To make a AR Target Tracker effect for a poster, you would first upload the master poster image into AR Studio. You would then create and upload the specific AR art experience designed to embed in the poster. Once it’s deployed, to enter the experience, a person simply opens their Facebook camera, launches the effect from the effect tray, and then can point at the poster. The poster triggers the AR experience for the user, as soon as they point at it. The person can also get to the effect directly through other mechanisms such as a pre-programmed QR code, such as what was done on the Wrinkle in Time and Ready Player One movie posters.

How it works

The poster or image that is uploaded into AR Studio forms the template image against which the video frames from the camera are compared to recognize the poster. Our technology analyzes the template poster to find uniquely identifiable low-level visual features and then uses an image matching and verification pipeline to compare it against under different conditions. The process takes the template poster and then looks for uniquely identifiable parts such as corners, or texture patterns. We then extract patches from those locations which are then encoded into a feature descriptor vector using a pre-trained encoding process. These vectors are then quantized and stored in an efficient index fast lookup and matching.

The key to the effectiveness of this algorithm is how well we can match to what was uploaded so the user will immediately be in the experience. Once they are there, the algorithm switches to tracking the poster. As the user moves their phone, we have to make sure that we don’t lose the tracking and that they continue to see the AR even if the image is partially obstructed by for example someone walking in front of it. To accomplish this we leverage the Viper tracker that we’ve used in the past in the SLAM tracking efforts for world effects. Our method handles this by detecting and tracking supporting structure in the environment outside of the poster – using a highly optimized algorithm that is fast enough to run on low-end phones.

Opportunities for creators and businesses

This new target tracking capability offers businesses entirely new ways to build their brands and connect with customers through immersive AR experiences. It’s a novel and dynamic way to engage customers across touch points and around the world. The movie poster (or any poster for that matter) experience can weave in all kinds of promotions, from a free popcorn to a scavenger hunt for free tickets. Restaurants and stores can create immersive and entertaining AR experiences tied to their storefronts (like fashion shows or chefs in the kitchen) to entice customers to walk in the door. And the opportunity to create games will take us on adventures way beyond what was previously imagined. The possibilities are endless! We can’t wait to see what creators and businesses dream up as we launch this next phase of our AR platform.