Meta Research

The Facebook Field Guide to Machine Learning is a six-part video series developed by the Facebook ads machine learning team. The series shares best real-world practices and provides practical tips about how to apply machine-learning capabilities to real-world problems.

If you’re interested in using machine learning to enhance your product in the real world, it’s important to understand how the entire development process works. It’s not only what happens during the training of your models, but everything that comes before and after, and how each step can either set you up for success or doom you to fail.

The Facebook ads machine learning team has developed a series of videos to help engineers and new researchers learn to apply their machine learning skills to real-world problems. The Facebook Field Guide to Machine Learning series breaks down the machine learning process into six steps:

1. Problem definition
2. Data
3. Evaluation
4. Features
5. Model
6. Experimentation

This video series covers each of these steps, explaining how the decisions you make along the way can help you successfully apply machine learning to your product or use case. Each lesson highlights examples and stories of non-obvious things that can be important in an applied setting.

Lesson 6: Experimentation

In this last lesson of the Facebook Field Guide to Machine Learning series, we cover experimentation, or making your experiments actionable. One of the key topics is the difference between offline and online experimentation.

Offline experimentation is a fast and reproducible way of testing a machine learning system in isolation. For some image and speech recognition applications, an offline evaluation is representative of using those models in real life. In most cases, however, this is not the case. However, online experimentation allows us to see how our model performs when deployed into the engineering stack, and also when interacting with the world around it.

There are few things more demoralizing than pouring yourself into a difficult project and getting great offline performance before finally testing it online and seeing that that things aren’t working the way you thought they would.

In this lesson, we share a few key principles for effective online testing:

1. Minimize the time to first online experiment
2. Isolate engineering bugs from machine learning performance
3. Test the model in the presence of real-world feedback loops

Some additional practical tips covered in this video include:

• Measure the right thing
• Measure everything
• Be able to triangulate each change
• Have a backup plan
• Calibrate – measure how well the average prediction matches the average response rate

Thank you for joining us for the Facebook Field Guide to Machine Learning, we hope these lessons are useful as you think about edge cases, and how machine learning models fit into the product or business outcome you are trying to achieve.