Research

Scarcity of positive labels for training machine learned models is a frequently encountered problem in many real world applications such as prediction of integrity harms in social networks. To address this issue, these classifiers are often trained on oversampled positive labels. This in turn leads to a biased model that needs to be calibrated to generalize well to the entire population in order to use the classifier predictions as probabilities in downstream use cases. Traditional supervised calibration approaches such as beta calibration often mis-calibrated due to the sampling bias. In this paper, we present a novel approach to augment labeled data with a random unlabeled sample of population, and use a semi-supervised approach to generate scores that better generalizes to the entire population. Specifically, we examine this approach in the context of low quality models used to detect posts such as engagement bait and clickbait in social media posts in order to mitigate their prevalence and spread. Due to the inherent scarcity of harmful and low quality posts, these classifiers are often trained on oversampled positive labels. To effectively scale these models globally, it is also common to oversample under represented languages. We evaluate our semi-supervised calibration approach using the calibration coefficient metric, that measures the similarity between the classifier estimated prevalence of positives and the population level prevalence of positives, and show that our model outperforms supervised approaches by moving this metric closer to one.