May 25, 2020

The methodology behind Facebook’s State of Small Business Survey

By: Pratiti Raychoudhury

Pratiti Raychoudhury is the Global Head of User Research at Facebook.

Small and medium-sized businesses (SMBs) in the United States have been disproportionately affected by COVID-19. At the same time, policymakers and researchers are facing challenges obtaining timely information about the needs and outlook of SMBs. We know many of the owners and operators of these businesses are on Facebook, and we wanted to use our platform to gather insight on the impact of COVID-19 on the SMB economy. Therefore, Facebook joined with the Small Business Roundtable to research and analyze the socioeconomic impacts of COVID-19 on SMBs and their employees through the release of our first State of Small Business Report.

The report is based upon a survey of approximately 86,000 SMB owners, managers, and employees in the U.S., and it covers a range of topics related to their current situations. These include whether their businesses are operating, how many people they employ, their access to capital, and their expected challenges in the coming months. The survey also inquired into related concepts, such as familial obligations, access to information, and survival strategies.

The use of Facebook as a sample source has many advantages:

  • It allows for the design, implementation, and launch of a survey in a timely manner.
  • Large sample sizes allow for more questions to be asked through random assignment of modules, without incurring the respondent fatigue or satisficing behaviors that affect longer surveys.
  • Facebook enables researchers to draw samples from diverse parts of the business population, including people who are typically omitted from business establishment surveys, such as informal businesses, side hustles, and new businesses. One did not have to be preidentified as an owner or a manager in order to be included in the survey.
  • Research has shown that Facebook’s user population in the United States is most representative in terms of Americans’ educational attainment and internet skills, writ large.
  • We were able to use more rigorous probabilistic sampling techniques and nonresponse adjustments than many nongovernmental establishment surveys.

The State of the Small Business Report provides a deep dive in the United States of English speaking Small and Medium Businesses who have been active on Facebook platform during the survey fielding period. We believe that consistent, high-quality data and research will be critical toward understanding and addressing the unique challenges facing SMBs. We hope our work can contribute to solving part of the puzzle, hand in hand with other research that is emerging.

About the study

Data was collected through an online survey on the Facebook platform administered from April 20 to 27, 2020. This survey is a complement to an ongoing international data collection collaboration with the World Bank and OECD called the Future of Business. The Future of Business Survey typically runs twice a year, and in more than 100 countries. However, the circumstances affecting businesses during the COVID-19 pandemic led to the decision to field six cross-sectional surveys in approximately 50 countries in the second half of 2020.

Sampling and Representativeness. The survey’s target population was owners and managers of SMBs with fewer than 500 employees and their staff, covering English-speaking U.S. Facebook users. In total, the 86,000 respondents reflect approximately the following:

  • 38,000 owners and managers — respondents who reported that they manage day-to-day operations of a business or own a business.
  • 9,000 personal businesses — respondents who reported that they are self-employed providing goods or services or that they produce goods sold for personal income. (Examples of personal businesses include selling clothing, making and selling handicrafts, freelance writing, and photography services. Personal businesses may or may not be full-time or part-time ventures and may or may not have employees.)
  • 39,000 people who reported working as a business employee/worker as their only response were considered employees.

The survey used a stratified sampling design, which oversampled two populations (Facebook page admins and active Marketplace sellers) with disproportionately high incidences of businesses and undersampled the general Facebook population to ensure coverage of all businesses and their employees.

We used a two-stage weighting procedure where we fit nonresponse weights using a logistic regression built on demographic and platform-side metadata for each population and then rescaled the weights to correct for oversampling. The sample size calculation achieved a 95% confidence level for estimating proportions with an average 3% error rate and an 80% power to detect differences across regions and by gender.

The sample is representative of Facebook users nationally, exclusive of territories, with breakdowns by region (Northeast, Southeast, Midwest, and West), industry, business type, and gender. However, it might not be representative of all SMBs and their employees. The table below shows a side-by-side comparison of business size and industry sector.

*Source: 2017 Census

Questionnaire. The survey presented 136 questions or statements that examined various aspects of the health of SMBs in the U.S. These followed five major thematic areas: 1) Business Operations During COVID-19, 2) Business Finances, 3) Business and Family Responsibilities, 4) Employment, and 5) Business Survival Strategy. The survey randomly assigned each respondent a maximum of 30 questions covering a core set of questions and one of the five thematic areas.

The questionnaire was developed in different stages, starting with a review of the literature regarding questionnaires administered to SMBs during disaster management, e.g., swine flu, hurricanes, and so on. The second step was identifying main dimensions where the pandemic could have an impact on both SMB owners/managers and their staff.

A group of economists and experts at multiple academic institutions reviewed the questionnaire and provided feedback prior to launch.

The random allocation of the modules across respondents, coupled with the random sampling protocol used, provides us a representative sample across all respondents and across all thematic areas. Respondents who identified themselves as employees of SMBs, with no management or executive power, received a separate module addressing employment, remote work, and work-family balance aspects, among other challenges that they might be facing.

The data from the State of Small Business Report and the rigor with which they were collected support the findings of other studies that have focused primarily on the economics of the COVID-19 pandemic and add complementary insights into how people and businesses are being impacted. The overall picture from the body of literature confirms fears of an immediate shock due to closures with an accompanying drop in demand and a higher level of unemployment.

Facebook’s efforts to support SMBs and their employees does not stop at this report. We understand that there remains a lot more to measure, both in the U.S. and across the world. We will be updating the report in the coming months with results from new iterations of the survey that will cover recovery attempts in the U.S. and the global situation of SMBs and their employees. We hope this debut and its future iterations provide valuable information on a crucial aspect of our economy and assist other researchers engaged in similar endeavors.

To inquire about the State of Small Business Report or the survey it is based upon, please reach out to Curtiss Cobb (ccobb@fb.com).