Fact checker interviews

Introduction#

In Autumn 2020 the Open Data Institute carried out interviews with fact checkers about their use of data from national statistical institutes. Here we present the high-level findings of that research.

For shorthand, we’ll refer to national statistical institutes as NSIs. We use this term broadly to include any organisation that publishes national statistics data used by fact checkers. Even though some countries we looked at do not have independent national statistical institutes but rely on a combination of other organisations.

Research#

We interviewed a total of 15 people. This consisted of:

• 9 fact checkers (or managers of fact checkers)
• 3 technologists who work in fact checking organisations
• 3 people who are experts on publishing of national statistical data (but are not fact checkers themselves)

The interviewees were from 8 different countries across 5 different continents. 11 of the interviews were conducted over a recorded call - with extensive note taking afterwards. 4 of the interviews were conducted through a questionnaire.

Research Findings#

The following are the main broad themes and high-level insights we found in the interviews. We have removed any information that could identify interviewees.

Communication#

The interaction or relationship between the fact checking organisation and the publisher of national statistics.

• Fact checkers contacted NSIs for a number of reasons. Most typically:
  • To get background information, methodologies or extra context on data that was published.
  • To find out if certain data exists and could they get access.
  • To get extra statistical analysis done which may be beyond skills of fact checkers
• In many countries, communication processes were often informal. It depended on who the fact checker knew or had connections with in the NSI.
• Some fact checkers said the NSI they rely on have very strict rules about who can talk or not talk publicly. That there wasn’t a sense of an open communication culture from them.
• People interviewed said they would like if points of contact were more formalised and reliable
• For lots of datasets, it was often uncertain who to contact to get more information.
• In some countries, the NSIs were initially sceptical of fact checkers. They found it took time to build a trusting relationship with the national statistics institute where they would speak more freely.
• Being able to call someone on the phone was usually seen as the more preferable option for fact checkers.
• Email was also seen as acceptable.
• However, email was seen as acceptable if it works. Email was often a dead end where no one would reply.
• An interesting suggestion was annual or biannual workshops to improve the relationships between fact checkers and NSIs and create a better understanding of each others’ needs.
• Others suggested more work on building the community of data users. They would like to learn from others who used NSI data.

Contexts & Methodologies#

Extra supporting material behind the published data. For instance, how was the data collected? Or a caveat or extra context to keep in mind when looking at the data. Often described with an asterix, a numbered reference, or an extra note.

• Fact checkers repeatedly said that understanding where the data came from, how it was made, and if there were things to watch out for was extremely important in their work.
• They often also include the data’s context or methodologies in the fact check they create.
• A lot of times fact checkers got the data context they needed context by directly contacting people in the NSIs.
• Extra context was sometimes needed to show to readers whether they should trust the numbers published by NSI.
• When methodologies were included it also helped managers of fact checkers double check the fact check produced.
• It also helped show the shortcomings of the data if they existed. This then helped build trust in the fact checkers amongst their readers.
• It also helped to know if more “experimental” methods were used in creating the data - such as modelling or sampling. This was especially relevant when more data science or machine learning approaches were used by the NSI.
• However, it can be a challenge to present contexts/methods in a way that’s consumable, digestible and understandable.
• The interviewees weren’t aware of standards or guidance for publishing contexts and methodologies.
• Often if methodology was included it was within a .pdf document.

Data publishing formats#

How the data was presented on the NSI website. What format was it in? What metadata did it include? How was the data structured?

• CSV files were by far the most requested format for publishing data.
• Although fact checkers often satisfied with an “Excel file” (assume this covers any spreadsheet format)
• Fact checkers often had to unbundle data from the medium it was published in. For instance, images with numbers or PDFs with tables. They then created their own spreadsheets or datasets with the data to do their own analysis.
• Fact checkers preferred if NSIs just published the spreadsheets alongside the report. As they must have used spreadsheets originally to create the tables.
• PDFs were useful but they lack the ability to be able to link to a specific section or table. Just a link to the page that holds the PDF. These can often be very long and contain lots of information.
• NSIs didn’t seem to have much reason for the formats they publish the data in. It even can vary a lot within a country. Fact checkers didn’t know what data publishing standards NSIs relied on.
• APIs were very popular with technologists. Tech-savvy fact checkers also called for APIs.

Data discovery#

How fact checkers find the national statistics data they need? How do they search for it? Is it hard to find the data they need?

• It was repeatedly said by fact checkers that because of their expertise and knowledge of the domain, they learned where to find the data they needed. And because of this experience, it wasn’t hard to find data they needed.
• However, they did appreciate a good website and information layout.
• Fact checkers often relied on external search engines to search NSI websites. Although one or two said they did use the search on the NSI websites.
• Technologists without subject matter expertise found it hard to find the datasets they needed
• It was said that in countries with less resources - it can be easy to find data because there isn’t much of it
• Sometimes fact checkers would contact experts in a domain and ask them where to find the data they needed.
• Some countries (UK, Spain, African countries) have single organisations which fact checkers mainly went to. Others (USA, India) there was no central body and had to go to multiple organisations.
• When data and information is spread across multiple organisations it creates a challenge when it comes to big structural questions like Covid, climate change, or the environment. This is less of an issue for specific, focused issues or topics.