What makes a claim checkworthy? An investigation into the selection criteria for fact-checking using news value theory

Abstract

Fact-checking is a key method for combating disinformation. However, the question arises as to how fact-checkers select the claims they check. This study examines the role of news factors for fact-checking. News factors and newsworthiness have been studied from multiple angles. There has been research on the selection criteria of journalists, but also on the factors that attract an audience to a specific piece of content. This study focuses on the journalistic perspective. However, as previous research suggests, the different perspectives are not always clearly separable. Especially, in the context of disinformation, the fact-checkers’ perspective is influenced by their target audience, and one can assume that the claims that are checked have been spread with news factors in mind, as well. Accordingly, this paper asks: What aim do news values serve for fact-checkers with respect to their target audience? (RQ1) Building on this, the question arises whether there is such a thing as a list of important news factors for fact-checking. Previous research finds that “classic” news factors such as timeliness, negativity, and prominence play a strong role. But specific factors such as virality, which are particularly typical of online environments, play a significant role, too. Therefore, the paper asks which news factors are more or less relevant for fact-checking. (RQ2) Finally, there is the question of how news factors can be measured or quantified. Metrics enable comparability. It can be assumed that this meets the transparency requirements of fact-checkers. The paper asks: How are news values measured for fact-checking? (RQ3) The study examines two data sources. Semi-structured interviews with fact-checkers from Germany and Austria (n = 10) and an analysis of the claims to which fact-checking articles link (n = 9189). In particular, the study consists of a total of eight interviews with ten fact-checkers from six organizations in the period from August to September 2023. Five of these organizations fall under the newsroom model, and one falls under the NGO model. Additionally, I scraped all fact-checks between 2019 and 2023 of the following organizations: AFP, dpa, Correctiv, Faktenfuchs. This resulted in a total of 4959 fact-check articles. The checked claims were extracted based on the links in the articles. Annotators manually evaluated the method to separate links to claims from links to the evidence sources used by fact-checkers on an artificial week of 596 articles. The interviews provide an in-depth understanding of the role of the factors, while the analysis of the linked claims is used for triangulation, especially in relation to the importance of different platforms and social media engagement. The news factors that inform fact-checkers’ efforts can be classified in two categories. First, “classic” news factors, such as timeliness or (geographical) proximity, inform the selection of claims to be fact-checked. Second, this study identified some news factors that are genuine to the fact-checking genre, such as the verifiability of the claim or the intention of the originator of the claim. In addition, classic factors such as reach play a special role. Fact-checkers are primarily aiming at an audience that is unsettled by misinformation but has not yet slipped into a conspiracy circle. There is also a strong concern that a premature fact-check can help a (false) claim gain more reach. Accordingly, the reach of a claim is primarily used to assess the public need to verify the claim. While many news factors are not easily measured, they are addressed in team meetings. For instance, this is the case with relevance, understood as personal involvement. Those characteristics of a claim that can be measured more easily, such as reach, clearly impact the decision whether it will be fact-checked or not. This means that news factors such as relevance play a role, yet not in a sense that can be justified through metrics. By contrast, news factors such as reach prove consequential and can be backed up with numbers. As expected, there is a lot of content from alternative media in the verified claims, however much of this content is only verified if it is also shared on social media and metrics such as likes or shares can be used to estimate its reach. Building on these findings, the study develops a schema that lists and takes into account the role, relevance, and different versions of news factors that are prevalent in the fact-checking process.