People struggle to separate argument quality from their own political opinions

A series of three experiments found that when people evaluate arguments on political topics, their prior beliefs about that topic are more important than the actual quality of those arguments. People do not evaluate arguments independently of the background beliefs held about them. The paper was recently published in Cognition.

Media literacy is the ability to access, analyze, evaluate, and create media messages while understanding how different media forms influence perception, beliefs, and behavior. It helps people navigate an information-saturated world more effectively. Media literacy teaches individuals to question the source, intent, and credibility of the information they encounter, rather than accepting it at face value.

A major part of media literacy is learning to identify biases, emotional manipulation, and persuasive techniques that appear in news, social media, advertising, and entertainment. To do this, people often need to evaluate the quality of arguments a media piece uses.

Evaluating the quality of arguments requires examining whether claims are backed by evidence, whether sources are reliable, and whether reasoning is logically consistent instead of relying on personal impressions or popularity.

In contrast to this approach, which often requires substantial effort, people often decide which arguments to trust based on cognitive shortcuts such as the credibility of the speaker, alignment with prior beliefs, emotional appeal, or social cues.

Media-literate individuals learn to slow down these automatic judgments and instead assess arguments based on facts, context, and method. This includes being able to differentiate between correlation and causation, understand basic statistical claims, cross-check claims with multiple independent sources, and recognize misleading visuals or headlines.

In their new study, Calvin Deans-Browne and Henrik Singmann wanted to explore how people evaluate the quality of everyday arguments about disputable political claims (e.g., “Abortions should be legal in the U.S.”).

“I started conducting research around this topic in 2020 during the COVID-19 pandemic. During this time we saw a proliferation of poor quality information, which unfortunately included misinformation about how COVID-19 is spread and how to keep ourselves safe from the virus,” explained Deans-Browne, a PhD student at University College London.

“Though the circulation of misinformation is not a new phenomenon, the COVID-19 pandemic highlighted that uncredible information both spreads easily and is persuasive to many. This got me interested in researching what it is that makes people persuaded by an argument. In this case, is it the quality of the argument people see, or how far the argument is in line with what they already believe?”

The study authors conducted a series of three experiments in which they manipulated argument quality by varying how well the information presented in each argument was connected to the argument’s central claim.

“Good arguments” contained evidence that provided strong support for the claim that was either statistical (e.g., “The United States’ gun-related homicide rate is 25 times higher than the average of 22 other comparable high-income nations”) or causal (e.g., “When we heat our homes, power our cars, and run our factories, the emissions released cause our planet to warm”).

Evidence for ‘bad’ arguments was substantially weaker, containing various flaws including circular reasoning (essentially restatements of the claim), appeals to authority, appeals to popularity, and appeals to tradition. In total, study authors prepared arguments about 10 different political claims (topics), with participants seeing 8 topics in each experiment.

A group of 49 participants pretested the materials used in the first two experiments. They were U.S. residents recruited through Prolific and mostly Democrats. The study authors used them to verify whether “good arguments” were really perceived as better than “bad arguments” when viewed side-by-side.

The first experiment consisted of two parts. In the first part, participants were introduced to the 8 political topics and asked to report their beliefs about them on a scale from “extremely false” to “extremely true.” In the second part of the experiment, they were shown arguments about those topics and asked to rate their quality.

The results showed that participants are able to distinguish between good and bad arguments. However, the difference in quality evaluations participants gave between good and bad arguments was much smaller than the difference in quality evaluations between arguments the participant agreed with and those they did not agree with. Quantitatively, the effect of belief consistency was roughly three times larger than the effect of argument quality.

The second experiment used the same materials but controlled the order in which participants reported their beliefs and rated argument quality. Half of the participants first evaluated the arguments and then reported their beliefs, while the other half first reported beliefs and then evaluated the arguments (like in Experiment 1).

This was done because study authors suspected that stating their own beliefs before rating arguments might have made participants believe that their own beliefs were important for evaluating argument quality. The results of the second experiment replicated those of the first. The order of the arguments did not seem to affect the argument quality evaluations.

Finally, the third experiment was conducted with the goal of understanding which specific feature of a bad argument makes an argument bad. Unlike the first two experiments, this sample was balanced to include a nearly equal number of Democrats and Republicans.

The study authors systematically manipulated two elements of bad arguments: half the bad arguments had inconsistent evidence and the other half were based on appeals to authority. Inconsistent evidence arguments were constructed so that some of the evidence supported the claim made, while the rest of the evidence opposed it.

Surprisingly, the results indicated that people find arguments with inconsistent evidence to be better than arguments based on appeals to authority, though “good” arguments were still rated the highest overall.

To determine if this preference was based on the content of the argument or the participant’s bias, the researchers performed a critical check. For each topic, the “inconsistent” arguments for both the left and right leaning perspectives used essentially the same sentences, just in a different order and ending with different conclusions.

They found that left-leaning participants rated the left-leaning inconsistent argument higher, and right-leaning participants rated the right-leaning version higher. Since the text was essentially identical, this confirmed that the rating was driven by the participant’s prior beliefs, not the argument itself.

“The inconsistent arguments we had in our study did not make much sense if you read them carefully, as an argument cannot simultaneously support opposing perspectives,” Deans-Browne told PsyPost. “We were therefore surprised to find that the people in our study thought these inconsistent arguments were better than the arguments based on the appeal to a well-known figure that were much easier to follow.”

So what are the primary takeaways? “Arguments with better evidence are usually judged as better arguments. However, arguments that are in line with what an individual believes are also more likely to be judged as better arguments,” Deans-Browne explained.

“When we compare these two factors, how far the argument is in line with what an individual believes has a greater effect than how good the evidence presented in the argument is. This highlights that two individuals can see the same argument and interpret it very differently depending on their pre-existing beliefs on the topic.”

The study contributes to the scientific understanding of the way humans reason and interpret arguments. However, all three experiments were conducted on groups of Prolific users. Results on other demographic groups might differ, though the authors noted that results replicated even when the education level of the sample dropped significantly in Experiment 2.

“The long term goal of this line of research is to get a better idea of what makes an argument persuasive,” Deans-Browne said. “I am particularly interested in the role of people’s existing beliefs and how far this predicts how an argument is received. Our immediate next step following this paper is to see whether we can change how people evaluate an argument by manipulating their beliefs directly.”

“I want to highlight that while people tend to perceive arguments positively when in line with what they believe as an individual, people also perceive arguments more positively when presented with good quality evidence,” he added. “While we highlight the limitations of trying to persuade people with good evidence alone, we do not want to give the impression that presenting arguments with good quality evidence is a fruitless endeavour or something that should not be done.”

The paper, “For everyday arguments prior beliefs play a larger role on perceived argument quality than argument quality itself,” was authored by Calvin Deans-Browne and Henrik Singmann.