Personalization algorithms create an illusion of competence, study finds

A new study published in the Journal of Experimental Psychology: General suggests that personalization algorithms used by online content platforms may actively hinder the learning process. The findings provide evidence that when algorithms tailor information to a user’s behavior, that user may develop a biased understanding of the subject while simultaneously feeling overconfident in their inaccurate knowledge

The study was conducted by Giwon Bahg from the Department of Psychology at Vanderbilt University, alongside Vladimir M. Sloutsky and Brandon M. Turner from the Department of Psychology at The Ohio State University. Previous scientific inquiries into personalization have often focused on how these systems reinforce existing beliefs, such as political ideologies or social attitudes. This phenomenon is often referred to as a “filter bubble.”

The research team sought to determine if these algorithms affect basic cognitive processes when a person attempts to learn about an entirely new topic where they have no prior opinions. They investigated whether the mechanism of tailoring content to increase consumption might inadvertently limit exposure to the broader environment.

This restriction could prevent users from forming an accurate mental map of reality. The researchers aimed to simulate how an individual might try to learn about a new domain, such as foreign cinema or a scientific concept, through a curated feed.

To test their hypothesis, the researchers recruited 343 participants through an online platform. After excluding data from sessions that were incomplete or failed to meet specific quality standards, the final analysis included 200 participants.

The researchers designed a task involving completely fictional categories to ensure that prior knowledge did not influence the results. Participants were asked to learn how to categorize strange, crystal-like “aliens.”

These digital creatures possessed six distinct visual features that defined their category. The features included location on a line, the radius of a circle, brightness, orientation, curvature, and spatial frequency. The goal for the participants was to learn the structure of these alien categories by observing various examples.

The experiment consisted of a learning phase followed by a testing phase. During the learning phase, the specific features of the aliens were initially hidden behind gray boxes. Participants had to click on the boxes to reveal specific features, a process the researchers called information sampling. This setup allowed the team to track exactly what information the participants chose to look at and what they ignored.

The researchers divided the participants into different groups to test the specific effects of algorithmic personalization. One group served as a control and viewed a random assortment of items with all features available to inspect. Another group engaged in active learning, where they freely chose which categories to study without algorithmic interference.

The experimental groups interacted with a personalization algorithm modeled after the collaborative filtering systems used by video-sharing platforms like YouTube. This algorithm tracked which features a participant tended to click during the trials.

It then recommended subsequent items that made it easier to continue that specific pattern of clicking. Consequently, the system created a feedback loop that presented items similar to those the user had already engaged with.

This setup mimicked how online platforms prioritize content engagement over information diversity to maximize revenue. The algorithm was trained to predict which items would result in the most clicks from the user. It then populated the user’s feed with those high-engagement items.

The data analysis revealed significant differences in how the different groups gathered information. Participants in the personalized conditions sampled substantially fewer features than those in the control or active learning groups.

As the learning phase progressed, these participants narrowed their focus even further. The data suggests that they tended to ignore dimensions of the aliens that the algorithm did not prioritize.

The analysis of sampling diversity used a measure called Shannon entropy. This metric showed that the personalized environment effectively trained users to pay attention to a limited slice of the available information. The algorithm successfully constrained the diversity of the categories presented to the users.

Following the learning phase, the researchers administered a categorization task to measure what the participants had learned. They showed the participants new alien examples and asked them to sort them into the correct groups.

The researchers found that individuals who learned through the personalized algorithm made more errors than those in the control group. Their internal representation of the alien categories was distorted.

The algorithm had prevented them from seeing the full variety of the alien population. This led to inaccurate generalizations about how the different features related to one another. The participants effectively learned a skewed version of the reality presented in the experiment.

In addition to accuracy, the study measured the participants’ confidence in their decisions using a rating scale from zero to ten. The analysis showed that participants in the personalized groups frequently reported high confidence levels even when their answers were wrong. This effect was particularly distinct when they encountered items from categories they had rarely or never seen during the learning phase.

Instead of recognizing their lack of knowledge regarding these unfamiliar items, the participants incorrectly applied their limited experience. The results show that when a test item came from an unobserved category, the participants did not report low confidence. They felt sure that their biased knowledge applied to these novel situations.

This indicates a disconnection between actual competence and perceived competence caused by the filtered learning environment. The participants were unaware that the algorithm had hidden significant portions of the information landscape from them. They assumed the limited sample they viewed was representative of the whole.

The authors note that the study utilized a highly controlled, artificial task to isolate the cognitive effects of the algorithms. Real-world interactions with personalization often involve complex semantic content and emotional preferences, which were not present in this experiment. The synthetic nature of the stimuli was a necessary design choice to rule out the influence of pre-existing beliefs.

Future research could investigate how these findings translate to more naturalistic settings, such as news consumption or educational tools. The researchers also suggest exploring how different types of user goals might mitigate the negative effects of personalization. For instance, an algorithm designed to maximize diversity rather than engagement might yield different cognitive outcomes.

The findings provide evidence that the structure of information delivery systems plays a significant role in shaping human cognition. By optimizing for engagement, current algorithms may inadvertently sacrifice the accuracy of user knowledge. This trade-off suggests that online platforms can shape not just what people see, but how they reason about the world.

The study, “Algorithmic Personalization of Information Can Cause Inaccurate Generalization and Overconfidence,” was authored by Giwon Bahg, Vladimir M. Sloutsky, and Brandon M. Turner.