Reddit has recently tightened its grip on who can access its content, blocking major search engines from indexing recent posts and comments.
This move has sparked discussions in the SEO and digital marketing communities about the future of content accessibility and AI training data.
First reported by 404 Media, Reddit updated its robots.txt file, preventing most web crawlers from accessing its latest content.
Google, however, remains an exception, likely due to a $60 million deal that allows the search giant to use Reddit’s content for AI training.
Brent Csutoras, founder of Search Engine Journal, offers some context:
“Since taking on new investors and starting their pathway to IPO, Reddit has moved away from being open-source and allowing anyone to scrape their content and use their APIs without paying.”
Currently, Google is the only major search engine able to display recent Reddit results when users search with “site:reddit.com.”
This exclusive access sets Google apart from competitors like Bing and DuckDuckGo.
For users who rely on appending “Reddit” to their searches to find human-generated answers, this change means they’ll be limited to using Google or search engines that pull from Google’s index.
It presents new challenges for SEO professionals and marketers in monitoring and analyzing discussions on one of the internet’s largest platforms.
Reddit’s move aligns with a broader trend of content creators and platforms seeking compensation for using their data in AI training.
As Csutoras points out:
“Publications, artists, and entertainers have been suing OpenAI and other AI companies, blocking AI companies, and fighting to avoid using public content for AI training.”
While this development may seem surprising, Csutoras suggests it’s a logical step for Reddit.
He notes:
“It seems smart on Reddit’s part, especially since similar moves in the past have allowed them to IPO and see strong growth for their valuation over the last two years.”
Reddit has updated its robots.txt file to block major search engines from indexing its latest posts and comments. This change exempts Google due to a $60 million deal, allowing Google to use Reddit’s content for AI training purposes.
Google has exclusive access to Reddit’s latest content because of a $60 million deal that allows Google to use Reddit’s content for AI training. This agreement sets Google apart from other search engines like Bing and DuckDuckGo, which are unable to index new Reddit posts and comments.
Reddit’s decision to limit search engine access aligns with a larger trend where content creators and platforms seek compensation for the use of their data in AI training. Many publications, artists, and entertainers are taking similar actions to either block or demand compensation from AI companies using their content.
Featured Image: Mamun sheikh K/Shutterstock
