Thanks to a new AI deal, Google is now the only search engine that can scrape Reddit to show new posts in search results; competitors like Bing and DuckDuckGo will have to pay the price if they want users to see the same thing

This is according to a new report from 404 Media, which was the first to discover that search engines other than Google are not showing Reddit results from last week in their search results If, like me, you frequently tag “Reddit” or “site:redditcom” in your search queries, you will find that Bing and DuckDuckGo search results are disappointing You will either see outdated Reddit posts or a link that says, “I would like to show a description here, but the site won't let me”

Reddit spokesman Tim Rathschmidt told the Verge that this development “has nothing to do with our recent partnership with Google”

“We have been in discussions with several search engines,” Reddit spokesman Tim Rathschmidt told the Verge We have not been able to reach agreement with all of the search engines because “some are unable or unwilling to make enforceable commitments regarding the use of Reddit's content, including its use for AI”

In other words, rather than some sort of exclusivity agreement, it appears to be related to Reddit's recent crackdown on web crawlers Earlier this month, Reddit tweaked the site's robotstxt file to tighten its policy against scraping By preventing certain search engines from scraping pages with AI bots, Reddit is essentially eliminating content from search results In short, Reddit is trying to take its ball and go home Unless, of course, Google wants to follow that rule

Microsoft spokeswoman Caitlin Roulston confirmed to the Verge that Bing stopped collecting content from Reddit after it updated its robotstxt file on July 1 She added, “Microsoft respects the standards of robotstxt and honors instructions provided by websites that do not want to use the page's content in our generated AI model”

In other words, for now, Google will have exclusive rights to Reddit's content in its search results At least until Reddit reaches an agreement with a competitor