Reddit results will no longer be available on other search engines such as Microsoft’s Bing and DuckDuck, among others. According to 404 Media, Reddit has updated its robots.txt file to stop all bots from taking data from its site.
According to sources, Reddit has been facing a traffic surge from search engines. This is due to the overuse of Reddit links in several search results. This initiative can help Reddit to avoid this issue.
Reddit limits its availability in search engines
The initiative is expected to come after several reports of ‘ misuse’ of data or ‘illegal data scraping’ from various social media/public discussions platforms. As reported by 404 media, Reddit has started blocking the vast majority of the internet’s search engines from crawling and indexing their website. This means that Reddit posts will no longer come up in search results for users on these search engines. However, you can still search Reddit on Google.
It is believed that this move of ‘restricting’ content to other search engines might rise to controversies. However, Reddit clearly claims that this step only aims to control the misuse of data for training AI-enabled malicious activities.
As reported by Bloomberg, in February of this year Google announced a new deal with Reddit which would see Reddit data used for training Google’s AI models as well as more prominently showing Reddit results within Google Search. The deal was for about $60 million.
“We block all crawlers that are unwilling to commit to not using crawl data for AI training, which is in line with enforcing our Public Content Policy and updated robots.txt file,” Reddit said in a statement as reported by Engadget.
What’s next
Amidst the ‘misuse of data’ and ‘AI-enabled scams’ such as selling data in the dark web and identity theft, among others, Reddit aims to avoid such engines with this initiative. According to sources, Reddit had explained that “ Reddit believes in an open internet platform. It does not promote or believe in the use of public content secretly for fraudulent purposes.”
Furthermore, Reddit also explained that the change was being made following “an uptick in obviously commercial entities who scrape Reddit.’ It also highlighted the search engines are using Reddit content for any use case they want. However, Reddit did not mention that the ‘content is being used explicitly,’ it seems to be pretty clear that AI training is a clear focus for this change in policy.
Follow FE Tech Bytes on Twitter, Instagram, LinkedIn, Facebook
