New updates were recently made by Reddit to its robots.txt file and that has impacted several leading search engines.
Microsoft was the first to confirm the news of how Bing was not able to access the page by speaking to Search Engine Land on the topic that described the matter in detail. The software giant says the news comes after Reddit implemented a new robots.txt file at the start of this month that inhibits most crawling activities on the site.
Reddit underwent updates for the the respective robots.txt file that stopped both AI tools and search engines from carrying out crawling but what’s interesting is how Google was not impacted by this change, contrary to what others may have assumed before.
Seeing results from Reddit drop from Bing’s Search Index was noted by some tech experts recently and with time, the news started to spread through social media. Seeing so many of Bing’s crawlers blocked was a shock to some. Reddit was utilizing IP detection to display search engines one variant of its robots.txt while humans were provided another variant of the file.
Bing is now no longer crawling any new material on the Reddit platform. Hence, you might see nothing or very few outputs when Reddit results are filtered through Bing Search.
Microsoft rolled out a statement on this front including how it respected the decision that Reddit has taken and therefore will not be crawling after the new file was implemented that stops all crawling activity to the site starting July 1st of this year.
Reddit was quick to react and its representative has confirmed that the change is not linked in any way, shape, or form to its collaboration with Google. They are carrying out discussions with various search engine partners to attain a consensus where all parties are in agreement.
But the platform did mention that some are not willing to make or stay true to promises on how the app’s content could be used such as AI and therefore the results are now in front of the world.
Reddit has already entered into a licensing deal with Android maker Google and therefore it makes sense why it’s playing rough with other search engines and AI endeavors. What we have confirmed is that Reddit is blocking all other search engines from getting content.
We must confirm that Google is currently driving a huge amount of traffic to the Reddit app recently and that entails testing special kinds of treatment across search results.
This makes a lot of us wonder if bigger websites could experiment with a similar route and if the answer is yes, it could leave out those working on a smaller scale.
For now, we just want you to know that you will not be seeing too much Reddit material surfacing across Bing Search in the future so please be aware.
Image: DIW-Aigen
Read next: Bing Combines AI Into Search To Give Users The Best Generative Search Experience
Microsoft was the first to confirm the news of how Bing was not able to access the page by speaking to Search Engine Land on the topic that described the matter in detail. The software giant says the news comes after Reddit implemented a new robots.txt file at the start of this month that inhibits most crawling activities on the site.
Reddit underwent updates for the the respective robots.txt file that stopped both AI tools and search engines from carrying out crawling but what’s interesting is how Google was not impacted by this change, contrary to what others may have assumed before.
Seeing results from Reddit drop from Bing’s Search Index was noted by some tech experts recently and with time, the news started to spread through social media. Seeing so many of Bing’s crawlers blocked was a shock to some. Reddit was utilizing IP detection to display search engines one variant of its robots.txt while humans were provided another variant of the file.
Bing is now no longer crawling any new material on the Reddit platform. Hence, you might see nothing or very few outputs when Reddit results are filtered through Bing Search.
Microsoft rolled out a statement on this front including how it respected the decision that Reddit has taken and therefore will not be crawling after the new file was implemented that stops all crawling activity to the site starting July 1st of this year.
Reddit was quick to react and its representative has confirmed that the change is not linked in any way, shape, or form to its collaboration with Google. They are carrying out discussions with various search engine partners to attain a consensus where all parties are in agreement.
But the platform did mention that some are not willing to make or stay true to promises on how the app’s content could be used such as AI and therefore the results are now in front of the world.
Reddit has already entered into a licensing deal with Android maker Google and therefore it makes sense why it’s playing rough with other search engines and AI endeavors. What we have confirmed is that Reddit is blocking all other search engines from getting content.
We must confirm that Google is currently driving a huge amount of traffic to the Reddit app recently and that entails testing special kinds of treatment across search results.
This makes a lot of us wonder if bigger websites could experiment with a similar route and if the answer is yes, it could leave out those working on a smaller scale.
For now, we just want you to know that you will not be seeing too much Reddit material surfacing across Bing Search in the future so please be aware.
Read next: Bing Combines AI Into Search To Give Users The Best Generative Search Experience
