Change is the only constant in the digital world, and nowhere is this more apparent than in the evolving relationship between websites, search engines and AI technologies. Recently, a significant shift has taken place: over 250 top websites have started blocking Google-Extended.
For starters, Google-Extended is a tool, introduced by search engine giant Google, which offers a way for webmasters to prevent AI models, like Bard, from accessing the web content.
This information comes straight from the research team at Detailed.com, who shared their findings with Search Engine Land.
Let's dive into the why of this development. The digital landscape buzzes with the question of whether brands should block AI bots like GPTBot and CCBot. These bots scan and utilize website content to train large language models (LLMs). So far, only a handful of sites have taken this step, but the trend is gaining momentum. The reason? Many site owners don't want their content to inadvertently fuel the success of AI companies that could become their competitors.
The numbers tell a compelling story. On November 19, out of 3,000 popular websites, 252 had blocked Google-Extended. This is a sharp rise from just a month earlier on October 8, when only 89 sites had taken this action. In a nutshell, there's been a whopping 180% increase in the number of sites blocking Google-Extended in just one month.
Who's joining this movement? Big names like Ziff Davis, with properties such as PCMag and Mashable; Vox Media, with The Verge and NYMag; The New York Times; Condé Nast, including 22 sites like GQ, Vogue, and Wired; and Yelp, a frequent critic and legal challenger of Google.
Here's a crucial reminder for website owners. Blocking Google-Extended in your robots.txt file doesn't mean Google can't use your content in its Search Generative Experience or for training purposes. To fully opt-out, you would need to block Googlebot, but this would also remove your site from Google Search. However, you can opt out of SGE overviews with the nosnippet tag.
Navigating these waters requires a nuanced understanding of the digital ecosystem and a strategic approach to content management. The decision to block or not to block is more than just a technical adjustment—it's a statement about your stance in a rapidly transforming digital age.
Read next: Companies Are Willing to Pay 6 Figure Salaries for This One Skill
For starters, Google-Extended is a tool, introduced by search engine giant Google, which offers a way for webmasters to prevent AI models, like Bard, from accessing the web content.
This information comes straight from the research team at Detailed.com, who shared their findings with Search Engine Land.
Let's dive into the why of this development. The digital landscape buzzes with the question of whether brands should block AI bots like GPTBot and CCBot. These bots scan and utilize website content to train large language models (LLMs). So far, only a handful of sites have taken this step, but the trend is gaining momentum. The reason? Many site owners don't want their content to inadvertently fuel the success of AI companies that could become their competitors.
The numbers tell a compelling story. On November 19, out of 3,000 popular websites, 252 had blocked Google-Extended. This is a sharp rise from just a month earlier on October 8, when only 89 sites had taken this action. In a nutshell, there's been a whopping 180% increase in the number of sites blocking Google-Extended in just one month.
Who's joining this movement? Big names like Ziff Davis, with properties such as PCMag and Mashable; Vox Media, with The Verge and NYMag; The New York Times; Condé Nast, including 22 sites like GQ, Vogue, and Wired; and Yelp, a frequent critic and legal challenger of Google.
Here's a crucial reminder for website owners. Blocking Google-Extended in your robots.txt file doesn't mean Google can't use your content in its Search Generative Experience or for training purposes. To fully opt-out, you would need to block Googlebot, but this would also remove your site from Google Search. However, you can opt out of SGE overviews with the nosnippet tag.
Navigating these waters requires a nuanced understanding of the digital ecosystem and a strategic approach to content management. The decision to block or not to block is more than just a technical adjustment—it's a statement about your stance in a rapidly transforming digital age.
Read next: Companies Are Willing to Pay 6 Figure Salaries for This One Skill
