It has only been one week since we saw the launch of SearchGPT from OpenAI.
However, things don’t seem to be going as planned. The company is facing a huge blow after some major news publishing websites have opted to block the tool from crawling their web pages.
The list entails 13 top news sites including the New York Times who have taken part in this block of the company’s web crawler (AKA OAI-SearchBot). Furthermore, the latter is designed to index data so that OpenAI can retrieve it and display accurate results linked to its users having SearchGPT.
OriginalityAI tracks all of the related materials and found that 14 out of the top 1000 publishers blocked OAI-SearchBot. Meanwhile, other publications present on the list included Wired, Vogue, Vanity Fair, GQ, and The New Yorker.
It’s all quite head-scratching, as per the CEO of Originalityai. Why would a publisher end up blocking it when, at the end of the day, publishers need traffic?
As revealed by OpenAI, SearchGPT stressed that the OAI-SearchBot doesn’t crawl for data collection on the web to train AI models. Instead, it’s designed for website owners to enable them to try the latest bot and ensure the page pops up across search results.
Without access to crawlers on websites, the SearchGPT offering from OpenAI risks its competence. After all, it’s competing with the likes of Google. There’s a complete lack of trust and more doubts related to search traffic here, it appears.
Meanwhile, another web crawler such as OpenAI ends up scooping online data for the sake of training AI. Many websites end up blocking the bot. And when we think about it, this makes sense. You want more traffic coming from top search engines but you don’t wish to give away any data for AI training of models that would compete against you.
For so long, OpenAI has been busy taking part in user data collection without attaining consent. It might be that publishers don’t trust the company when it claims that it won’t suck up their written material for the sake of training data.
Search results don’t always direct users to the websites used for creating the original content. A giant chunk of the goal is related to these latest AI search engines keeping users glued for a while by displaying summaries. If the publisher isn’t seeing a huge figure for traffic from search engines, then why bother enabling these web crawlers in the first place?
Experts also note how the ChatGPT maker has been busy cutting out deals related to publishers using content archives. It almost looks like this was done intentionally: first, get close to publishers by signing partnership deals, and then roll out the massive SearchGPT initiative.
We must add that seeing The New York Times here is a little astonishing. The latter has already rolled out a few lawsuits against OpenAI and its biggest software giant investor Microsoft. They have gone on to allege that the tech giants are working illegally to use their content to produce competing products.
For those who might not be aware, The New York Times does not authorize using its content for Generative Search or training of AI models without any form of consent being taken. This is even though we do or don’t block any bot from carrying out crawling of content online.
As part of its complaints against the company, the NYT says it’s noticing how many search engine giants are turning out to be more powerful in regards to AI and siphoning traffic that comes from these publishing sites.
Other mentions in the complaint include how defendants continue to make use of Microsoft’s Bing Search Index, which categorizes online material to produce responses that entail excerpts and a detailed summary of the Times articles that are much longer and detailed than those returned by classic search engines.
By giving out Times content without any permission taken, the defendant's tools undermine and cause damage to the relationship with its reader and deprive them of licenses, subscriptions, and ads, not to mention revenue attained through affiliates.
Read next:
• OpenAI Creates New Tool That Catches Students Cheating On Assignments With ChatGPT
• Elon Musk Called Out For Pushing False Narratives About X’s Success Again
However, things don’t seem to be going as planned. The company is facing a huge blow after some major news publishing websites have opted to block the tool from crawling their web pages.
The list entails 13 top news sites including the New York Times who have taken part in this block of the company’s web crawler (AKA OAI-SearchBot). Furthermore, the latter is designed to index data so that OpenAI can retrieve it and display accurate results linked to its users having SearchGPT.
OriginalityAI tracks all of the related materials and found that 14 out of the top 1000 publishers blocked OAI-SearchBot. Meanwhile, other publications present on the list included Wired, Vogue, Vanity Fair, GQ, and The New Yorker.
It’s all quite head-scratching, as per the CEO of Originalityai. Why would a publisher end up blocking it when, at the end of the day, publishers need traffic?
As revealed by OpenAI, SearchGPT stressed that the OAI-SearchBot doesn’t crawl for data collection on the web to train AI models. Instead, it’s designed for website owners to enable them to try the latest bot and ensure the page pops up across search results.
Without access to crawlers on websites, the SearchGPT offering from OpenAI risks its competence. After all, it’s competing with the likes of Google. There’s a complete lack of trust and more doubts related to search traffic here, it appears.
Meanwhile, another web crawler such as OpenAI ends up scooping online data for the sake of training AI. Many websites end up blocking the bot. And when we think about it, this makes sense. You want more traffic coming from top search engines but you don’t wish to give away any data for AI training of models that would compete against you.
For so long, OpenAI has been busy taking part in user data collection without attaining consent. It might be that publishers don’t trust the company when it claims that it won’t suck up their written material for the sake of training data.
Search results don’t always direct users to the websites used for creating the original content. A giant chunk of the goal is related to these latest AI search engines keeping users glued for a while by displaying summaries. If the publisher isn’t seeing a huge figure for traffic from search engines, then why bother enabling these web crawlers in the first place?
Experts also note how the ChatGPT maker has been busy cutting out deals related to publishers using content archives. It almost looks like this was done intentionally: first, get close to publishers by signing partnership deals, and then roll out the massive SearchGPT initiative.
We must add that seeing The New York Times here is a little astonishing. The latter has already rolled out a few lawsuits against OpenAI and its biggest software giant investor Microsoft. They have gone on to allege that the tech giants are working illegally to use their content to produce competing products.
For those who might not be aware, The New York Times does not authorize using its content for Generative Search or training of AI models without any form of consent being taken. This is even though we do or don’t block any bot from carrying out crawling of content online.
As part of its complaints against the company, the NYT says it’s noticing how many search engine giants are turning out to be more powerful in regards to AI and siphoning traffic that comes from these publishing sites.
Other mentions in the complaint include how defendants continue to make use of Microsoft’s Bing Search Index, which categorizes online material to produce responses that entail excerpts and a detailed summary of the Times articles that are much longer and detailed than those returned by classic search engines.
By giving out Times content without any permission taken, the defendant's tools undermine and cause damage to the relationship with its reader and deprive them of licenses, subscriptions, and ads, not to mention revenue attained through affiliates.
Read next:
• OpenAI Creates New Tool That Catches Students Cheating On Assignments With ChatGPT
• Elon Musk Called Out For Pushing False Narratives About X’s Success Again
