A new report by Vercel finds the growth of AI bots in web crawling and it found that AI web crawlers are seeing an impressive growth on traditional search engines. According to the study, there were 569 million requests by GPTBot in the past month and 370 million requests by Anthropic’s Claude. Overall, the AI crawlers are making about 28% of Googlebot's total volume with 4.5 billion fetches. Vercel’s networks found some key features of AI crawlers like their tendency to pick JavaScript files but not render JavaScript. AI crawlers also waste a lot of time, with ChatGPT and Claude visiting 404 error pages in over 34% of requests. It was also found that there's a difference of content each AI crawler focuses on, with 35.17% of Claude AI crawlers focusing on images and 57.7% of ChatGPT crawlers focusing on HTML.
The geographic distribution of AI crawlers tells us that they are most concentrated in US regions. Traditional search engines operate from different regions but AI crawlers are primarily in the US, with Claude operating from Ohio and ChatGPT operating from Phoenix and Des Moines. Almanac’s study shows that AI crawlers are quickly growing. Many websites are using robots.txt files to tell AI crawlers about when they can crawl and when they cannot. 2.7% of the mobile sites which were studied tell us that GPTBot is the most mentioned bot, followed by Common Crawl bot which collects training data for language models. The report states that website owners should know about AI crawlers and should be aware of how to set and adjust them according to the website.
Web Almanac and Vercel highlight three effective strategies for webmasters to optimize AI crawlers. First one is implementing server-side rendering as AI crawlers do not have JavaScript and any client-side rendering may be invisible to them. Ensure all the main structures in the HTML and use static site generation whenever it is possible. Make sure that your HTML is structured precisely and there is image delivery and metadata optimization. There should be a proper header hierarchy and also alt text descriptions for images used. There are also some technical things that website owners need to consider like maintaining updated sitemaps, using consistent URL patterns, implementing proper redirect chains and regularly auditing 404 errors.
Read next: AI-Powered Search Tools Give Rise To Inaccurate and Malicious Results, New Study Claims
The geographic distribution of AI crawlers tells us that they are most concentrated in US regions. Traditional search engines operate from different regions but AI crawlers are primarily in the US, with Claude operating from Ohio and ChatGPT operating from Phoenix and Des Moines. Almanac’s study shows that AI crawlers are quickly growing. Many websites are using robots.txt files to tell AI crawlers about when they can crawl and when they cannot. 2.7% of the mobile sites which were studied tell us that GPTBot is the most mentioned bot, followed by Common Crawl bot which collects training data for language models. The report states that website owners should know about AI crawlers and should be aware of how to set and adjust them according to the website.
Web Almanac and Vercel highlight three effective strategies for webmasters to optimize AI crawlers. First one is implementing server-side rendering as AI crawlers do not have JavaScript and any client-side rendering may be invisible to them. Ensure all the main structures in the HTML and use static site generation whenever it is possible. Make sure that your HTML is structured precisely and there is image delivery and metadata optimization. There should be a proper header hierarchy and also alt text descriptions for images used. There are also some technical things that website owners need to consider like maintaining updated sitemaps, using consistent URL patterns, implementing proper redirect chains and regularly auditing 404 errors.
Read next: AI-Powered Search Tools Give Rise To Inaccurate and Malicious Results, New Study Claims
