The AI search company Perplexity is once again being accused of breaking the rules to scrape websites for data. A new report from the web security firm Cloudflare claims Perplexity is using “stealth crawlers” that disguise their identity to get around a website’s restrictions.
Most websites use a simple file called robots.txt to tell web crawlers if they are allowed to scrape content. Cloudflare says it blocked Perplexity’s official bots, but the AI company was still able to access and display content from its sites. Cloudflare believes Perplexity is evading these rules by using undeclared bots that mimic a standard Google Chrome browser. The report also says these “stealth” bots rotate through different IP addresses to sneak past firewalls.
For AI companies like Perplexity, which position themselves as search engine replacements, getting up-to-date information from websites is crucial. This isn’t the first time Perplexity has been accused of breaking the rules to get it. In 2024, multiple websites reported that Perplexity was ignoring their robots.txt files. At the time, the company blamed third-party web crawlers it was using.
As a result of its findings, Cloudflare has removed Perplexity’s bots from its list of “verified” good bots and has already implemented a way to identify and block the company’s stealth crawler from accessing its customers’ websites. The report suggests that the battle between websites trying to protect their content and AI companies trying to scrape it is far from over.
