Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
Perplexity is repeatedly modifying their user agent and changing IPs and ASNs to hide their crawling activity, in direct conflict with explicit no-crawl preferences expressed by websites.
Discussion around the web
- Hacker News1315 · 748 💬
Score breakdown
- Technical depth91
- Practical value45
- Originality60
- Writing quality82
- Source reputation90
- Recency0
- External engagement44
- On-site engagement0
More like this
Engineering at Meta57
Zoncolan: How Facebook uses static analysis to detect and prevent security issues
unknown·10 min read
Pinterest Engineering61
Smarter URL Normalization at Scale: How MIQPS Powers Content Deduplication at Pinterest
Pinterest Engineering·unknown·11 min read
Dropbox Tech57
Investigating the impact of HTTP3 on network latency for search
Tiffany Fong·unknown·8 min read
Etsy — Code as Craft60
Improving performance by prefetching product pages from Etsy Search
David Weinzimmer·unknown·11 min read
Engradar shows a summary and links to the original article. The full article is hosted on blog.cloudflare.com.