nayaread
HomeFavouritesTopicsSources
HomeFavouritesTopicsSources
NayaRead links to original engineering blogs. All copyright belongs to the respective authors.
The Cloudflare Blog·Aug 4, 2025·13 min read58

Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives

Perplexity is repeatedly modifying their user agent and changing IPs and ASNs to hide their crawling activity, in direct conflict with explicit no-crawl preferences expressed by websites.

Read original on blog.cloudflare.com

Discussion around the web

  • Hacker News1315 · 748 💬

Score breakdown

  • Technical depth91
  • Practical value45
  • Originality60
  • Writing quality82
  • Source reputation90
  • Recency0
  • External engagement44
  • On-site engagement10

NayaRead shows a summary and links to the original article. The full article is hosted on blog.cloudflare.com.

More like this

Engineering at Meta

Zoncolan: How Facebook uses static analysis to detect and prevent security issues

Aug 15, 2019·10 min read
Pinterest Engineering

Smarter URL Normalization at Scale: How MIQPS Powers Content Deduplication at Pinterest

Pinterest Engineering·Apr 20, 2026·11 min read
Dropbox Tech

Investigating the impact of HTTP3 on network latency for search

Tiffany Fong·May 22, 2023·8 min read
CNCF Case Studies

Ericom

Sep 9, 2022·8 min read