Learning my lesson that Python virtual environments aren't always movable
The author of the blog Wandering Thoughts explains that visitors using old browser versions or certain user agents may be blocked due to anti-crawler measures. This is in response to a surge in high-volume crawlers, often using outdated Chrome user agents, potentially for LLM training data collection. The author clarifies that legitimate services like Inoreader and archive.org are not blocked, but some archiving services and browsers like Vivaldi may trigger restrictions.
- ▪The blog Wandering Thoughts is blocking access to browsers with old or suspicious user agents to combat high-volume crawlers.
- ▪Crawlers using old Chrome user agents are common, especially in efforts to gather data for training large language models.
- ▪Vivaldi users may need to adjust their 'User Agent Brand Masking' setting to identify as Vivaldi to avoid being blocked.
- ▪The author distinguishes between well-behaved archivers like archive.org and problematic ones like archive.today, which use indistinguishable malicious-like crawl patterns.
- ▪Inoreader's feed fetcher is not blocked, but users may see the warning page if the service uses outdated user agents for page fetches.
Opening excerpt (first ~120 words) tap to expand
You're using a suspiciously old browser You're probably reading this page because you've attempted to access some part of my blog (Wandering Thoughts) or CSpace, the wiki thing it's part of. Unfortunately you're using a browser version that my anti-crawler precautions consider suspicious, most often because it's too old (most often this applies to versions of Chrome). Unfortunately, as of early 2025 there's a plague of high volume crawlers (apparently in part to gather data for LLM training) that use a variety of old browser user agents, especially Chrome user agents. To reduce the load on Wandering Thoughts I'm experimenting with (attempting to) block all of them, and you've run into this.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at Utoronto.
Discussion
0 commentsMore from Utoronto
