Classifying AI Website Fetches on Behalf of Users
The article explains how AI providers like OpenAI and Anthropic use server-side fetches to retrieve web content for user queries, which often bypasses traditional browser analytics. The author developed a tool called Canonry to distinguish these user-triggered fetches from automatic crawlers and other traffic types using server logs. Analysis of traffic data revealed that many requests claiming AI user identities did not match official IP ranges, suggesting potential misuse by scanners.
- ▪AI providers may fetch web pages server-side using specific user agents like ChatGPT-User or Claude-User, which prevents standard JavaScript-based analytics from firing.
- ▪The Canonry tool classifies server traffic into distinct categories including crawlers, user fetches, citations, and referrals to prevent misleading aggregate metrics.
- ▪In a seven-day test period, 947 requests used AI user-fetch identities, but only a small fraction matched the official IP ranges published by the providers.
- ▪Many unmatched requests exhibited patterns typical of security scanners, such as probing for sensitive files like .env and .ssh/id_rsa, despite using AI user agents.
Hacker News (AI / LLM) files mainly under ai. We currently carry 4,512 of its stories.
Story provenance
Source · retrieval · rights · ranking — open for full record
inspect →
Story provenance
Attribution is not the same as permission. This drawer separates discovery metadata, excerpts, WeSearch-generated summaries, reuse status, and whether the publisher receives the visit. Nothing here claims a legal grant the publisher has not made.
Record
| Original publisher | Canonry |
| Canonical URL | https://canonry.ai/blog/classifying-ai-crawlers-user-fetches-server-logs |
| Publication time | Sat, 12 Sep 2026 02:25:29 +0000 |
| Retrieval time | 2026-09-12T02:27:50.835Z |
| Last seen | 2026-09-12T02:27:50.835Z |
| Headline source | Publisher (no WeSearch rewrite) |
| Excerpt source | publisher body |
| Excerpt method | First ~120 words (~800 chars) of extracted publisher body, fair-use limited. |
| Summary | WeSearch · cerebras-chat (WeSearch summarizer) |
| Summary source text | contentText |
| Citation coverage | Summary is a WeSearch-generated derivative; primary citation is the original publisher URL. |
| Cluster | ba29gtTO0487 · 1 stories |
| Cluster logic | Grouped by semantic title/content similarity across sources within a rolling window. Same-publisher template collisions are excluded from coverage comparison. |
| Ranking reason | Story pages are not engagement-ranked. Hub feeds use recency, with optional source-diversified chronological ordering (cap consecutive stories per source). No personalized ranking. |
| Publisher visit | Yes — open original |
| Substitutes article? | No — link-out required for full text |
Rights status (four layers)
WeSearch handling by dimension
| Indexing | May the item be indexed (stored, ranked, made findable)? | Allowed |
| Snippet | May a short excerpt of the publisher's text be shown? | Allowed |
| AI summary | May WeSearch generate its own short summary of the article? | Limited |
| Retrieval / RAG | May the content be exposed for third-party retrieval-augmented generation? | Not asserted |
| Model training | May the content be used to train AI models? | Not asserted |
| Commercial reuse | May the content be reused commercially? | Not permitted |
Basis: Derived from the published RSS/Atom feed. Contact: [email protected]. Reviewed: 2026-07-24.
Opening excerpt (first ~120 words) tap to expand
ResearchClassifying AI Crawlers and User Fetches from Server LogsArber Xhindoli · August 13, 2026 · 3 min readAuthorshipHuman ideas · AI-written · Human-reviewedArber Xhindoli developed the ideas and examples, directed the argument, and checked the final article. AI wrote the prose.OpenAI crawler documentationAnthropic crawler documentationCanonry on GitHubTable of contentsWhen the fetch happens on the provider's networkOne AI traffic number hides different eventsWhat showed up on canonry.aiServer logs are where AI crawlers and user fetches show up. I couldn't find a good self-hosted way to separate the two, so I built it into Canonry. When the fetch happens on the provider's network A question in ChatGPT or Claude goes to the provider, not your site.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at Canonry.
Discussion
0 commentsMore from Canonry
