Blocking AI Crawlers on Your Mastodon Server, Nginx Edition
Blocking AI Crawlers on your Mastodon Server, Nginx Edition I run but a small server; for the love of all things just leave it alone AI crawlers. Tue Jul 28, 2026 3 min read Justin Ribeiro https://justinribeiro.com Filed in oss mastodon nginx There are few things worse than getting an alert notification from a server at night. Your mind races—is my redundancy working, should I grab a fire extinguisher—knowing that the glaring light of your SSH session in bed will wake your spouse.
- ▪Blocking AI Crawlers on your Mastodon Server, Nginx Edition I run but a small server; for the love of all things just leave it alone AI crawlers.
- ▪Tue Jul 28, 2026 3 min read Justin Ribeiro https://justinribeiro.com Filed in oss mastodon nginx There are few things worse than getting an alert notification from a server at night.
- ▪Your mind races—is my redundancy working, should I grab a fire extinguisher—knowing that the glaring light of your SSH session in bed will wake your spouse.
Hacker News (AI / LLM) files mainly under ai. We currently carry 2,814 of its stories.
Story provenance
Source · retrieval · rights · ranking — open for full record
inspect →
Story provenance
Attribution is not the same as permission. This drawer separates discovery metadata, excerpts, WeSearch-generated summaries, reuse status, and whether the publisher receives the visit. Nothing here claims a legal grant the publisher has not made.
Record
| Original publisher | Justin Ribeiro, Ph.D. |
| Canonical URL | https://justinribeiro.com/chronicle/2026/07/28/blocking-ai-crawlers-on-your-mastodon-server-nginx-edition/ |
| Publication time | Wed, 29 Jul 2026 17:39:57 +0000 |
| Retrieval time | 2026-07-29T17:46:07.586Z |
| Last seen | 2026-07-29T17:46:07.586Z |
| Headline source | Publisher (no WeSearch rewrite) |
| Excerpt source | publisher body |
| Excerpt method | First ~120 words (~800 chars) of extracted publisher body, fair-use limited. |
| Summary | WeSearch · cerebras-chat (WeSearch summarizer) |
| Summary source text | contentText |
| Citation coverage | Summary is a WeSearch-generated derivative; primary citation is the original publisher URL. |
| Cluster | 3a4vecH9Hr2L · 1 stories |
| Cluster logic | Grouped by semantic title/content similarity across sources within a rolling window. Same-publisher template collisions are excluded from coverage comparison. |
| Ranking reason | Story pages are not engagement-ranked. Hub feeds use recency, with optional source-diversified chronological ordering (cap consecutive stories per source). No personalized ranking. |
| Publisher visit | Yes — open original |
| Substitutes article? | No — link-out required for full text |
Rights status (four layers)
WeSearch handling by dimension
| Indexing | May the item be indexed (stored, ranked, made findable)? | Allowed |
| Snippet | May a short excerpt of the publisher's text be shown? | Allowed |
| AI summary | May WeSearch generate its own short summary of the article? | Limited |
| Retrieval / RAG | May the content be exposed for third-party retrieval-augmented generation? | Not asserted |
| Model training | May the content be used to train AI models? | Not asserted |
| Commercial reuse | May the content be reused commercially? | Not permitted |
Basis: Derived from the published RSS/Atom feed. Contact: [email protected]. Reviewed: 2026-07-24.
Opening excerpt (first ~120 words) tap to expand
Blocking AI Crawlers on your Mastodon Server, Nginx Edition I run but a small server; for the love of all things just leave it alone AI crawlers. Tue Jul 28, 2026 3 min read Justin Ribeiro https://justinribeiro.com Filed in oss mastodon nginx There are few things worse than getting an alert notification from a server at night. Your mind races—is my redundancy working, should I grab a fire extinguisher—knowing that the glaring light of your SSH session in bed will wake your spouse. Then you’ll have two problems. Such an event found me the other day, for of all things, my single no-frills instance that runs my Mastodon server. This is a tiny server with few resources because it doesn’t need them; I am the only daily active user with other family using it sparingly.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at Justin Ribeiro, Ph.D..
Discussion
0 commentsMore from Justin Ribeiro, Ph.D.
