# Terms for automated access (rate, identification, no rotating clients): # https://mcpobservatory.com/terms User-agent: * Allow: / # Don't crawl the FACET space. The faceted browser generates a combinatorial set # of filter permutations that all resolve to the same canonical content — # crawling it wastes crawl budget and hammers the DB-backed query path. # # These are named one param at a time, and deliberately NOT collapsed back to a # blanket `Disallow: /*?`. That blanket rule is what this replaces, and it was # costing the entire catalogue: /servers paginates with `?page=N`, so the rule # forbade Googlebot from following the pager past page 1. The listing links 26 # servers and the six permission pages ~25 each, so ~176 of 72,645 server pages # had ANY crawlable internal link — the rest were reachable only from the # sitemap, which announces that a URL exists but gives it no importance signal. # # It also contradicted the application: src/pages/servers/index.astro sets # `X-Robots-Tag: noindex` for filtered views ONLY, states in its own comment that # "bare /servers (and plain ?page=N pagination) stays indexable", and keeps # `page` in the canonical (`canonicalParams={["page"]}`). robots.txt was # overruling all of that. The two now agree: facets are closed, paging is open. # # Unlisted params (?utm_source=, ?fbclid=, ?ref=) stay crawlable and are handled # correctly rather than blocked: they render the unfiltered result set and # canonicalise back to the bare path, so they self-resolve as duplicates. # (`*`/`$` are the standard wildcard extensions; `?` and `&` are literal.) Disallow: /*?*q= Disallow: /*&q= Disallow: /*?*perm= Disallow: /*&perm= Disallow: /*?*transport= Disallow: /*&transport= Disallow: /*?*verified= Disallow: /*&verified= Disallow: /*?*kind= Disallow: /*&kind= Disallow: /*?*minTools= Disallow: /*&minTools= Disallow: /*?*confidence= Disallow: /*&confidence= # `sort` is a facet for this purpose: a non-default ordering is a distinct # permutation of the same set, and index.astro's `isFiltered` already counts it # as one. The pager omits `sort` at the default, so paging stays crawlable. Disallow: /*?*sort= Disallow: /*&sort= # The live event stream (Server-Sent Events) never finishes and is not a page: # Googlebot fetched it 22 times in 14 days (Apache log, 2026-11-03..17), about # one in ten of its visits, each one a held connection with nothing to index. Disallow: /api/events # /keys is the API-key sign-in page: per-visitor, noindex, nothing to crawl. Disallow: /keys # The private admin panel (GitHub sign-in, allowlisted). Disallow: /admin # SEO BACKLINK CRAWLERS — blocked site-wide. # # Measured against the Apache access log on 2026-09-19, counting requests to the # advisory permalink archive by user agent: AhrefsBot 694, LivelapBot 431, # serpstatbot 25 — 1,150 requests, far ahead of the next crawler (bingbot, 137). # None of the three send readers; they crawl to build backlink databases they # sell access to. # # The archive is their worst case rather than their only target: 44,901 # near-duplicate `noindex` permalinks, fully enumerable 25 at a time from # /security/drift's deliberately-crawlable `?page=N` pager, and every hit is a # guaranteed cache MISS (a crawler walking 44,901 distinct URLs once each never # revisits one inside its 300s TTL, so mod_cache absorbs none of it and each # request is a full SSR render with four DB reads). Blocking just that path # would move them onto the 83,208 server pages, so the rule is site-wide. # # Deliberately NOT blocked: bingbot and Googlebot, which have to FETCH a page to # see its `noindex` — denying them the fetch is what leaves a URL indexed with # no content. The browser-UA long tail is also left alone; some of it is real # readers and a UA string can't tell us which. # # A crawler matching a named group ignores the `*` group entirely (robots.txt # precedence is most-specific-group-wins, not additive) — which is why these # carry a bare `Disallow: /` and don't repeat the facet patterns above. User-agent: AhrefsBot User-agent: DataForSeoBot User-agent: SemrushBot User-agent: MJ12bot User-agent: serpstatbot User-agent: LivelapBot User-agent: Barkrowler # 2026-11-14 access log (564k requests): QlyzeBot 27,565, SERankingBacklinksBot # 14,585, DotBot 1,926. Also 403'd by ~/lib/botBlock for when they don't listen. User-agent: QlyzeBot User-agent: SERankingBacklinksBot User-agent: DotBot Disallow: / # AI TRAINING CRAWLERS — blocked site-wide. # # Same 2026-09-19 access log, counted across the WHOLE site rather than one # path: meta-externalagent 102,447 (summed over the 16 UA variants it rotates # through), GPTBot 35,573, ClaudeBot 1,936, ShapBot 1,185 — 141,141 requests. # That is ~22x the combined total of every search engine, social preview and # feed reader we serve (6,336), and Meta alone is about two thirds of all # crawler traffic here. # # These four scrape to build training corpora. They send no readers, surface no # citation, and return nothing for the CPU — which this box does not have spare # (the worker peaks at 1.44 of its 1.5-core cap). # # DELIBERATELY NOT BLOCKED — the AI crawlers that cite their sources: # PerplexityBot (2,010), OAI-SearchBot (51) and Google-NotebookLM (23) answer # questions with a link back, so they are a discovery channel in the same way # Googlebot is. meta-webindexer (25) is Meta's SEARCH indexer, not its training # agent, and stays open for the same reason. The Observatory is a public # accountability project: being findable and quotable serves that. Being # silently ingested at 100k requests a day does not. User-agent: meta-externalagent User-agent: GPTBot User-agent: ClaudeBot User-agent: ShapBot # Added 2026-11-15 from the 2026-11-14 log, the operator's call: ExaSearchBot # 30,308 (incl. ExaSearchBot-Images), ElevenlabsCrawler 12,033, Reflectionbot # 8,592, KeenableBot 4,947. Exa is an AI search engine, so this may cost some # AI-search visibility; the volume outweighed it. Also 403'd by ~/lib/botBlock. User-agent: ExaSearchBot User-agent: ExaSearchBot-Images User-agent: ElevenlabsCrawler User-agent: Reflectionbot User-agent: KeenableBot # Added 2026-11-17, the operator's call: Epochal-Search, 11,226 pages in a day # from 19 addresses, an unknown search startup. Also 403'd by ~/lib/botBlock. User-agent: Epochal-Search Disallow: / Sitemap: https://mcpobservatory.com/sitemap.xml