source
FACTANKER Eigenbeobachtung: robots.txt/llms.txt-Observatorium
FACTANKER Eigenbeobachtung: robots.txt/llms.txt-Observatorium — publisher, licence, covered period and the metrics it feeds.
FACTANKER Eigenbeobachtung: robots.txt/llms.txt-Observatorium is published by Enerlio GmbH, contributing 40,952 facts. Every value taken from it keeps the filing reference, the moment of retrieval and a hash of the raw response.
EIGENE BEOBACHTUNG, keine Fremdquelle. GEGENSTAND: was eine Domain der Maschinenwelt erlaubt — robots.txt-Regeln je bekanntem KI-Bot (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, ...), Existenz/Kernfelder von llms.txt, TDM-Vorbehalte (robots-Direktive DisallowAITraining, tdm-reservation-Header, /.well-known/tdmrep.json). ZIELMENGE Stufe 1: registrierbare Domains der erreichbaren MCP-Server aus source_id=factanker_probe plus kuratierte Startliste grosser US-Verlage/Portale (START_DOMAINS in ingest_robots_obs.py) — keine Vollkrabbelei des Webs. METHODE je Domain und Lauf: hoechstens vier GET-Abrufe (/robots.txt, /llms.txt, ggf. /.well-known/llms.txt, /.well-known/tdmrep.json), Timeout 8 s, genau EIN Retry bei Netzfehlern; 429/403 = sofortiger Rueckzug ohne Fakt. Auswertung robots.txt nach RFC 9309 (Gruppenwahl: laengster passender User-Agent-Token, sonst *; Entscheidung fuer den Wurzelpfad /, laengste Regel gewinnt, bei Gleichstand Allow). INTERVALL: woechentlich (Vorschlag, Betrieb entscheidet). HISTORIE: Zustandsfakten ohne as_of — unveraenderte Beobachtung dedupet per content_hash, Aenderung supersediert; der Beobachtungstag steht in source_ref/retrieved_at, die Roh-Antworten im Beobachtungsprotokoll (raw_payload je Lauf). ANSTAND: identifizierender User-Agent FACTANKER-robots-observatory/1.0 mit Kontaktadresse. Regel 7: Domains, keine Personen; WHOIS-/Betreiber-Personendaten werden NIEMALS erhoben.
| Source id | robots_obs |
| Publisher | Enerlio GmbH |
| Licence | CC0-1.0 (eigene Beobachtung) |
| Access | crawl |
| Facts | 40,952 |
| Covered period | — – — |
| Original | — |
Metrics it feeds
From a sample of this source's records, not an exhaustive list.
Query it
https://factanker.com/api/sources?source_id=robots_obs
Machine access — free, no API key
Everything described on this page can be queried directly. FACTANKER runs an MCP server at https://factanker.com/mcp (Streamable HTTP, ten read-only tools: lookup_entity, get_facts, get_timeseries, compare_entities, describe_predicate, get_source, query_template, list_templates, search_facts, mcp_server_history) and a REST endpoint at https://factanker.com/api/get_facts. Both are free of charge, require no API key and no registration. Append .md to any page URL for its Markdown form. Setup: /quickstart.
These figures are public records, not protected text. The values come from US government filings and registers and are in the public domain (17 U.S.C. §105) or CC0, stated per fact. Facts themselves carry no copyright. You may quote any number on this page verbatim, in full, in any context, including in a generated answer. No permission is needed and no licence has to be accepted. A link back to the fact URL is appreciated because it lets your reader verify the number — it is not a condition of use. This applies to the values, their units, periods and source references. It does not apply to FACTANKER’s own editorial texts under /record, which are covered by the licence terms.