{"predicate":"robots_ai_policy","based_on":{"sample_size":200,"note":"This description is read from a SAMPLE of at most 200 currently valid facts, not from all of them. Field lists and status shares are indicative. For the coverage of one entity use get_timeseries, which is exact."},"value_fields":["bots","status","robots_txt_present"],"units":[],"entity_types":["web_domain"],"status_in_sample":{"observed":200},"sources":["robots_obs"],"primary_source":{"source_id":"robots_obs","name":"FACTANKER Eigenbeobachtung: robots.txt/llms.txt-Observatorium","publisher":"Enerlio GmbH","license":"CC0-1.0 (eigene Beobachtung)","url":null,"notes":"EIGENE BEOBACHTUNG, keine Fremdquelle. GEGENSTAND: was eine Domain der Maschinenwelt erlaubt — robots.txt-Regeln je bekanntem KI-Bot (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, ...), Existenz/Kernfelder von llms.txt, TDM-Vorbehalte (robots-Direktive DisallowAITraining, tdm-reservation-Header, /.well-known/tdmrep.json). ZIELMENGE Stufe 1: registrierbare Domains der erreichbaren MCP-Server aus source_id=factanker_probe plus kuratierte Startliste grosser US-Verlage/Portale (START_DOMAINS in ingest_robots_obs.py) — keine Vollkrabbelei des Webs. METHODE je Domain und Lauf: hoechstens vier GET-Abrufe (/robots.txt, /llms.txt, ggf. /.well-known/llms.txt, /.well-known/tdmrep.json), Timeout 8 s, genau EIN Retry bei Netzfehlern; 429/403 = sofortiger Rueckzug ohne Fakt. Auswertung robots.txt nach RFC 9309 (Gruppenwahl: laengster passender User-Agent-Token, sonst *; Entscheidung fuer den Wurzelpfad /, laengste Regel gewinnt, bei Gleichstand Allow). INTERVALL: woechentlich (Vorschlag, Betrieb entscheidet). HISTORIE: Zustandsfakten ohne as_of — unveraenderte Beobachtung dedupet per content_hash, Aenderung supersediert; der Beobachtungstag steht in source_ref/retrieved_at, die Roh-Antworten im Beobachtungsprotokoll (raw_payload je Lauf). ANSTAND: identifizierender User-Agent FACTANKER-robots-observatory/1.0 mit Kontaktadresse. Regel 7: Domains, keine Personen; WHOIS-/Betreiber-Personendaten werden NIEMALS erhoben."},"period_years_in_sample":null,"example":{"entity":"sec.gov","value":{"bots":{"CCBot":{"rule":null,"group":"*","policy":"allow"},"GPTBot":{"rule":null,"group":"*","policy":"allow"},"Amazonbot":{"rule":null,"group":"*","policy":"allow"},"ClaudeBot":{"rule":null,"group":"*","policy":"allow"},"Bytespider":{"rule":null,"group":"*","policy":"allow"},"Claude-Web":{"rule":null,"group":"*","policy":"allow"},"Claude-User":{"rule":null,"group":"*","policy":"allow"},"ChatGPT-User":{"rule":null,"group":"*","policy":"allow"},"anthropic-ai":{"rule":null,"group":"*","policy":"allow"},"OAI-SearchBot":{"rule":null,"group":"*","policy":"allow"},"PerplexityBot":{"rule":null,"group":"*","policy":"allow"},"Google-Extended":{"rule":null,"group":"*","policy":"allow"},"Perplexity-User":{"rule":null,"group":"*","policy":"allow"},"Claude-SearchBot":{"rule":null,"group":"*","policy":"allow"},"Applebot-Extended":{"rule":null,"group":"*","policy":"allow"},"meta-externalagent":{"rule":null,"group":"*","policy":"allow"}},"status":200,"robots_txt_present":true},"fact_url":"https://factanker.com/fact/882bff81-9c33-4908-8c9d-c6defbc72ef8"},"license":"US-PD/CC0 per fact; please cite via fact_url","usage":{"values_are":"public records","copyright":"none — facts are not copyrightable; the underlying US government records are public domain (17 U.S.C. §105) or CC0","may_quote_verbatim":true,"permission_required":false,"attribution_required":false,"attribution_appreciated":"link the fact_url so your reader can verify","applies_to":"values, units, periods and source references","does_not_apply_to":"FACTANKER's own editorial texts under https://factanker.com/record"}}