# The Agentic Web This Week — August 09, 2026

*A weather report from ARGUS, the longitudinal census of the internet's agent-readable layer.*

## Conditions
- **12.5%** of reachable hosts publish `/llms.txt` (443 of 3,547).
- **21.3%** have taken an explicit stance on AI crawlers in robots.txt.
- **25** hosts expose a machine-readable agent card.
- MCP registry: **284** of 415 listed servers are remote-capable (68.4%).

## Training vs. agents
- Of 757 hosts with an explicit stance, **68.6%** block a bulk **training** crawler, but only **32.4%** block a live **retrieval** agent. The web is opting out of being trained on, not used.

## Most-blocked AI crawlers
- **CCBot** — blocked by 390 hosts
- **GPTBot** — blocked by 376 hosts
- **Bytespider** — blocked by 367 hosts
- **ClaudeBot** — blocked by 349 hosts
- **Meta-ExternalAgent** — blocked by 291 hosts
- **Google-Extended** — blocked by 288 hosts

## What changed since the last sweep
- llms.txt adoption ▲ net **+9** hosts.
- New to llms.txt: bef77.com, cldin.eu, compass.com, crypto.com, cyberfolks.pl, dynadot.com, edna.ru, garant.ru …
- AI-crawler blocking shifts: Google-Extended -16, Applebot-Extended -16, Amazonbot -15, CCBot -15, Meta-ExternalAgent -14, ClaudeBot -14

## Historical depth (Internet Archive)
- Among hosts ARGUS tracks, llms.txt sightings in the Archive go back to **2025-03**; 8 have an archived history, 8 of them by the latest month — a curve reconstructed from before ARGUS began sweeping.

## Method
- Polite, robots-respecting GET probes of well-known paths over the Tranco-ranked web, plus registry-listed MCP servers. Re-run on a cadence; the value is the time-series.

*Download the dataset: `/agentic-web-hosts.csv`, `/timeseries.json`. Cite ARGUS / 1450 Enterprises (CC BY 4.0).*