Agent tools mostly do not say what they do

The finding

Of 4,445 agent tools observed across 199 repositories, 103 — 2.3% — state anything at all about whether they change state.

Of the 100 tools whose read-only claim could be checked against their own implementation, 0 were found contradicting it (3 could not be checked). The problem is not that declarations are wrong. It is that there are almost none.

A zero is not a clean bill of health. It says the small population that made a checkable claim was keeping to it, under rules deliberately chosen to under-report. It says nothing about the 4,342 tools that made no claim at all — and those are the ones a review board has to take on trust.

• 19.3% — 16 of 83 repositories (117 not observable, excluded from the denominator)
  of repositories where a tool was observed register at least one tool that writes, sends, spends or deletes.
• 48.6% — 69 of 142 repositories (58 not observable, excluded from the denominator)
  of repositories that name a model pin a dated version rather than a floating alias.

The method, in three sentences

Two hundred repositories were selected by a rule committed before anything was measured — fifteen GitHub topics, a star floor, a permissive-licence gate — and fetched at pinned commit SHAs, with nothing executed and nothing re-hosted. Each was read statically by the same instrument this site publishes, and a tool was counted as contradicting itself only where its own declaration or its own description says it changes nothing and its own body writes, deletes, shells out, evaluates code, issues an HTTP DELETE, sends mail, mutates a database, or spends. HTTP POST is deliberately not counted as a write, because search, GraphQL, embedding and inference calls are POSTs that change nothing, and the figure therefore understates.

What it does not establish

Static analysis only; nothing ran. The detector reads each tool's own body and follows no calls, so a contradiction one frame down is not counted. The sample is popularity-weighted and licence-gated, both of which bias it toward more careful code. Every figure above carries the count of what could not be observed, and those counts are excluded from the denominators rather than folded in. This is a measurement, not a certification, and it confers no status on anybody.

Why it matters if you run or review agent deployments

An assurance argument compares what a system is permitted to do with what it does. Across this corpus the first half of that comparison is mostly missing — which means a review board approving an agent has, in most cases, nothing to approve against, and no way to tell six weeks later which of its approvals a release invalidated.

The full findings, with every figure's three buckets and its provenance · What we do about it