Network Working Group E. Arsentev Internet-Draft Independent Intended status: Informational 15 November 2026 Expires: 19 May 2027 Reporting Crawler Traffic by Verification Level draft-arsentev-crawler-traffic-reporting-00 Abstract The User-Agent header is written by the client. It is a claim, not an identity. Reports of crawler and AI crawler traffic that count requests by User-Agent token alone present claims as activity. This document defines four counts a crawler traffic report should give per crawler token: declared, route-consistent, verified, and unverifiable. It states which verification check was possible at the vantage point where the log was taken. It also defines a small JSON report object. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 19 May 2027. Copyright Notice Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Arsentev Expires 19 May 2027 [Page 1] Internet-Draft Crawler Traffic Verification Levels November 2026 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Conventions and Terminology . . . . . . . . . . . . . . . . . 3 3. The Four Counts . . . . . . . . . . . . . . . . . . . . . . . 3 4. Stating the Vantage Point and the Checks . . . . . . . . . . 4 5. Report Object . . . . . . . . . . . . . . . . . . . . . . . . 4 6. Operational Experience . . . . . . . . . . . . . . . . . . . 5 7. Security Considerations . . . . . . . . . . . . . . . . . . . 6 8. Privacy Considerations . . . . . . . . . . . . . . . . . . . 6 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 6 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 6 10.1. Normative References . . . . . . . . . . . . . . . . . . 6 10.2. Informative References . . . . . . . . . . . . . . . . . 7 Appendix A. Example Report . . . . . . . . . . . . . . . . . . . 7 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 8 1. Introduction Site operators, CDNs, analytics vendors and researchers publish counts of "AI crawler traffic". They produce these counts by matching the User-Agent string (Section 10.1.5 of [RFC9110]) against lists of crawler tokens. The User-Agent value is chosen by the client. Crawler operators increasingly publish IP address ranges for their crawlers, and some offer reverse DNS checks. Work on cryptographic bot authentication with HTTP Message Signatures ([RFC9421]) is under way in the IETF (webbotauth). Whether a check can be applied depends on where the log is recorded. An origin server behind a CDN usually sees the CDN's address, not the client's address. A count that mixes checked and uncheckable requests cannot be compared across sites. The goal of this document is to make the verification level of every reported number explicit. This document does not define how to verify a crawler. It does not define access policy, which is the subject of robots.txt ([RFC9309]) and of AI preference work. It does not say whether unverified traffic should be blocked. Arsentev Expires 19 May 2027 [Page 2] Internet-Draft Crawler Traffic Verification Levels November 2026 2. Conventions and Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. crawler token: a product token a crawler operator documents for its crawler's User-Agent. vantage point: the component where the log record is produced (origin server, reverse proxy, CDN edge). expected route: the network path a legitimate request to the site takes given the site's deployment, for example through the site's CDN. published address range: a set of IP prefixes a crawler operator publishes as the source of its crawler. 3. The Four Counts For each crawler token and reporting period, the report gives the following counts. declared: requests whose User-Agent contains the token. This count is always available. route-consistent: declared requests that arrived by the expected route. A declared request that reached the origin directly, while the site is served through a CDN, is not route-consistent. verified: declared requests whose source was confirmed by a check the report names. The check is a published address range match, forward-confirmed reverse DNS, or a verified HTTP message signature. unverifiable: declared requests for which no check was possible at the vantage point, for example because the logged address is the CDN's address. The report MUST give the declared count together with at least one of the other three counts. A report MUST NOT present the declared count alone as crawler activity. Arsentev Expires 19 May 2027 [Page 3] Internet-Draft Crawler Traffic Verification Levels November 2026 Requests that were checked and failed are the declared count minus the verified count minus the unverifiable count. The report SHOULD state that number. When the operator of a token publishes no ranges and no other check exists, all requests under that token are unverifiable. The report says so instead of reporting zero verified. 4. Stating the Vantage Point and the Checks The report names the vantage point, the period, and the token list with its version or date. For each token, the report states which checks were available: address range, reverse DNS, message signature, or none. A count from a CDN edge and a count from an origin are different measurements. They are not added together. If address ranges were fetched, the report gives the date they were fetched, because ranges change. 5. Report Object The report is carried as a JSON object ([RFC8259]). The object carries one entry per crawler token with the four counts, the checks performed, and the vantage point. The example in the appendix uses illustrative numbers, not measurements. The report is a JSON object [RFC8259] with the following members. vantage_point String, REQUIRED. One of "origin", "reverse-proxy", "cdn-edge", "other". period Object, REQUIRED, with members "start" and "end", each a date-time as defined in [RFC3339]. token_list Object, REQUIRED, with members "source" (string naming the list of crawler tokens used) and "as_of" (full-date as defined in [RFC3339]). tokens Array, REQUIRED, one object per crawler token, with the members below. Arsentev Expires 19 May 2027 [Page 4] Internet-Draft Crawler Traffic Verification Levels November 2026 tokens[].token String, REQUIRED. The crawler token as matched. tokens[].declared Non-negative integer, REQUIRED. tokens[].route_consistent Non-negative integer, or null when the expected route could not be determined at the vantage point. tokens[].verified Non-negative integer, or null when no check was available. tokens[].unverifiable Non-negative integer, REQUIRED. tokens[].failed Non-negative integer, RECOMMENDED. Equal to declared minus verified minus unverifiable. tokens[].checks Array of strings, REQUIRED. Each one of "address-range", "reverse-dns", "http-signature", "none". tokens[].ranges_fetched full-date as defined in [RFC3339]. REQUIRED when "checks" contains "address-range". A report MUST satisfy, for every entry, verified + unverifiable + failed = declared when all three are present. Members not defined here MAY be added; consumers MUST ignore members they do not understand. 6. Operational Experience This section reports one case. The data come from one origin server behind a CDN, with full logs for 15 days. The server logged 345,808 requests in total. Of these, 46,155 carried one of 22 published crawler tokens. Of those declared requests, 35.7% reached the origin directly, bypassing the CDN. None of the 11,276 direct requests under tokens whose operators publish address ranges came from those ranges. In other words, these requests were not confirmed by the published ranges. The direct traffic never requested robots.txt or the sitemap. It used about 3.5 times more distinct User-Agent strings per request. Arsentev Expires 19 May 2027 [Page 5] Internet-Draft Crawler Traffic Verification Levels November 2026 A token that Google states it never sends as a User-Agent arrived through the CDN 293 times. On this origin, no request could be verified by address, because CDN traffic is logged with the CDN's address. A declared-only report from this server would have shown 46,155 crawler requests. The four counts show that none of them was verified. This is one server, and the result is not a statement about the web in general. The analysis is published as a preprint on Qeios, DOI 10.32388/SRD4TB [QEIOS]. 7. Security Considerations Verification results can themselves be wrong. Published ranges change. Shared cloud addresses may host other clients. Reverse DNS needs forward confirmation. A report that names its checks lets readers judge these limits. Publishing per-token counts may help an attacker learn which tokens pass a site's filters. Operators MAY delay or aggregate reports. 8. Privacy Considerations The report carries counts only. It MUST NOT carry client IP addresses, full User-Agent strings of individual requests, request paths with query strings, or visitor identifiers. Small counts for rare tokens can still point to individuals or organisations. Operators SHOULD suppress or round small cells. 9. IANA Considerations This document has no IANA actions. 10. References 10.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC3339] Klyne, G. and C. Newman, "Date and Time on the Internet: Timestamps", RFC 3339, DOI 10.17487/RFC3339, July 2002, . Arsentev Expires 19 May 2027 [Page 6] Internet-Draft Crawler Traffic Verification Levels November 2026 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data Interchange Format", STD 90, RFC 8259, DOI 10.17487/RFC8259, December 2017, . [RFC9110] Fielding, R., Ed., Nottingham, M., Ed., and J. Reschke, Ed., "HTTP Semantics", STD 97, RFC 9110, DOI 10.17487/RFC9110, June 2022, . 10.2. Informative References [QEIOS] Arsentev, E., "Report AI Crawler Traffic in Two Columns, Declared and Verified", Qeios, preprint, DOI 10.32388/SRD4TB, November 2026, . [RFC9309] Koster, M., Illyes, G., Zeller, H., and L. Sassman, "Robots Exclusion Protocol", RFC 9309, DOI 10.17487/RFC9309, September 2022, . [RFC9421] Backman, A., Ed., Richer, J., Ed., and M. Sporny, "HTTP Message Signatures", RFC 9421, DOI 10.17487/RFC9421, February 2024, . Appendix A. Example Report The numbers in this example are illustrative and are not measurements. Arsentev Expires 19 May 2027 [Page 7] Internet-Draft Crawler Traffic Verification Levels November 2026 { "vantage_point": "origin", "period": {"start": "2026-11-01T00:00:00Z", "end": "2026-11-15T00:00:00Z"}, "token_list": {"source": "example-token-list", "as_of": "2026-11-01"}, "tokens": [ {"token": "ExampleBot", "declared": 1200, "route_consistent": 800, "verified": 0, "unverifiable": 800, "failed": 400, "checks": ["address-range"], "ranges_fetched": "2026-11-01"}, {"token": "OtherBot", "declared": 300, "route_consistent": 300, "verified": null, "unverifiable": 300, "failed": 0, "checks": ["none"]} ] } In the first entry, 800 requests arrived through the CDN and are logged with the CDN's address, so they cannot be checked at the origin; 400 arrived directly and none of them came from the published ranges. In the second entry the operator publishes no ranges, so every request is unverifiable. Author's Address Evgenii Arsentev Independent Email: hello@arsentev.ai URI: https://arsentev.ai/ Arsentev Expires 19 May 2027 [Page 8]