<?xml version="1.0" encoding="utf-8"?>
<rfc xmlns:xi="http://www.w3.org/2001/XInclude"
     ipr="trust200902"
     docName="draft-arsentev-crawler-traffic-reporting-00"
     category="info"
     submissionType="independent"
     tocInclude="true"
     tocDepth="3"
     sortRefs="true"
     symRefs="true"
     version="3">
  <front>
    <title abbrev="Crawler Traffic Verification Levels">Reporting Crawler Traffic by Verification Level</title>
    <seriesInfo name="Internet-Draft" value="draft-arsentev-crawler-traffic-reporting-00" stream="independent"/>
    <author fullname="Evgenii Arsentev" initials="E." surname="Arsentev">
      <organization>Independent</organization>
      <address>
        <email>hello@arsentev.ai</email>
        <uri>https://arsentev.ai/</uri>
      </address>
    </author>
    <date/>
    <keyword>crawler</keyword>
    <keyword>bot</keyword>
    <keyword>User-Agent</keyword>
    <keyword>measurement</keyword>
    <keyword>AI crawler</keyword>
    <abstract>
<t>The User-Agent header is written by the client. It is a claim, not an identity. Reports of crawler and AI crawler traffic that count requests by User-Agent token alone present claims as activity. This document defines four counts a crawler traffic report should give per crawler token: declared, route-consistent, verified, and unverifiable. It states which verification check was possible at the vantage point where the log was taken. It also defines a small JSON report object.</t>
    </abstract>
  </front>
  <middle>
<section anchor="intro"><name>Introduction</name>
<t>Site operators, CDNs, analytics vendors and researchers publish counts of "AI crawler traffic". They produce these counts by matching the User-Agent string (<xref target="RFC9110" section="10.1.5"/>) against lists of crawler tokens. The User-Agent value is chosen by the client.</t>
<t>Crawler operators increasingly publish IP address ranges for their crawlers, and some offer reverse DNS checks. Work on cryptographic bot authentication with HTTP Message Signatures (<xref target="RFC9421"/>) is under way in the IETF (webbotauth).</t>
<t>Whether a check can be applied depends on where the log is recorded. An origin server behind a CDN usually sees the CDN's address, not the client's address. A count that mixes checked and uncheckable requests cannot be compared across sites.</t>
<t>The goal of this document is to make the verification level of every reported number explicit.</t>
<t>This document does not define how to verify a crawler. It does not define access policy, which is the subject of robots.txt (<xref target="RFC9309"/>) and of AI preference work. It does not say whether unverified traffic should be blocked.</t>
</section>
<section anchor="conventions-and-terminology"><name>Conventions and Terminology</name>
<t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 <xref target="RFC2119"/> <xref target="RFC8174"/> when, and only when, they appear in all capitals, as shown here.</t>
<dl newline="false" spacing="normal"><dt>crawler token:</dt><dd>a product token a crawler operator documents for its crawler's User-Agent.</dd><dt>vantage point:</dt><dd>the component where the log record is produced (origin server, reverse proxy, CDN edge).</dd><dt>expected route:</dt><dd>the network path a legitimate request to the site takes given the site's deployment, for example through the site's CDN.</dd><dt>published address range:</dt><dd>a set of IP prefixes a crawler operator publishes as the source of its crawler.</dd></dl>
</section>
<section anchor="the-four-counts"><name>The Four Counts</name>
<t>For each crawler token and reporting period, the report gives the following counts.</t>
<dl newline="false" spacing="normal"><dt>declared:</dt><dd>requests whose User-Agent contains the token. This count is always available.</dd><dt>route-consistent:</dt><dd>declared requests that arrived by the expected route. A declared request that reached the origin directly, while the site is served through a CDN, is not route-consistent.</dd><dt>verified:</dt><dd>declared requests whose source was confirmed by a check the report names. The check is a published address range match, forward-confirmed reverse DNS, or a verified HTTP message signature.</dd><dt>unverifiable:</dt><dd>declared requests for which no check was possible at the vantage point, for example because the logged address is the CDN's address.</dd></dl>
<t>The report MUST give the declared count together with at least one of the other three counts. A report MUST NOT present the declared count alone as crawler activity.</t>
<t>Requests that were checked and failed are the declared count minus the verified count minus the unverifiable count. The report SHOULD state that number.</t>
<t>When the operator of a token publishes no ranges and no other check exists, all requests under that token are unverifiable. The report says so instead of reporting zero verified.</t>
</section>
<section anchor="stating-the-vantage-point-and-the-checks"><name>Stating the Vantage Point and the Checks</name>
<t>The report names the vantage point, the period, and the token list with its version or date. For each token, the report states which checks were available: address range, reverse DNS, message signature, or none.</t>
<t>A count from a CDN edge and a count from an origin are different measurements. They are not added together.</t>
<t>If address ranges were fetched, the report gives the date they were fetched, because ranges change.</t>
</section>
<section anchor="report-object"><name>Report Object</name>
<t>The report is carried as a JSON object (<xref target="RFC8259"/>). The object carries one entry per crawler token with the four counts, the checks performed, and the vantage point. The example in the appendix uses illustrative numbers, not measurements.</t>
<t>The report is a JSON object <xref target="RFC8259"/> with the following members.</t>
<dl newline="true" spacing="normal">
<dt>vantage_point</dt><dd>String, REQUIRED. One of "origin", "reverse-proxy", "cdn-edge", "other".</dd>
<dt>period</dt><dd>Object, REQUIRED, with members "start" and "end", each a date-time as defined in <xref target="RFC3339"/>.</dd>
<dt>token_list</dt><dd>Object, REQUIRED, with members "source" (string naming the list of crawler tokens used) and "as_of" (full-date as defined in <xref target="RFC3339"/>).</dd>
<dt>tokens</dt><dd>Array, REQUIRED, one object per crawler token, with the members below.</dd>
<dt>tokens[].token</dt><dd>String, REQUIRED. The crawler token as matched.</dd>
<dt>tokens[].declared</dt><dd>Non-negative integer, REQUIRED.</dd>
<dt>tokens[].route_consistent</dt><dd>Non-negative integer, or null when the expected route could not be determined at the vantage point.</dd>
<dt>tokens[].verified</dt><dd>Non-negative integer, or null when no check was available.</dd>
<dt>tokens[].unverifiable</dt><dd>Non-negative integer, REQUIRED.</dd>
<dt>tokens[].failed</dt><dd>Non-negative integer, RECOMMENDED. Equal to declared minus verified minus unverifiable.</dd>
<dt>tokens[].checks</dt><dd>Array of strings, REQUIRED. Each one of "address-range", "reverse-dns", "http-signature", "none".</dd>
<dt>tokens[].ranges_fetched</dt><dd>full-date as defined in <xref target="RFC3339"/>. REQUIRED when "checks" contains "address-range".</dd>
</dl>
<t>A report MUST satisfy, for every entry, verified + unverifiable + failed = declared when all three are present. Members not defined here MAY be added; consumers MUST ignore members they do not understand.</t>

</section>
<section anchor="operational-experience"><name>Operational Experience</name>
<t>This section reports one case. The data come from one origin server behind a CDN, with full logs for 15 days. The server logged 345,808 requests in total. Of these, 46,155 carried one of 22 published crawler tokens.</t>
<t>Of those declared requests, 35.7% reached the origin directly, bypassing the CDN. None of the 11,276 direct requests under tokens whose operators publish address ranges came from those ranges. In other words, these requests were not confirmed by the published ranges.</t>
<t>The direct traffic never requested robots.txt or the sitemap. It used about 3.5 times more distinct User-Agent strings per request.</t>
<t>A token that Google states it never sends as a User-Agent arrived through the CDN 293 times.</t>
<t>On this origin, no request could be verified by address, because CDN traffic is logged with the CDN's address. A declared-only report from this server would have shown 46,155 crawler requests. The four counts show that none of them was verified.</t>
<t>This is one server, and the result is not a statement about the web in general. The analysis is published as a preprint on Qeios, DOI 10.32388/SRD4TB <xref target="QEIOS"/>.</t>
</section>
<section anchor="security"><name>Security Considerations</name>
<t>Verification results can themselves be wrong. Published ranges change. Shared cloud addresses may host other clients. Reverse DNS needs forward confirmation. A report that names its checks lets readers judge these limits.</t>
<t>Publishing per-token counts may help an attacker learn which tokens pass a site's filters. Operators MAY delay or aggregate reports.</t>
</section>
<section anchor="privacy"><name>Privacy Considerations</name>
<t>The report carries counts only. It MUST NOT carry client IP addresses, full User-Agent strings of individual requests, request paths with query strings, or visitor identifiers.</t>
<t>Small counts for rare tokens can still point to individuals or organisations. Operators SHOULD suppress or round small cells.</t>
</section>
<section anchor="iana"><name>IANA Considerations</name>
<t>This document has no IANA actions.</t>
</section>
  </middle>
  <back>
    <references>
      <name>References</name>
      <references>
        <name>Normative References</name>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.2119.xml"/><xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.3339.xml"/><xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.8174.xml"/><xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.8259.xml"/><xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.9110.xml"/>
      </references>
      <references>
        <name>Informative References</name>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.9309.xml"/><xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.9421.xml"/>
        <reference anchor="QEIOS" target="https://doi.org/10.32388/SRD4TB">
          <front>
            <title>Report AI Crawler Traffic in Two Columns, Declared and Verified</title>
            <author fullname="Evgenii Arsentev" initials="E." surname="Arsentev"/>
            <date year="2026" month="November"/>
          </front>
          <seriesInfo name="DOI" value="10.32388/SRD4TB"/>
          <refcontent>Qeios, preprint</refcontent>
        </reference>
      </references>
    </references>
<section anchor="example"><name>Example Report</name>
<t>The numbers in this example are illustrative and are not measurements.</t>
<sourcecode type="json"><![CDATA[
{
  "vantage_point": "origin",
  "period": {"start": "2026-11-01T00:00:00Z",
             "end":   "2026-11-15T00:00:00Z"},
  "token_list": {"source": "example-token-list",
                 "as_of": "2026-11-01"},
  "tokens": [
    {"token": "ExampleBot",
     "declared": 1200, "route_consistent": 800,
     "verified": 0, "unverifiable": 800, "failed": 400,
     "checks": ["address-range"], "ranges_fetched": "2026-11-01"},
    {"token": "OtherBot",
     "declared": 300, "route_consistent": 300,
     "verified": null, "unverifiable": 300, "failed": 0,
     "checks": ["none"]}
  ]
}
]]></sourcecode>
<t>In the first entry, 800 requests arrived through the CDN and are logged with the CDN's address, so they cannot be checked at the origin; 400 arrived directly and none of them came from the published ranges. In the second entry the operator publishes no ranges, so every request is unverifiable.</t>
</section>

  </back>
</rfc>
