A JSON-Based Format for Publishing IP Ranges of Automated HTTP Clients

2.1. Top-Level Object Definition

An IP range publication file MUST be a single JSON object. The text encoding of the file MUST be UTF-8. This top-level object serves as the root container for essential metadata and the list of IP prefixes.¶

The top-level object contains the fields defined in Table 1.¶

2.2. The prefixes Array

The prefixes member of the top-level object MUST contain an array of JSON objects. Each of these objects is a Prefix Object, as defined in Section 2.3.¶

To simplify implementation for consumers, there MUST be a single, unified array for both IPv4 and IPv6 prefixes.¶

2.3. The Prefix Object

Each object within the prefixes array represents a single IP address range and its associated metadata.¶

A Prefix Object MUST contain exactly one of either the ipv4Prefix field or the ipv6Prefix field. An object containing both or neither of these fields is invalid and MUST be ignored by consumers. The fields are defined in Table 2.¶

3.1. Fetching and Caching

Consumers SHOULD fetch the IP range publication file from a stable URL provided by the publisher in a machine readable format. The file location MUST be disclosed by the publisher of the file in the page that describes the crawler.¶

Publishers SHOULD update the file when there's any change to the prefixes or every 24 hours, even if the only update is to creationTime. Consumers SHOULD implement a polling mechanism to check for updates at a reasonable interval, such as once every 24 hours. Consumers MUST NOT poll more frequently than once per hour unless explicitly permitted by the publisher's documentation or HTTP caching headers.¶

To minimize server load for the publisher and reduce unnecessary bandwidth usage for the consumer, consumers MUST respect standard HTTP caching headers that may be present in the response, such as Cache-Control, ETag, and Last-Modified. Publishers SHOULD provide these headers to facilitate efficient caching.¶

3.2. Handling Format Versioning

To ensure long-term stability and allow for future evolution of this specification, consumers MUST inspect the formatVersion field in the top-level object of the file. The version is expressed as "major.minor".¶

• Major Version Changes: A change in the major version number (e.g., from "1.0" to "2.0") indicates a non-backwards compatible change to the specification. If a consumer encounters a file with a major version number greater than the major version it is programmed to handle, the consumer MUST NOT attempt to parse the file. It SHOULD treat this situation as an error and MAY continue to use its last known valid list until it can be updated to support the new version. This prevents the misinterpretation of data from a significantly altered schema.¶

• Minor Version Changes: A change in the minor version number (e.g., from "1.0" to "1.1") indicates the addition of new features or fields that are backward-compatible. For example, a new OPTIONAL field might be added to the Prefix Object. A consumer programmed to handle version "1.0" MUST be able to correctly parse a file with version "1.1". The minor version number increases numerically independently of the major version number, for instance: 1.9 -> 1.10 -> 1.11. The consumer MUST ignore any unrecognized fields or properties within the JSON objects.¶

3.3. Prefix Aggregation and Specificity

A publication file MAY contain overlapping IP address ranges. For instance, a publisher might list a broad range like 198.51.100.0/22 with a generic service tag, and also list a more specific range within it, such as 198.51.100.0/24, with a more specific service tag.¶

When a consumer evaluates a specific IP address against the list, it MAY match multiple Prefix Objects. In such cases, the consumer's logic SHOULD use the data from the Prefix Object with the most specific CIDR range (i.e., the one with the largest prefix length) for that IP address.¶

4.1. Example 1: Basic Publication

This example demonstrates a minimal, valid file for a publisher with a single type of automated client. It uses only the required fields, making it simple to generate and consume.¶

{ "formatVersion": "1.0", "creationTime": "2025-08-15T14:30:00Z", "prefixes": [ { "ipv4Prefix": "66.249.64.0/20" }, { "ipv4Prefix": "34.64.0.0/12" }, { "ipv6Prefix": "2001:4860:4000::/36" } ] } ¶

4.2. Example 2: Advanced Publication with Multiple Services

This example illustrates a more complex file from a large cloud provider that uses optional fields to provide richer context. It differentiates between IP ranges used for a general web crawler, a specialized ads crawler, and a user-triggered fetching service. This allows consumers to implement more granular access policies.¶

{ "formatVersion": "1.0", "creationTime": "2025-09-01T10:00:00Z", "notes": "IP ranges for ExampleCloud services. For verification, FCrDNS is also recommended.", "Prefixes": "FILL OUT WITH EXAMPLE" } ¶

4.3. Example 3: Aggregated Publication by a Third Party

This example shows how a third-party security provider or an open-source project could aggregate IP lists from multiple publishers into a single file. The service field is used to attribute each prefix to its original source, allowing consumers to apply policies based on the original publisher. This reflects the use case of services that provide curated lists of known bots.¶

{ "formatVersion": "1.0", "creationTime": "2025-09-02T00:00:00Z", "notes": "Aggregated list of known good bots. Data is updated daily from original publisher sources.", "Prefixes": FILL OUT WITH EXAMPLE } ¶