Cloudflare's AI-Crawler Defaults and University Websites: A Webometrics DNS Study

On 15 September 2026, Cloudflare introduced a new edge-level classification for web crawlers — Search, Training, and Agent — along with an “Accountable” designation for crawler operators and a “Disallow AI Training” toggle that lets a website remain indexable in search while opting out of AI-model training use. This configuration operates at the content-delivery-network (CDN) layer, separately from a website’s own robots.txt file, which Webometrics examined across 18 research universities in a companion study published 18 September 2026. That earlier study found that none of the sampled universities’ robots.txt files named an Anthropic or Google AI-training-specific crawler token. It did not, and by its design could not, determine which of those universities’ websites are actually routed through Cloudflare’s network, where these new defaults apply. This study addresses that gap directly. Using the same 18-institution sample, Webometrics resolved each university’s standard public website hostname via DNS on 20 September 2026 and checked the resulting IP address against Cloudflare’s own published IPv4 ranges. Two of 18 universities (11.1%) resolved to an address inside Cloudflare’s published range; 16 (88.9%) did not. For both Cloudflare-routed institutions, the bare domain and the “www” subdomain pointed to different infrastructure entirely, a finding that has its own implications for how CDN-level studies should be designed.

Key Findings

1. 2 of 18 sampled universities (11.1%) — the University of Oxford and the Universidad Nacional Autónoma de México (UNAM) — had a standard public website hostname that resolved, on 20 September 2026, to an IPv4 address within Cloudflare’s published IP ranges.
2. 16 of 18 (88.9%) resolved to addresses outside Cloudflare’s published ranges at the hostname tested, indicating that, at the DNS layer observed, their public websites were not being proxied through Cloudflare’s edge network at collection time.
3. For both Cloudflare-routed institutions, the bare apex domain (e.g., ox.ac.uk) and the “www” subdomain (www.ox.ac.uk) resolved to different IP addresses on different networks; only the “www” hostname fell inside Cloudflare’s range.
4. Neither Oxford’s nor UNAM’s robots.txt file, read two days earlier for Webometrics’ 18 September 2026 companion study, contained a Content-signal line or any of the 11 AI-crawler tokens tracked in that study.
5. Cloudflare’s own 15 September 2026 announcement states that Apple, Google, and Microsoft met its “Accountable” operator criteria immediately, alongside Amazon, Anthropic, Meta, and OpenAI, on the basis that each operates functionally separate Search and Training crawlers.
6. Cloudflare states that 17% of sites on its network currently enable some mechanism to block AI-training crawlers, compared with under 1% that block conventional search indexing — a gap the new Search/Training/Agent classification and “Disallow AI Training” toggle are designed to make easier to act on without one affecting the other.

Why This Matters for Universities

A university’s robots.txt file has, until now, been the most visible mechanism through which an outside observer — including a study like Webometrics’ own 18 September 2026 research — could assess how an institution treats AI crawlers. Cloudflare’s 15 September 2026 announcement adds a second, largely invisible layer: edge-level bot classification that a CDN can apply to a domain’s traffic independently of what that domain’s own robots.txt file says, and that migrates automatically for many existing Cloudflare customers. If a university’s website sits behind Cloudflare’s network, its actual reachability to Search, Training, and Agent crawlers may now depend as much on Cloudflare account-level settings as on the institution’s own published robots.txt directives — settings that are not visible from outside the Cloudflare account itself. This matters for accuracy in any future Webometrics or third-party study that infers AI-crawler policy purely from robots.txt content, and it matters operationally for university IT and communications teams who may not be the same team that manages the institution’s CDN configuration.

Authoritative Background

Cloudflare’s official announcement, published 15 September 2026 on the Cloudflare Blog, introduces three behavioral classifications for crawler traffic — Search (crawling that builds search indexes), Training (crawling used to train or fine-tune AI models), and Agent (a crawler acting on behalf of a specific user, in response to that user’s own request) — replacing an earlier, undifferentiated “block AI bots” setting. Alongside this classification, Cloudflare defines an “Accountable” designation for crawler operators that meet four stated criteria: providing a robots.txt-or-equivalent mechanism for AI-training opt-out; providing a separate mechanism to opt out of AI summaries (at the operator level immediately, with a Cloudflare-wide mechanism planned for early 2027); giving site owners URL-level visibility into which pages were made available for training, alongside search-appearance metrics; and assuring site owners that an AI-training opt-out will not affect the same domain’s ordinary search inclusion or ranking. Apple, Google, and Microsoft met these criteria immediately; Cloudflare states that Amazon, Anthropic, Meta, and OpenAI also qualified, because each operates separately identifiable Search and Training crawlers. A new “Disallow AI Training” toggle publishes a no-training preference while continuing to allow Accountable mixed-use crawlers, such as Googlebot, Bingbot, and Applebot, to index content for search. Cloudflare states that existing domains with a legacy “block AI bots” setting migrate automatically to this new control, and that new domains onboarded to Cloudflare receive a default preset based on whether the site carries advertising. Webometrics reviewed this announcement directly at blog.cloudflare.com rather than relying on secondary reporting.

This sits alongside, not in place of, the crawler-specific documentation Anthropic, OpenAI, and Google each publish for their own bots (ClaudeBot, Claude-User, and Claude-SearchBot; GPTBot, OAI-SearchBot, and ChatGPT-User; and Google-Extended, respectively), which was the evidence base for Webometrics’ 18 September 2026 study of institutional robots.txt files. That study is the direct predecessor to this one.

Research Question

Among the same 18 research universities examined in Webometrics’ 18 September 2026 robots.txt and llms.txt study, how many have a standard public website hostname that resolves to an IP address within Cloudflare’s own published IPv4 ranges — the precondition for Cloudflare’s edge-level Search/Training/Agent crawler classification and any account-level AI-training defaults to apply to that website’s traffic?

Methodology

Collection date: 20 September 2026, a single point-in-time DNS snapshot, two days after the companion robots.txt study’s own 18 September 2026 collection date.

Sample: the same 18 research universities used in Webometrics’ 18 September 2026 companion study (Harvard University, Stanford University, the Massachusetts Institute of Technology, and the University of Toronto in North America; the University of Oxford, the Technical University of Munich, the University of Helsinki, Sorbonne Université, and ETH Zurich in Europe; the University of Tokyo, the National University of Singapore, the University of Melbourne, and the Australian National University in Asia-Pacific; the Universidad Nacional Autónoma de México and the Universidade de São Paulo in Latin America; and the University of Cape Town, the University of Nairobi, and Cairo University in Africa). Reusing the identical sample allows this study’s DNS-layer observations to be read alongside that study’s robots.txt-layer observations for the same institutions. This remains a purposive convenience sample of prominent research universities, not a random or Webometrics-rank-stratified sample.

Hostname selection: for each institution, this study attempted standard DNS resolution of the bare institutional domain and of the “www” subdomain of that domain, and reports the result for each hostname that resolved. Where the two hostnames resolved to different IP addresses, both are reported, since (as the Results section shows) this difference is itself a finding.

Variables: the resolved IPv4 address for each hostname; whether that address falls within any of the 15 IPv4 CIDR ranges Cloudflare publishes as its current network at cloudflare.com/ips-v4, cross-checked against the equivalent list at cloudflare.com/ips/.

Evidence source: standard DNS A-record resolution performed on 20 September 2026, and Cloudflare’s own published IPv4 range list, retrieved the same day.

Analysis approach: descriptive frequency counts with stated denominators; no statistical inference; no causal interpretation. Falling within Cloudflare’s published IP range is treated only as evidence that a domain’s traffic is proxied through Cloudflare’s network at the collection date (a precondition for Cloudflare’s edge-level crawler classification to apply), not as evidence of which specific bot-management or AI-training settings that Cloudflare account has configured, which is not observable from outside the account.

Limitations: disclosed in full below.

Results

Summary: taking each institution’s “www” hostname as its standard public website address (the convention this study used to classify each institution once), 2 of 18 (11.1%) — the University of Oxford and UNAM — resolved inside Cloudflare’s published IPv4 ranges on 20 September 2026; 16 of 18 (88.9%) did not.

Analysis

The headline frequency, 2 of 18 sampled research universities resolving into Cloudflare’s published ranges at their standard public hostname, is lower than the roughly one-in-six figure (17%) Cloudflare itself cites for the share of sites on its network that already act on an AI-training opt-out. These are not directly comparable figures: Cloudflare’s 17% describes the subset of Cloudflare’s own customers that use a blocking mechanism, while this study’s 11.1% describes how many of 18 major research universities, an unrelated sample, are Cloudflare customers at all at the hostname tested. Because the sample is a small, purposive selection of prominent research universities rather than a random or representative one, this finding should not be read as an estimate of Cloudflare adoption across universities generally; it describes only the 18 institutions examined.

The more structurally significant observation is that, for both Oxford and UNAM, the bare apex domain and the “www” subdomain point to entirely different infrastructure. Oxford’s apex domain, ox.ac.uk, resolved to an address in a non-Cloudflare range, while www.ox.ac.uk resolved to an address inside Cloudflare’s published range; UNAM showed the same pattern. A study that tested only the apex domain, or only the “www” subdomain, could reach a different classification for the same institution. This is a methodological finding in its own right: any future CDN- or edge-level study of university websites needs to specify, and ideally test, more than one canonical hostname per institution, because the hostname a browser actually lands on (commonly “www”) is not always the one that first appears in institutional documentation, in DNS zone files, or in a single-hostname automated check.

Read alongside the 18 September 2026 companion study, neither Oxford’s nor UNAM’s robots.txt file contained a Content-signal line, of the kind the University of Helsinki used, or any AI-training-specific crawler directive. If Cloudflare’s stated automatic migration behavior applies to either institution’s account, an AI-training preference could now be set at the CDN layer without appearing anywhere in the domain’s own robots.txt file, which is precisely the invisible-layer scenario this study set out to examine. This study did not, and could not from outside each institution’s Cloudflare account, verify whether such a setting is actually enabled for either domain; it verified only that both domains’ public-facing hostname is currently routed through Cloudflare’s network, which is the technical precondition for that setting to have any effect.

University Examples

The University of Oxford’s central website is reachable at both ox.ac.uk and www.ox.ac.uk, but only the “www” hostname resolved, on the collection date, to an address inside Cloudflare’s published IPv4 ranges; the bare domain resolved to a separate, non-Cloudflare address. The same pattern held for UNAM’s unam.mx and www.unam.mx. Both observations are reported as DNS-layer technical facts from the collection date, not as an evaluation of either institution’s technical choices.

Implications for Higher Education

University web and IT teams that manage CDN or bot-management settings separately from the team that maintains robots.txt may want to confirm that both layers reflect the same intended policy toward AI crawlers, since this study’s evidence suggests the two can diverge without either team necessarily being aware of the other’s configuration. Institutions using Cloudflare specifically may want to review the new Search/Training/Agent classification and the “Disallow AI Training” toggle directly in their Cloudflare account, since Cloudflare states that migration for existing accounts is automatic and preset-driven rather than requiring an active choice. For research and communications offices, this study’s finding that apex and “www” hostnames can route through entirely different infrastructure is a reminder that a single canonical hostname should be used consistently in institutional documentation, sitemaps, and structured data, both for this reason and for the general search-indexing and canonicalization reasons Google’s own documentation describes.

Recommended Actions

• Confirm, within the institution’s own CDN or bot-management dashboard (Cloudflare or otherwise), what AI-crawler-related defaults are currently active, rather than assuming that an unmodified robots.txt file is a complete description of the institution’s AI-crawler posture.
• Where the institution uses Cloudflare, review the new Search/Training/Agent classification and the “Disallow AI Training” control directly, and confirm whether the automatic migration Cloudflare describes has applied the intended setting.
• Standardize on a single canonical public hostname (typically the “www” subdomain or the bare apex domain, whichever the institution treats as primary) across sitemaps, robots.txt, canonical tags, and structured data, since this study observed that apex and “www” hostnames can resolve to materially different infrastructure.

These actions are not presented as guaranteeing any change in AI-search visibility, citation frequency, or ranking outcome. No source cited in this study makes such a claim, and this study did not test for one.

Limitations

This study observed DNS resolution results and compared them against Cloudflare’s published IP ranges; it did not access any institution’s Cloudflare account, bot-management dashboard, or internal network configuration, and cannot confirm which specific AI-crawler settings, if any, are enabled for the two institutions found to resolve into Cloudflare’s range. A domain resolving inside Cloudflare’s published range indicates that its traffic is proxied through Cloudflare’s network at the collection date; it does not indicate that the domain uses Cloudflare exclusively, permanently, or for every subdomain, and DNS configurations can change at any time. This study checked only IPv4 A records against Cloudflare’s published IPv4 ranges; it did not check IPv6 (AAAA) records against Cloudflare’s separately published IPv6 ranges, and did not check subdomains beyond the bare apex and “www” host. A small number of large organizations operate dedicated, non-anycast address ranges under agreement with Cloudflare that would not appear in the standard published list; this study’s method would not detect such an arrangement, so the 11.1% figure should be read as a lower-bound estimate based on Cloudflare’s standard published ranges, not a definitive count of all Cloudflare-fronted institutions in the sample. The sample of 18 is a purposive convenience sample of prominent research universities, not random or Webometrics-rank-stratified, and its findings should not be generalized to the global university population. This study reflects a single DNS snapshot collected 20 September 2026; results could differ at another time or from another DNS resolver’s vantage point, although membership in Cloudflare’s published address ranges is a static property of the ranges themselves rather than of resolver location.

Conclusion

Of the 18 research universities examined in Webometrics’ 18 September 2026 robots.txt and llms.txt study, 2 (11.1%) — the University of Oxford and the Universidad Nacional Autónoma de México — had a standard public website hostname that resolved, on 20 September 2026, to an IP address inside Cloudflare’s published network, the precondition for Cloudflare’s new Search/Training/Agent crawler classification and any account-level AI-training defaults to apply. For both institutions, this depended on which hostname was tested: their bare apex domain resolved to different, non-Cloudflare infrastructure than their “www” subdomain. Neither institution’s robots.txt file, examined two days earlier, named an AI-training-specific directive, meaning that if an account-level AI-training preference is active for either domain, it is not currently visible anywhere in that domain’s own published robots.txt file. This study’s design can observe that a precondition is met; it cannot observe the account-level setting itself, and readers should treat this as an initial, narrowly scoped extension of Webometrics’ existing robots.txt work rather than a complete picture of university AI-crawler governance.

Sources

• Cloudflare, “Have it both ways: stay discoverable in search while disallowing AI training,” Cloudflare Blog, published 15 September 2026 — blog.cloudflare.com/accountable-mixed-use-ai-crawlers
• Cloudflare, official published IPv4 address ranges — cloudflare.com/ips-v4 and cloudflare.com/ips
• Webometrics by UNIRANKS, “AI Crawler Access at Universities: A Webometrics Study of robots.txt and llms.txt Across 18 University Domains,” 18 September 2026 (companion study; source of the 18-institution sample and the robots.txt-layer evidence discussed in the Analysis section) — read the companion study on Webometrics.org
• Direct DNS resolution of the 18 sampled university domains’ apex and “www” hostnames, collected 20 September 2026.

Suggested Citation

Webometrics by UNIRANKS (2026). Beyond robots.txt: Which University Websites Run Through Cloudflare’s New AI-Crawler Defaults? A DNS Study of 18 Research Universities. Webometrics.org, 20 September 2026.