One Institution, Several Policies? A Webometrics Study of AI-Crawler Directive Consistency Across University Web Properties

Executive Summary

Anthropic, OpenAI, and Google each document specific, individually named web crawlers that a website can address in its robots.txt file, and a May 2026 IETF/IAB report (RFC 9969) observes that robots.txt preferences are set independently by each site's own administrator and are not automatically carried over when content is copied or hosted on a different site. Universities commonly operate more than one public web property under the same institutional identity: a main institutional domain and at least one separate property, such as an institutional repository, that runs on different software, often a different subdomain, and in practice a separate robots.txt file. Webometrics examined the robots.txt files of the main domain and one additional web property (an institutional repository in most cases) for 18 research universities across five world regions, collected on 24 September 2026, to observe whether the same institution's AI-crawler-specific directives are consistent across its own separate properties. Among 9 institutions where both properties returned a readable robots.txt file, 3 (33.3%) named different AI-crawler tokens, or named tokens at one property but not the other. A tenth institution, ETH Zurich, showed an even sharper version of the same pattern: no robots.txt file could be retrieved at its main domain, while its research repository explicitly disallowed OpenAI's GPTBot. These findings are reported as observations from a single collection date and a purposive, non-random sample; they describe what was found, not why any individual institution's configuration takes the form it does.

Key Findings

1. Of 18 sampled institutions, a readable robots.txt file was obtained at both the main domain and a secondary web property for 9 institutions, allowing a direct same-day comparison.
2. Among those 9 paired institutions, 6 (66.7%) had no AI-crawler-specific directive at either property, and 3 (33.3%) — the University of Tokyo, the Universidad Nacional Autónoma de México (UNAM), and the University of Cape Town — showed a difference in AI-crawler-specific configuration between the two properties.
3. At the University of Tokyo, the main domain's robots.txt named no AI-crawler tokens, while its institutional repository (repository.dl.itc.u-tokyo.ac.jp) named GPTBot and ChatGPT-User with a 30-second crawl-delay rule, not a full block.
4. At UNAM, the main domain named no AI-crawler tokens, while its institutional repository (ru.dgb.unam.mx) fully disallowed GPTBot, ChatGPT-User, and a token labeled “ChatGPT Agent.”
5. At the University of Cape Town, the main domain's robots.txt named GPTBot (fully disallowed), OAI-SearchBot (subject to the same path exclusions applied to conventional search engines), and Bytespider (fully disallowed), while its repository (open.uct.ac.za) named only ClaudeBot (fully disallowed) — the two properties shared no common named crawler.
6. ETH Zurich returned no robots.txt file at all at its main domain (www.ethz.ch), while its research repository (research-collection.ethz.ch) contained an explicit “Disallow: /” rule for GPTBot.
7. Across all properties examined in this study, Anthropic's ClaudeBot appeared exactly once, at a secondary property (University of Cape Town's repository) and never at a main institutional domain.

Why This Matters for Universities

A university's research output is rarely confined to a single website. Institutional repositories, thesis archives, journal-hosting platforms, and library catalogs typically run as separate systems from the main institutional domain, often on different software with their own independently administered robots.txt file. If an institution has taken a position on how AI-training or AI-search crawlers should treat its content, that position is only as consistent as each property's own, separately maintained configuration. This study asks a narrow, checkable question: for institutions that operate both a main domain and at least one other public web property, does the AI-crawler-specific configuration actually match across the two.

Authoritative Background

In May 2026, the Internet Architecture Board (IAB) published RFC 9969, an informational report from an IAB workshop on mechanisms for expressing content-usage preferences to AI systems (the “AI-CONTROL” workshop). The report states that “the emerging practice of using the Robots Exclusion Protocol...has not been coordinated between AI crawlers, resulting in considerable differences in how they treat it.” Discussing robots.txt specifically, the report identifies three structural limitations: it “does not scale to offer granular control over large sites where authors might want to express different policies for a range of content”; it “is also typically under the control of the site administrator,” so that if a site hosts content from multiple creators, “the administrator may not allow them to express their preferences fully, or at all”; and “if content is copied or moved to a different site, the preferences at the new site need to be explicitly transferred because robots.txt is a separate resource.” The report concludes that “robots.txt cannot be the only solution to opt-out.”

RFC 9969 does not itself study universities, and it does not claim that any specific organization's preferences are inconsistent across its own properties; its scope is the general architecture of preference-expression mechanisms, and it explicitly identifies enforcement, registries, and crawler authentication as separate, out-of-scope questions. What the report does establish, as a documented fact about how robots.txt works, is that each site's rules are set independently by that site's own administrator and do not automatically travel with content that appears on more than one site. A university that operates a main domain and a separately administered repository is, in effect, a real-world instance of exactly the structural condition RFC 9969 describes: two robots.txt files, potentially two different administrators or teams, and no built-in mechanism to keep them in step. Whether that structural condition actually produces observable inconsistency at real institutions is an empirical question RFC 9969 does not answer, and this study, distinct from the RFC's own scope, is Webometrics' attempt to test it directly.

Separately, Anthropic's, OpenAI's, and Google's own crawler documentation, reviewed in Webometrics' 18 September 2026 study of the same 18-institution sample, each names specific, independently addressable crawlers: Anthropic's ClaudeBot, Claude-User, and Claude-SearchBot; OpenAI's GPTBot, OAI-SearchBot, and ChatGPT-User; and Google's Google-Extended. Each provider's documentation states that a site can address these tokens separately from one another and from conventional search-indexing crawlers.

Research Question

Among a sample of research universities that operate both a main institutional domain and at least one other public web property, are the AI-crawler-specific directives found in each property's robots.txt file consistent with one another, or do they diverge?

Methodology

Collection date: 24 September 2026, a single point-in-time snapshot.

Sample: the same 18 research universities examined in Webometrics' 18 September 2026 robots.txt/llms.txt study, reused here for direct comparability: Harvard University, Stanford University, the Massachusetts Institute of Technology, and the University of Toronto (North America); the University of Oxford, the Technical University of Munich, the University of Helsinki, Sorbonne Université, and ETH Zurich (Europe); the University of Tokyo, the National University of Singapore, the University of Melbourne, and the Australian National University (Asia-Pacific); the Universidad Nacional Autónoma de México and the Universidade de São Paulo (Latin America); and the University of Cape Town, the University of Nairobi, and Cairo University (Africa). This is a purposive convenience sample of prominent research universities, not a random or Webometrics-rank-stratified sample, and findings should not be generalized beyond the sample.

Secondary property selection: for each institution, one additional public web property was identified, in nearly all cases the institution's own DSpace, WEKO, or comparable institutional repository (for example, dash.harvard.edu for Harvard, ora.ox.ac.uk for Oxford, or research-collection.ethz.ch for ETH Zurich). No secondary property could be confidently identified for Cairo University within the scope of this study, and it is excluded from the secondary-property results.

Variables: presence and readability of robots.txt at each of the two properties; presence and Allow/Disallow/Crawl-delay status of 11 named AI-crawler tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, Google-Extended, PerplexityBot, CCBot, Bytespider), the same token list used in Webometrics' 18 September 2026 study; presence of a Content-signal line; and, for paired institutions, whether the two properties' AI-crawler-specific configuration matches.

Evidence source: direct retrieval of https://<domain>/robots.txt for each institution's main domain and identified secondary property, cross-referenced against each provider's own published token list.

Analysis approach: descriptive frequency counts with stated denominators; no statistical inference; no causal interpretation.

Limitations: disclosed in full below.

Results

Table 1. Main-domain robots.txt results (18 institutions, collected 24 September 2026)

Table 2. Secondary-property robots.txt results (17 institutions with an identified property)

Table 3. Cross-property consistency among the 9 institutions with a determinate result at both properties

Analysis

Among the 9 institutions where both properties returned a readable robots.txt file, 6 (66.7%) had identical AI-crawler configuration in the narrow sense that neither property named any of the 11 tokens checked. The remaining 3 (33.3%) diverged, and each diverged in a different way. At the University of Tokyo and UNAM, the main domain was silent on AI crawlers while the repository was not, consistent with a pattern in which repository platforms, several of which run on shared open-source software such as WEKO or DSpace, arrive with AI-crawler-aware defaults or templates that an institution's main content-management system does not. At the University of Cape Town, both properties had taken a position, but on different crawlers: the main domain's rules cover OpenAI's and ByteDance's crawlers without mentioning Anthropic, while the repository's rules cover Anthropic's ClaudeBot without mentioning OpenAI's or ByteDance's crawlers. Read together, a crawler operator relying solely on either property's robots.txt file would see an incomplete picture of the institution's overall position.

ETH Zurich is reported separately from the 9-institution comparison because its main domain returned no robots.txt file at all, meaning there is no main-domain configuration to compare against a repository configuration; there is, by construction, no way for the two to be “consistent,” since one does not exist. This is nonetheless a directly relevant observation for the same research question: an outside crawler operator checking only the main domain would find no AI-crawler-specific signal of any kind, while one checking the repository would find an explicit rule disallowing GPTBot.

This study's design, a single-snapshot inspection of two robots.txt files per institution, cannot determine why any given pair is consistent or not. It cannot distinguish, for example, between a deliberate institutional decision to configure only the repository, an oversight at the main domain, or simply two teams that have never coordinated. Distinguishing between these explanations would require evidence this study did not collect, such as internal governance documentation or interviews with the relevant IT teams.

University Examples

The University of Tokyo's main domain robots.txt (www.u-tokyo.ac.jp) disallows only a single internal path (/gakunai/) and contains a commented-out sitemap reference; it names no AI-crawler tokens. Its institutional repository (repository.dl.itc.u-tokyo.ac.jp) separately names Bingbot, PetalBot, ChatGPT-User, and GPTBot, applying a 30-second crawl-delay to the latter two rather than disallowing them outright — a distinction, permitting reduced-rate access rather than blocking it, that neither the main domain's file nor most other institutions in this sample make for any crawler.

UNAM's repository (ru.dgb.unam.mx) fully disallows GPTBot, ChatGPT-User, and a third token labeled “ChatGPT Agent” in a block explicitly commented “# Extra Dissalow,” appended to an otherwise standard DSpace configuration; UNAM's main domain (www.unam.mx) contains no equivalent rule for any of the three.

The University of Cape Town's main domain (www.uct.ac.za) disallows GPTBot and Bytespider outright and applies standard search-style path exclusions to OAI-SearchBot, the same treatment given to conventional search engines; its repository (open.uct.ac.za) instead disallows ClaudeBot and does not mention GPTBot, OAI-SearchBot, or Bytespider at all.

Implications for Higher Education

University web and IT teams that have already configured AI-crawler-specific rules at one property, most often the main domain, should not assume the same configuration is present at a research repository, library catalog, or other separately administered property; this study found that when the two properties diverged, the divergence ran in both directions, adding rules at the repository that were absent at the main domain, and vice versa in terms of which specific crawler was named. Research offices and library IT teams that manage a repository independently of the central web team are, based on this sample, a plausible point where AI-crawler configuration can end up out of step with the main site's, whether by design or by omission. Because RFC 9969 documents that robots.txt preferences are not automatically transferred between separately administered sites, an institution that wants a single, coherent AI-crawler position across all of its public web properties needs to configure that position at each property individually; no current mechanism does this for an institution automatically.

Recommended Actions

• Inventory every publicly reachable web property under the institution's own domain umbrella, including repositories, journal-hosting platforms, and library catalogs, and check each one's robots.txt file against the same list of provider-documented AI-crawler tokens.
• Where a repository or other secondary property is administered separately from the main website, establish a shared checklist or communication point so that a change in AI-crawler policy at one property is considered at the others.
• Where the goal is a fully attributable position, be aware that a full Disallow rule and a Crawl-delay rule serve different purposes, one blocking a crawler outright and the other permitting reduced-rate access, and confirm which behavior is actually intended at each property.
• Treat robots.txt as one layer among several, alongside content-delivery-network or firewall-level bot management, since RFC 9969 explicitly excludes enforcement mechanisms from what robots.txt-style preference declarations can achieve on their own.

These actions are not presented as guaranteeing any specific AI-training or AI-search outcome. No source cited in this study makes such a claim, and this study did not test for one.

Limitations

Two of the 18 sampled institutions' main domains returned a certificate-verification error rather than a determinate result (University of Nairobi, Cairo University), and a secondary property could not be confidently identified for Cairo University at all; both are excluded from the quantitative comparison rather than assumed to lack AI-crawler configuration. Seven of the 17 attempted secondary properties returned an inconclusive result (a 403 or 405 error, a bot-protection challenge page, or an access-denied response) rather than a readable robots.txt file, and are likewise excluded from the paired comparison in Table 3. The sample is the same purposive, non-random, five-region convenience sample used in Webometrics' 18 September 2026 study and cannot be generalized to the global university population. Exactly one secondary property was checked per institution; an institution may operate additional public web properties, such as a separate journal platform or library catalog, that were not examined and could show a different pattern again. This study reflects a single snapshot collected 24 September 2026; robots.txt content can change at any time. This study did not verify whether the crawlers named in any file actually honor the directive observed, only what the directive states, consistent with each provider's own documentation that robots.txt compliance is voluntary. Where a main-domain hostname differs slightly from the exact hostname checked in Webometrics' 18 September 2026 study (for example, www5.usp.br here versus the apex domain previously checked for the Universidade de São Paulo), the two collection dates' main-domain results are not treated as a direct re-test of the identical URL.

Conclusion

Among 9 sampled research universities where both a main domain and a secondary web property returned a readable robots.txt file on 24 September 2026, 3 (33.3%) showed a difference in AI-crawler-specific configuration between the two properties, and a tenth institution, ETH Zurich, showed no robots.txt file at all at its main domain alongside an explicit AI-crawler disallow rule at its repository. RFC 9969 documents, as a general property of the robots.txt mechanism, that preferences are set independently at each site and are not automatically carried between separately administered properties; this study's observations are consistent with that documented structural feature producing exactly the kind of property-level inconsistency it describes, at real institutions, in a sample examined for this purpose. These findings describe what was observed on a single date in a non-random sample and do not establish why any individual institution's configuration takes the form it does.

Sources

• Internet Architecture Board, “Report from the IAB Workshop on AI-CONTROL,” RFC 9969, May 2026 — rfc-editor.org/rfc/rfc9969
• Anthropic, “Does Anthropic crawl data from the web, and how can site owners block the crawler?”, Claude Help Center, last updated 7 April 2026 — support.claude.com/en/articles/8896518
• OpenAI, “Overview of OpenAI Crawlers” — developers.openai.com/api/docs/bots
• Google, “Things to Know about Google's Web Crawling,” Google for Developers, last updated 3 March 2026 — developers.google.com/crawling/docs/about-crawling
• Webometrics' companion study on AI crawler access at universities (18 September 2026), source of the 18-institution sample and token list
• Webometrics' companion study on Cloudflare AI-crawler defaults at universities (20 September 2026)
• Direct retrieval of the main-domain and secondary-property robots.txt files for the 18 sampled universities, collected 24 September 2026.

Suggested Citation

Webometrics by UNIRANKS (2026). One Institution, Several Policies? A Webometrics Study of AI-Crawler Directive Consistency Across University Web Properties. Webometrics.org, 24 September 2026.

Frequently Asked Questions

Does RFC 9969 say universities have inconsistent AI-crawler policies?

No. RFC 9969 is a general report on the architecture of AI-usage-preference mechanisms; it does not study universities or any other specific organization. It documents that robots.txt preferences are set independently by each site's administrator and are not automatically transferred when content appears on more than one site. This study uses that documented structural feature as the basis for an original, university-specific test, and the resulting findings are Webometrics' own observations, not a claim made by the RFC itself.

Is a Crawl-delay rule the same as blocking a crawler?

No. A Crawl-delay rule asks a crawler to wait a specified number of seconds between requests; it does not prevent the crawler from accessing the site. A Disallow rule blocks access to the specified path entirely. This study reports the two separately, and the University of Tokyo's repository is an example that uses Crawl-delay rather than Disallow for the AI-crawler tokens it names.

Did every institution in the sample show inconsistent AI-crawler policies?

No. Among the 9 institutions where both properties could be read, 6 showed no AI-crawler-specific configuration at either property, which this study counts as consistent. Only 3 of the 9 showed a difference, plus ETH Zurich's separate case of a missing main-domain file alongside a repository-level rule.