AI Crawler Access at Universities: A robots.txt Study of 18 Domains

Executive Summary

Anthropic, OpenAI, and Google each publish official documentation naming specific web crawlers, including ClaudeBot, GPTBot, OAI-SearchBot, and Google-Extended, that a website can address individually through its robots.txt file. Webometrics examined the robots.txt and llms.txt files of 18 research universities across five world regions, collected on 18 September 2026, to observe whether and how these specific crawler tokens appear in institutional robots.txt files. Of 16 universities with a determinate robots.txt result, 13 (81.3%) had a readable file at the standard path. Of those 13, none named any of Anthropic's three documented crawlers or Google's Google-Extended token. One university, the University of Cape Town, named OpenAI's GPTBot and OAI-SearchBot separately, applying different rules to each. One university, the University of Helsinki, used an emerging, provider-neutral robots.txt extension, Cloudflare's Content Signals Policy, to express an AI-training opt-out. Only 1 of 13 universities checked for llms.txt had a well-formed file. This study reports these frequencies as direct observations from a single collection date; it does not, and by its design cannot, establish why any individual directive is present or absent.

Key Findings

1. Of 16 universities yielding a determinate robots.txt result, 13 (81.3%) had a readable robots.txt file at the standard root path; 3 (18.8%) had none (National University of Singapore, Universidade de São Paulo, ETH Zurich).
2. 0 of 13 readable robots.txt files (0%) contained a rule naming ClaudeBot, Claude-User, or Claude-SearchBot, Anthropic's three documented crawlers.
3. 0 of 13 (0%) contained a rule naming Google-Extended, the token Google documents specifically for AI-model training opt-out.
4. 1 of 13 (7.7%), the University of Cape Town, contained separate rules for GPTBot (fully disallowed) and OAI-SearchBot (allowed, subject to the same path exclusions applied to major search engines), and separately disallowed Bytespider, ByteDance, ByteDanceSpider, and Yandex.
5. 1 of 13 (7.7%), the University of Helsinki, included a Content-signal: search=yes, ai-input=yes, ai-train=no, use=reference line, a mechanism distinct from individual-bot Allow/Disallow rules.
6. 8 of 13 readable robots.txt files (61.5%) included a Sitemap: directive, compared with at most 1 of 13 (7.7%) containing any AI-crawler-specific directive.
7. Of 13 universities checked for an llms.txt file, 1 (7.7%), the University of Toronto, had a well-formed file matching the published llms.txt convention; 11 (84.6%) returned no file; 1 (7.7%) returned a response this study could not resolve to a definite outcome.

Why This Matters for Universities

A university's robots.txt file is one of a small number of machine-readable mechanisms, alongside others such as content-delivery-network or firewall-level bot management and HTTP response headers, through which an institution can address a specific web crawler by name. Anthropic, OpenAI, and Google each operate more than one crawler for more than one purpose, covering model training, live user queries, and search indexing, and each documents that a site can address these separately. Whether a given university's current robots.txt configuration reflects the outcome the institution intends is a question this study's evidence can inform but not answer on its own.

What the Authoritative Source Says

Anthropic's official crawler documentation, last updated 7 April 2026, names three bots: ClaudeBot, described as collecting web content that could potentially contribute to model training; Claude-User, which supports live Claude queries; and Claude-SearchBot, which navigates the web to improve search result quality. The documentation states that Anthropic's bots respect "do not crawl" signals by honoring industry-standard robots.txt directives, and that disallowing ClaudeBot signals that a site's future materials should be excluded from AI model training datasets.

OpenAI's official crawler documentation, which does not display a publish or last-updated date, separately names OAI-SearchBot, used to surface websites in ChatGPT's search features; GPTBot, used to crawl content that may be used in training OpenAI's generative AI foundation models; and ChatGPT-User, for which OpenAI notes that robots.txt rules may not apply because the fetch is user-triggered rather than automatic. OpenAI states that a webmaster can allow OAI-SearchBot to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training.

Google's crawling documentation, last updated 3 March 2026, states that sites can use Google-Extended in robots.txt to control whether their content helps train future Gemini models, and specifies that using Google-Extended does not affect a site's inclusion in Search and is not used as a ranking signal.

The llms.txt convention was proposed by Jeremy Howard, first published 3 September 2024 and since revised. It defines a separate, non-robots.txt Markdown file intended to help an AI system locate a site's most important pages. The proposal states that some major AI providers publish llms.txt files for their own developer-documentation sites. This is a different claim from whether those providers' web crawlers read and act on other sites' llms.txt files during ordinary crawling, and none of the crawler-behavior documentation from Anthropic, OpenAI, or Google reviewed for this study makes that latter claim. llms.txt is best understood as a proposed and unevenly adopted convention rather than an established or universally supported standard.

Cloudflare's Content Signals Policy defines three independent, voluntary signals, search, ai-input, and ai-train, that a robots.txt file can set without naming individual bots. Cloudflare's own documentation notes that robots.txt compliance is voluntary and that some crawler operators may disregard published directives.

Research Question

Among a sample of 18 research universities, which specific AI-crawler tokens documented by Anthropic, OpenAI, and Google appear in institutional robots.txt files, and how many institutions in the sample publish an llms.txt file?

Methodology

Collection date: 18 September 2026, a single point-in-time snapshot.

Sample: 18 research universities selected for spread across five regions: Harvard University, Stanford University, the Massachusetts Institute of Technology, and the University of Toronto (North America); the University of Oxford, the Technical University of Munich, the University of Helsinki, Sorbonne Université, and ETH Zurich (Europe); the University of Tokyo, the National University of Singapore, the University of Melbourne, and the Australian National University (Asia-Pacific); the Universidad Nacional Autónoma de México and the Universidade de São Paulo (Latin America); and the University of Cape Town, the University of Nairobi, and Cairo University (Africa). This is a purposive convenience sample of prominent research universities, not a random or Webometrics-rank-stratified sample, and its findings should not be generalized beyond the sample.

Selection method: institutions were chosen for regional and linguistic diversity among widely recognized research universities, without regard to any expected result.

Variables: presence and readability of robots.txt; presence and Allow/Disallow status of 11 named AI-crawler tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, Google-Extended, PerplexityBot, CCBot, Bytespider); presence of a Content-signal line; presence of a Sitemap: directive; and the presence, response status, and structural completeness of an llms.txt file, checked for 13 of the 18 domains.

Evidence source: direct retrieval of https://<domain>/robots.txt and https://<domain>/llms.txt for each institution, cross-referenced against each provider's own published token list.

Analysis approach: descriptive frequency counts with stated denominators; no statistical inference; no causal interpretation of any pattern.

Limitations: disclosed in full below.

Results

llms.txt results (13 of 18 domains checked)

Analysis

Across the 13 universities with a readable robots.txt file, engagement with AI-crawler-specific configuration was limited to two institutions, each using a different mechanism. The University of Cape Town used per-bot Allow/Disallow rules to distinguish OpenAI's training crawler from its search crawler, in the same file that also blocks several non-AI automated crawlers, including Yandex, MJ12bot, AhrefsBot, and SemrushBot, using identical syntax. The University of Helsinki used the newer Content Signals Policy to express a training-specific opt-out without naming any individual bot.

This study observes that none of the 13 readable files named any Anthropic or Google-Extended token, and that AI-crawler-specific configuration, at 7.7%, was markedly less frequent than Sitemap: adoption, at 61.5%, in the same files. This study's design, a single-snapshot inspection of published robots.txt content, cannot determine why any individual university's file does or does not contain a given directive. Distinguishing between the range of possible explanations would require evidence beyond what this study collected.

The single confirmed llms.txt adoption in this sample, at the University of Toronto, is reported as an individual observation; with only one instance, it cannot support a claim about regional, ranking-based, or institutional-type patterns in adoption.

University Examples

The University of Cape Town's robots.txt file applies a full disallow rule to the user agent GPTBot, while applying the same standard path exclusions used for major search engines to OAI-SearchBot. The same file separately applies a full disallow rule to Bytespider, ByteDance, ByteDanceSpider, and Yandex. This is reported as an observation of the published configuration on the collection date, not an evaluation of its appropriateness.

The University of Helsinki's robots.txt file includes the line Content-signal: search=yes, ai-input=yes, ai-train=no, use=reference under its general user-agent block.

The University of Toronto's llms.txt file opens with a heading naming the institution, a one-sentence description, and six thematically organized sections, About the University, Academics and Students, Research and Libraries, Campuses and University Life, News and Community, and Official Resources, containing 23 links, each with a short descriptive annotation.

Implications for Higher Education

University leadership may find it useful to know that Anthropic, OpenAI, and Google each document separate, independently controllable crawlers, and that this study observed most sampled institutions' robots.txt files not naming any of them specifically.

Web and IT teams already maintain robots.txt for conventional purposes, with 8 of 13 files in this sample including a Sitemap: directive, and the syntax for adding AI-crawler-specific rules, where a decision has been made to do so, is the same syntax already used in these files for other crawlers.

Communications and marketing teams may want to evaluate llms.txt as a curation mechanism, distinct from robots.txt's restriction function, given that only 1 of 13 universities checked in this study had adopted one.

Research offices and libraries were not separately examined; this study looked only at each institution's top-level domain, not research-repository or library subdomains, which may carry different configurations.

Recommended Actions

• Review the current robots.txt file against the specific crawler tokens each provider documents, including Anthropic's ClaudeBot, Claude-User, and Claude-SearchBot; OpenAI's GPTBot, OAI-SearchBot, and ChatGPT-User; and Google's Google-Extended, and confirm the resulting configuration is the one the institution intends.
• Where a distinction between AI-training use and AI-search or retrieval use matters to an institution, note that per-bot rules, as used by the University of Cape Town, and the Cloudflare Content Signals Policy, as used by the University of Helsinki, are two currently available, independent mechanisms for expressing that distinction.
• Consider whether an llms.txt file, alongside the existing sitemap, would serve the institution's goals for directing AI systems to accurate official information; the University of Toronto's file is a publicly available example of the format.

These actions are not presented as guaranteeing any change in AI-search visibility, citation frequency, or ranking outcome. No source cited in this study makes such a claim, and this study did not test for one.

Limitations

This study attempted 18 university domains and obtained a determinate robots.txt result for 16; two domains, the University of Nairobi and Cairo University, returned a technical retrieval error unconnected to the substantive question under study and are excluded from the quantitative findings. The sample is a purposive convenience sample across five regions, not random or Webometrics-rank-stratified, and its findings cannot be generalized to the global university population or to any specific country's higher-education sector. Only the top-level institutional domain was examined per university; subdomains may carry different configurations. Content can change at any time, and this study reflects a single snapshot collected 18 September 2026. The response for the University of Helsinki's llms.txt is reported as indeterminate rather than assumed to indicate either presence or absence. This study did not verify whether the crawlers named in this article actually honor the robots.txt directives observed, only what the directives state; robots.txt and the Content Signals Policy are both voluntary mechanisms, as each provider's and Cloudflare's own documentation states. This study also did not examine other mechanisms, such as content-delivery-network or firewall-level bot management or HTTP response headers, through which an institution might separately control crawler access; these were outside its scope.

Conclusion

Among 13 university robots.txt files read in full for this study, none named Anthropic's or Google's documented AI-specific crawler tokens, and one distinguished between OpenAI's training and search crawlers. One university used a provider-neutral mechanism to express an AI-training-specific opt-out. One of thirteen universities checked had adopted an llms.txt file. Sitemap adoption was more frequent than any AI-crawler-specific configuration in the same sample. These findings are reported as observations collected on a single date from a non-random sample; this study's design does not establish why any individual institution's configuration takes the form it does.

Sources

• Anthropic, "Does Anthropic crawl data from the web, and how can site owners block the crawler?", Claude Help Center, last updated 7 April 2026 — support.claude.com/en/articles/8896518
• OpenAI, "Overview of OpenAI Crawlers" — developers.openai.com/api/docs/bots
• Google, "Things to Know about Google's Web Crawling", Google for Developers, last updated 3 March 2026 — developers.google.com/crawling/docs/about-crawling
• Google, "AI Features and Your Website", Google Search Central — developers.google.com/search/docs/appearance/ai-features
• llms.txt specification, Jeremy Howard, first published 3 September 2024 — llmstxt.org
• Cloudflare, "Managed robots.txt / Content Signals Policy" — developers.cloudflare.com/bots/additional-configurations/managed-robots-txt
• Direct review of publicly available robots.txt and llms.txt files for the 18 sampled university domains, collected 18 September 2026.

Suggested Citation

Webometrics by UNIRANKS (2026). AI Crawler Access at Universities: A Webometrics Study of robots.txt and llms.txt Across 18 University Domains. Webometrics.org, 18 September 2026.