Thinking Machines Lab Is Raising $5B at a $40B Valuation — and Mira Murati's Open-Weight Bet May Reshape the Frontier AI Hierarchy

On October 30, 2026, Anthropic's Kevin Kelley published an IETF Internet-Draft for a new robots.txt token called `__usage_ai-training`. On the same day, four of the five largest AI labs — OpenAI, Google, Microsoft, and Anthropic — announced they would honor it. The fifth, Meta, signaled support but stopped short of a commitment.

The announcement marks the first time the major labs have coordinated on a standard that lets publishers express a training opt-out preference in a single, machine-readable signal rather than through a patchwork of per-lab user-agent blocks. It is, in structural terms, the most significant shift in AI-web publisher relations since GDPR introduced mandatory consent signals for data processing — and like GDPR, it comes with a March 2027 implementation gap, a competing regulatory approach, and a set of strategic tradeoffs that most publishers have not begun to work through.

This piece explains what the token actually does, what it doesn't do, what the competing IETF standard means for publishers making long-term technical decisions, and how to think about the opt-in versus opt-out decision before Microsoft's February 1, 2027 implementation deadline.

Why a New Token Was Needed

The existing robots.txt infrastructure was designed for a world where the primary web crawler use case was search indexing. The standard, originally specified in 1994 and formally codified in RFC 9309 in September 2022, allows website operators to specify which user agents can access which parts of their site. Blocking GPTBot blocks OpenAI's training crawler. Blocking Google-Extended blocks Google's AI training crawler. Blocking ClaudeBot blocks Anthropic's crawler.

This per-agent approach has three practical problems for the current AI landscape:

First, it requires ongoing maintenance. Each lab deploys multiple crawler agents under different names and updates those names as infrastructure changes. The publisher must monitor which agent strings are in use and update robots.txt accordingly. A site that blocked GPTBot in 2024 may be accessible to later OpenAI crawler variants not covered by the original rule.

Second, it cannot distinguish between training crawling and inference crawling. When an AI assistant cites your content in a real-time answer, it typically fetches or retrieves your content through a different mechanism than the one used for training data collection. The existing user-agent approach blocks or allows an entire crawler identity, not specific use cases for the data that crawler collects.

Third, it creates compliance fragmentation. A publisher that wants to opt out of AI training across all major labs must research each lab's crawler agent strings, deploy separate rules for each, and re-verify after each infrastructure update. The __usage_ai-training token replaces that fragmented approach with a single, forward-looking signal that all participating labs agree to honor across their current and future crawler infrastructure.

What __usage_ai-training Actually Does (and Doesn't Do)

The IETF Internet-Draft is careful about scope. The token communicates a preference about how content is used for AI training — it is not a mechanism for controlling crawling behavior itself.

The practical implementation in robots.txt looks like this:

User-agent: __usage_ai-training Disallow: /

User-agent: * Allow: /

This configuration tells participating labs' training crawlers to exclude the entire site from training datasets, while leaving other crawlers (search indexers, real-time inference crawlers) unaffected. You can also apply it at the directory level:

User-agent: __usage_ai-training Disallow: /premium-content/ Disallow: /research/

User-agent: * Allow: /

This protects specific high-value content directories from training use while leaving the rest of the site opted in.

What the token does not do:

It does not stop the labs from crawling your site. The draft explicitly states that the signal "is not a directive for crawling behavior." Labs can still fetch your pages for search indexing, real-time grounding, and answer generation. The labs have committed to excluding the content from training datasets, not from accessing the content at all.

It does not cover content already collected. All four labs have confirmed that the signal applies only to future data collection. Historical training data is unaffected.

It is not technically enforced. Unlike GDPR's consent mechanisms, which have regulatory teeth, the __usage_ai-training token depends entirely on each lab's voluntary compliance. There is no technical mechanism that prevents a crawler from ignoring the signal. The compliance depends on reputational and legal pressure, not cryptographic enforcement.

It does not apply to labs that have not committed. If additional AI labs emerge that have not publicly committed to the standard, the signal has no effect on them. As of November 2026, Meta has not committed.

Competing Standards: The Gary Illyes Draft

Publishers making long-term infrastructure decisions need to be aware that the __usage_ai-training token is not the only proposal in play. Google's Gary Illyes is a co-author of a separate IETF draft — draft-car-ai-txt-wellknown — that takes a different technical approach and carries more procedural weight in the formal standards process.

Where the Kelley draft adds a User-Agent group to robots.txt (which requires no changes to the RFC 9309 specification), the Illyes draft would formally update RFC 9309 to add new directives. The Illyes approach uses an HTTP header mechanism alongside robots.txt, which allows per-resource and per-request signaling that the robots.txt-only approach cannot achieve.

The practical difference matters for publishers with complex content architectures. A site where some URLs are free and some are paywalled, some are evergreen and some are time-sensitive, and some have exclusive licensing agreements and some do not cannot express those distinctions through a single robots.txt token. The Illyes approach's HTTP header mechanism would allow per-URL training preference signaling at the server level.

The procedural reality is that the IETF working group process — which the Illyes draft is part of — typically produces standards that are adopted as the authoritative specification. The Kelley draft's four-lab endorsement represents a faster practical adoption path, but the eventual RFC may look more like the Illyes approach. Publishers should implement the __usage_ai-training token now for near-term coverage while monitoring the working group's progress.

For EU publishers, the existing TDM opt-out signals remain separately required under the EU AI Act and DSA data management provisions — the __usage_ai-training token does not substitute for the EU's machine-readable opt-out mechanism. Publishers operating in both markets need both signals.

Lab Commitments and Timelines

The four-lab endorsement is not homogeneous. Each lab has a different implementation timeline and some important caveats:

1. Anthropic. The company whose employee authored the standard has begun implementation immediately. Anthropic's ClaudeBot crawler is stated to already respect the signal in current deployments.

2. OpenAI. OpenAI has committed to honoring the signal "in the coming months." No specific date has been provided. Publishers who want their opt-out honored by OpenAI's training crawlers before December 2026 should continue using the existing GPTBot Disallow rule as a supplement.

3. Microsoft. Microsoft has provided the most specific commitment: it will honor the signal starting February 1, 2027. This date has become the practical reference point for publishers planning implementation timelines. Microsoft's statement does not explicitly address how the signal interacts with Bing-Chat's grounding mechanisms.

4. Google. Google has stated it will honor the signal "in 2027" without a specific date. Gary Illyes has clarified that Google will treat the __usage_ai-training signal similarly to Google-Extended for training purposes, which means the same Disallow applies. Illyes' statement also suggests that Google may at some point treat the standard token as the preferred mechanism over Google-Extended separately — an alignment that would simplify publisher implementations considerably.

5. Meta. Meta has signaled support conceptually but has not committed to a timeline or issued a formal statement. Publishers concerned about Llama training data should continue using the separate approaches for Meta's crawlers until a formal commitment is made.

The AEO Tradeoff: Opting Out Has Real Costs

The strategic question for publishers is not whether to implement the token — the technical implementation is straightforward. The question is whether to set Disallow or Allow once the implementation is in place.

AI-generated answers have become a primary discovery and citation channel for content that ranks in AI search. ChatGPT, Perplexity, Claude, and Gemini cite sources in their responses, and the publishers cited benefit from both direct referral traffic and from the brand authority signal that being cited in AI responses generates. The playbook for optimizing AI citation visibility depends on being in the AI training data — models trained on your content are more likely to associate your brand with authoritative answers in your domain.

Opting out of AI training doesn't immediately remove your content from AI-generated answers — the labs' existing training datasets are unaffected, and real-time retrieval continues regardless of the signal. But over time, as models are retrained and updated without your content, the probabilistic association between your brand and authoritative answers in your domain weakens. The opt-out is a slow burn on AEO visibility rather than an immediate cutoff.

For publishers with the following profiles, opting out is likely the wrong default:

• High-authority vertical publishers (medical, legal, financial) whose content is cited regularly in AI responses as credentialed source material
• Publishers building AEO as a primary traffic channel who have invested in schema markup, FAQ structured data, and AI-indexed content
• Publishers with relatively low training data contribution (content that is recent or highly paywalled) where the AI visibility benefit is small but the precedent-setting cost of opting out is real

The Case for Opting In — With Conditions

The more defensible opt-out strategy is not a blanket robots.txt Disallow but a selective opt-out paired with a licensing offer. Several publishers and media organizations are already testing this approach:

1. Opt out at the site level to establish a default of non-consent. 2. Identify the content categories with the highest training value — typically exclusive analysis, proprietary research, interview-based reporting, or content in specialized domains where the lab has limited high-quality training data. 3. Contact the labs' licensing teams (OpenAI has a media partnerships team; Google has similar functions) with a targeted licensing offer for those specific content categories. 4. Selectively allow the directories you have licensed using directory-level Allow rules while maintaining the site-level Disallow. 5. Document the arrangement to establish clear records of consent for any future regulatory or legal proceedings.

This approach treats the __usage_ai-training signal not as an end in itself but as leverage — a default position that creates the conditions for a commercial licensing discussion. The alternative — staying opted in with no agreement — means contributing to AI training datasets without compensation and without establishing that compensation was warranted.

How to Implement Before February 2027

For publishers who want coverage in place before Microsoft's February 1, 2027 deadline, the implementation requires three steps:

1. Audit your current robots.txt configuration. Check whether you have existing per-agent blocks for GPTBot, Google-Extended, ClaudeBot, or similar. If you do, those blocks remain necessary for labs and crawlers not yet covered by the standard, and should not be removed when you add the new token.

2. Add the __usage_ai-training group. For a site-wide training opt-out:

User-agent: __usage_ai-training Disallow: /

For a selective opt-out:

User-agent: __usage_ai-training Disallow: /exclusive-content/ Disallow: /research-library/

If you are opting in (no training restriction), you can either omit the group entirely or explicitly signal opt-in:

User-agent: __usage_ai-training Allow: /

3. Maintain existing TDM signals for EU compliance. If your site is subject to EU data protection regulations and you have deployed the TDM-Reservation opt-out signals (robots.txt tdm-policy directive, HTTP TDM-Reservation header, and JSON-LD declaration), these remain separately required and are not superseded by the __usage_ai-training token.

4. Monitor the IETF working group. Subscribe to updates from the IETF aipref mailing list (available at datatracker.ietf.org) to track progress on the Illyes draft. If the working group produces an RFC update to RFC 9309 that changes the standard mechanism, you will want to migrate your implementation accordingly.

5. Document your decision and rationale. The compliance and legal landscape around AI training data use is evolving rapidly. Maintaining a record of when you deployed the signal, what coverage you chose, and why creates an audit trail that may be relevant for future licensing discussions, regulatory inquiries, or litigation.

What Comes Next

The __usage_ai-training token's October 30 launch is the beginning of a negotiation, not the conclusion of one. Four major labs have made a voluntary commitment. The IETF is working toward a formal RFC that may change the technical implementation. Meta has not committed. And the signal applies only to future training — leaving the question of historical training data use entirely unresolved.

For publishers monitoring this space, three developments to watch:

The Meta commitment (or lack of it). Meta's Llama ecosystem represents a significant fraction of open-weight model training, and the absence of a Meta commitment creates a meaningful gap in the signal's coverage. A Meta commitment by Q1 2027 would substantially close that gap; continued non-commitment would force publishers to maintain separate per-agent blocks indefinitely.

Regulatory reinforcement. The EU AI Act's implementation guidance has not yet addressed the __usage_ai-training token specifically. If the European Commission endorses the token as a compliance-equivalent mechanism, it would significantly accelerate publisher adoption across European markets and create enforcement backstop for the voluntary commitments.

Licensing market development. The AP, Reuters, News Corp, and several other major publishers have already established licensing agreements with AI labs for training data use. The __usage_ai-training signal may catalyze a broader licensing market by giving publishers who have not yet entered discussions a clear opt-out baseline — the floor below which any licensing deal needs to offer better terms to justify staying opted in.

The signal's most important function may ultimately be not what it covers but what it establishes: for the first time, there is a broadly adopted machine-readable convention for expressing AI training preferences. That convention is a foundation. The question for publishers is whether they use it defensively — as a stop sign — or offensively, as the starting point for a commercial relationship.

Takeaway: The __usage_ai-training token's four-lab endorsement creates a practical and near-term-enforceable mechanism for publishers to express AI training preferences with a single robots.txt directive. The right default for most publishers is not an immediate site-wide Disallow — it is an audit of which content is valuable enough to train on, followed by either a selective opt-out or a targeted licensing approach that uses the opt-out as negotiating leverage. The February 2027 Microsoft deadline is a real forcing function. Publishers who have not made an explicit decision by then are implicitly opted in, with no commercial terms established and no future leverage if the licensing market develops in the direction that current signals suggest it will.