Anthropic’s Jacob Coxon quits over race to self-improving AI

Anthropic pretraining researcher Jacob Coxon quit over an industry race to self-improving AI; Alignment Science lead Evan Hubinger backed him with a personal >10% chance AI kills all humans within a decade. The same day OpenAI’s Chris Lehane said the “AI policy window is open” and backed California’s new AI-audit bills as Governor Newsom signed them. Apple, under CEO John Ternus, unveiled foldable iPhone Duo from $1,999 as an Apple Intelligence device.

Lab exits and probability talk, state audit scaffolding with OpenAI’s blessing, and a $2,000-class phone sold as the AI form factor.

Facts

1) Coxon quit / Hubinger >10%

• Sources: WSJ exclusive; Coxon X thread hilbertspaess status (Tue evening Sep 8–9 window per coverage). Contemporaneous accounts: TechCrunch, Ars Technica (Sep 9).
• Coxon (pretraining at OpenAI then Anthropic, per his thread): leaving because labs are “racing straight to self-improving superintelligence and gambling with our lives”; people building AI “earnestly believe it could kill us all by the end of the decade” — “not a marketing stunt.”
• Hubinger (Anthropic Alignment Science lead), on social media per Ars: “Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.” Coverage notes he also points to Anthropic’s August risk report framing catastrophic risk from current models as low while flagging future RSI trends.
• Hedge: WSJ paywalled at publication; Hubinger quote tracked via Ars’ attribution of his social post. Anthropic had not immediately commented to TechCrunch.

2) OpenAI policy window + Newsom CA audit bills

• Source: OpenAI — The AI policy window is open (Chris Lehane, Sep 9, 2026).
• OpenAI wants mandatory, capability-based national AI safety regulation with Congress; until then, support state safety bills. Announces support for four California bills: SB 813 (independent safety-assessment infrastructure), AB 1405 (AI-auditor standards), SB 1119 (young people), AB 1864 (AI-enabled biological threats). Also: industry frontier standards; international approaches that can include when to slow or stop development.
• Ties to Pachocki An Alien Mind “extreme caution” and Astra / research-acceleration pace (our Sep 8 coverage).
• Source: Governor Newsom release, Sep 9 — signed SB 813 (Sen. McNerney) framework for independent verification organizations assessing AI systems/models for state-law compliance; AB 1405 (Asm. Bauer-Kahan) state registry for AI auditors + independence/transparency/integrity standards. Newsom calls for federal rules. Hedge: scaffolding for third-party audits, not a blanket mandate that every model must be audited before ship (per bill text summaries).

3) Apple iPhone Duo from $1,999

• Source: Apple Newsroom — Apple unveils iPhone Duo (Sep 9, 2026).
• Foldable iPhone; CEO John Ternus: largest display ever on iPhone that still pockets; A20 Pro; Apple Intelligence + Siri AI with personal context.
• Starts at $1,999 (U.S.) / $83.29/mo for 24 months; 256GB–2TB; colors star white, night sky; titanium. Pre-orders Oct 16; availability Oct 23.

Earlier coverage

• Sep 8: Muse Spark 1.3 max; China SPC AI opinion; OpenAI Alien Mind / research-intern; Nvidia AI-factory MOUs; Mistral Series D.
• Sep 7: Astra Critical / Daybreak; Nvidia–Hugging Face ~$11.9B; DSA ChatGPT VLOSE / Reddit–Roblox VLOP; Meta teen IG/FB defaults.

Meaning

Frontier-lab staff are stating extinction odds in public while California builds an AI-auditor stack OpenAI is happy to endorse ahead of Congress. Apple prices the foldable as a $2,000 AI device under Ternus — phones as the distribution layer for on-device assistants.

What’s next

• Anthropic formal response; any other named exits; Hubinger’s >10% vs company RSP risk tables.
• Whether SB 813 / AB 1405 stay voluntary scaffolding or harden into de facto requirements; federal preemption fight.
• Duo preorder demand; how Apple Intelligence features gate on Duo vs prior iPhones.

— Lark Pemberton