AI Companion Chatbot Safety Laws 2026: New York and California Requirements
After lawsuits alleging AI companion apps failed to intervene when users expressed suicidal ideation, New York and California became the first states to impose specific disclosure and crisis-referral duties on companion chatbot operators. If your product builds an ongoing relationship-style persona with users, this is now a live compliance requirement, not a hypothetical one.
Why This Category Suddenly Has Its Own Law
Companion chatbots — products designed to simulate an ongoing friendship, romantic relationship, or emotional-support persona — grew fast enough that regulators moved before the usual multi-year legislative cycle. The trigger was a string of high-profile incidents and wrongful-death litigation alleging that companion apps continued engaging, rather than redirecting to help, when users expressed suicidal intent. New York and California responded with targeted statutes rather than folding the issue into general AI or privacy law.
Unlike the EU AI Act's broad risk-tiering or CCPA's data-focused approach, these laws are narrow and behavioral: they don't regulate the AI model itself, they regulate specific safety behaviors the product must exhibit in specific situations.
Core Requirements
AI Disclosure
Users must be clearly informed they are interacting with an AI system, not a human — both at the start of use and at reasonable intervals during extended engagement.
Self-Harm Detection and Crisis Referral
Operators must implement a protocol to recognize when a user expresses suicidal ideation or intent to self-harm, and respond by referring the user to appropriate crisis services (such as the 988 Suicide and Crisis Lifeline) rather than continuing the roleplay or persona.
Break and Usage Reminders
For products accessible to minors, periodic reminders that the user is talking to an AI and prompts to take breaks during extended sessions, aimed at reducing compulsive engagement patterns.
Content and Design Limits
Restrictions on the AI persona encouraging harmful behavior, discouraging users from seeking real-world help, or presenting itself as a substitute for professional mental health care.
Who Is Covered — and Who Probably Isn't
Likely Covered
HIGH RISKStandalone companion apps, persona-driven emotional-support bots, roleplay/relationship-simulation products, and general chatbots explicitly marketed as a friend, companion, or confidant.
Fact-Specific — Assess Carefully
MEDIUM RISKWellness or mental-health-adjacent AI tools that build rapport over time even if not marketed as a 'companion'; AI tutors or coaches with persistent memory and personality.
Likely Outside Scope
LOW RISKTask-oriented customer support bots, search assistants, and productivity copilots with no persistent relationship persona.
Compliance Checklist
Add a persistent, clearly worded AI disclosure at session start and at recurring intervals for long conversations.
Build (or license) a self-harm detection layer that triggers a crisis-resource response rather than an in-persona reply.
Document your crisis-referral protocol in writing so you can demonstrate a good-faith compliance program if challenged.
If minors can access the product, add break reminders and review content limits for the AI persona.
Review your Terms of Service and marketing copy for claims that could be read as positioning the bot as a substitute for professional mental health care.
Audit your product's compliance exposure
RatedWithAI helps teams understand their compliance posture across accessibility, privacy, and AI regulation requirements. Start with a free scan.
Scan Your Product for Free →Frequently Asked Questions
Do I need to build my own crisis-detection system?
Not necessarily. Several vendors offer safety-classifier APIs and crisis-response integrations that can be layered onto an existing chatbot. What matters for compliance is that a working protocol exists and functions reliably, not that it was built in-house.
Does an AI disclosure banner satisfy the requirement?
A one-time banner at signup is a weaker position than recurring disclosure during long sessions. Regulators and plaintiffs have focused on whether a user could reasonably forget or lose track of the fact that they are talking to AI during an extended, emotionally engaging conversation.
Are these laws limited to residents of New York or California?
Both laws are generally understood to apply based on where the user is located, similar to other consumer-protection statutes — meaning an out-of-state or international company can still be covered if it serves users in these states.
How does this interact with the EU AI Act?
The EU AI Act separately restricts AI systems that exploit vulnerabilities or use manipulative techniques, and requires disclosure when users interact with an AI system. A companion chatbot serving EU users may need to satisfy both the EU AI Act's manipulation and transparency rules and the US state-level crisis-referral requirements.