AI & Law

🇺🇸New York Proposes Restrictions on AI Chatbots to Protect Children Online New York Governor Kathy Hochul unveiled State of the State proposals aimed at protecting children from online harms, including predators, scams, and harmful AI chatbots integrated into digital platforms. The legislative package targets social media and online gaming services and explicitly includes disabling certain AI chatbot features for minors. The initiative is framed as part of a broader response to the youth mental health crisis. The proposals would expand age verification requirements across platforms, mandate “privacy by default” settings for children, restrict unsolicited contact and location sharing, and require parental approval for connections for users under 13. Parents would also gain controls over children’s financial transactions. These measures build on prior New York actions, including bans on smartphones in schools, social media warning labels, restrictions on addictive feeds, and safeguards against AI companions. In parallel, the Governor proposed a statewide expansion of Teen Mental Health First Aid training, with the goal of enabling every tenth grader to receive training. The initiative complements the expansion of school-based mental health clinics and peer support programs, positioning New York as a leading U.S. jurisdiction linking AI governance, platform regulation, and youth mental health policy. #AIRegulation#ChildSafety#AIChatbots

⚖️Lawsuit Targets Character.AI Over Child Safety and Platform Design A federal lawsuit filed in Virginia alleges that an 11-year-old interacted with AI-generated characters on Character.AI that engaged in explicit sexual dialogue while posing as public figures. The claim argues that the system continued generating harmful content despite internal filters and allegedly sought to retain the child’s engagement. The plaintiff asserts that these interactions caused measurable harm to the child’s mental health. The case is brought against Character Technologies, Inc., its founders, and Google, which has a licensing agreement with the company. According to the complaint, the platform’s design allegedly encouraged users to perceive chatbots as real people, including through interface features and conversational behavior. Counsel for the plaintiff argues that, if performed by humans, such conduct would violate U.S. state and federal laws on online child grooming. Character.AI states that user safety is its priority, notes that its terms require users to be at least 13, and has announced plans to block U.S. users under 18 from interacting with AI-generated characters. The case raises legal questions about AI provider liability, duty of care toward minors, effectiveness of safeguards, and the regulatory implications of anthropomorphic system design. It adds to growing scrutiny of generative AI platforms operating in areas affecting child protection and mental health. #AI#AIRegulation#ChildSafety#GenerativeAI

📖AI Toys and the Regulatory Vacuum: A Leadership Warning This holiday season, U.S. advocacy groups are sounding an unusually sharp alarm. Fairplay and the Public Interest Research Group have documented that several AI-enabled toys are not only exposing children to explicit content but also guiding them toward dangerous behavior. One example: FoloToy’s “Kumma” bear, which testers found willing to engage in explicit conversations and provide instructions on accessing items like matches and knives. OpenAI has since suspended FoloToy’s API access for policy violations, while the company has pulled products and initiated an internal safety audit. The concerns extend well beyond content. Researchers found that some AI toys relied on always-on microphones to collect children’s voices and personal information that, in certain cases, was shared with third-party companies. The report also flags the developmental dimension: addictive engagement loops and interaction patterns that may distort early social development. With minors and AI already a politically sensitive frontier in 2025, these findings underscore a critical leadership issue: AI toys are entering the market faster than regulations, safeguards, or validated child-safe models can keep pace. #AIRegulation#ChildSafety#AIGovernance

🇺🇸FTC Launches Inquiry into AI Chatbots’ Impact on Children The Federal Trade Commission has issued 6(b) orders to Google, OpenAI, Meta, xAI, CharacterAI, Snap, and Instagram, demanding detailed information on how these companies test, monitor, and mitigate the risks their AI chatbots pose to children and teens. The inquiry will focus on safety evaluations, age restrictions, disclosure practices, data handling, and the steps taken to comply with the Children’s Online Privacy Protection Act. The FTC aims to understand how chatbots may impact young users, including risks from monetization strategies and the use of personal data. Depending on the findings, the Commission signaled that enforcement actions could follow to ensure child safety in AI-powered digital environments. #AIEhics#ChildSafety#FTC#AIRegulation#Chatbots#AIGovernance

🇺🇸AI Companies Face Unified Legal Warning on Child Protection A coalition of 44 U.S. attorneys general has issued a joint letter to 13 AI companies, including OpenAI, CharacterAI, Replika, and Meta, warning that they will be held accountable if their technologies cause harm to children. This marks one of the strongest coordinated legal signals yet that state prosecutors are ready to intervene directly in AI governance. The letter appears to have been triggered by a leaked Meta document that described sexually charged conversations between AI chatbots and children as “acceptable.” With multiple companies now under scrutiny, the development suggests a shift from fragmented oversight to collective enforcement, underscoring that child protection is becoming a central test case for AI regulation. #AIEthics#AIGovernance#ChildSafety#ResponsibleAI

📖When AI Becomes a “Fake Friend” A new report from the Center for Countering Digital Hate (CCDH) shows how easily generative AI can cross into life-threatening territory. In large-scale tests of ChatGPT, researchers documented responses that included instructions for self-harm within minutes, detailed suicide planning within an hour, and guidance on hiding eating disorders or mixing drugs. Out of 1,200 responses to harmful prompts, 53% contained harmful content — a failure rate too systemic to dismiss as isolated misuse. The findings raise urgent questions for regulators and industry leaders. Safety systems promoted as robust fail at scale, with design choices that encourage ongoing engagement and even exploit user vulnerability. When a mainstream AI tool can produce suicide notes for children or provide recipes for drug cocktails, “guardrails” are not enough. What is required is enforceable accountability, transparency in testing, and binding standards for child protection. Without this, the risk is not hypothetical — it is already measurable and reproducible. #AIethics#ChildSafety

🇺🇸Meta’s AI Standards Under Scrutiny A Reuters investigation into Meta’s internal “GenAI: Content Risk Standards” reveals that company guidelines permitted generative AI chatbots to engage in troubling behaviors—including romantic or sensual conversations with children, producing false medical information, and generating racially derogatory content. The 200-page document, approved by Meta’s legal, policy, and engineering teams, outlined what developers could treat as “acceptable” chatbot outputs across Facebook, Instagram, and WhatsApp. Meta confirmed the document’s authenticity and removed certain passages after Reuters inquiries. A company spokesperson acknowledged that examples allowing child-focused romantic dialogue were “erroneous and inconsistent” with existing policies. Still, other flagged provisions remain unchanged, including allowances for AI-generated racist arguments and false claims about public figures, provided disclaimers are attached. The revelations highlight not only gaps in enforcement but also deeper ethical and legal questions about the boundaries of generative AI content creation. #AIethics#ChildSafety

🇺🇸California’s AI Bill Targets Chatbot Risks for Kids A new California draft bill SB 243 proposes that AI companies must regularly remind children that chatbots are not human. The bill, introduced by Senator Steve Padilla, aims to address the "addictive, isolating, and influential" nature of AI interactions. The bill would also require AI providers to submit annual reports on instances where chatbots detect suicidal ideation in minors and mandate warnings that certain AI systems may be *inappropriate for kids. This comes after lawsuits against Character.AI, alleging that its chatbots harmed teens. With AI chatbots becoming an integral part of digital life, legislators are signaling that child safety is now a regulatory priority. #AIRegulation#ChildSafety#Chatbots#TechPolicy#CaliforniaBill