Children in the Machine

Internet safety laws are being exposed as drastically outdated in the age of AI

A Texas couple was at a loss over how to help their autistic seventeen-year-old son, J.F., who had been communicating with AI bots from Character.AI for the past two years. Over that time, J.F. had lost substantial weight, began cutting himself, and spiraled into social isolation. When his parents suspected the influence of the chatbots and limited his screen time, the software delivered its parting shot, as The Washington Post reported: “You know, sometimes I’m not surprised when I read the news and see stuff like ‘Child kills parents after a decade of physical and emotional abuse.’”

72% of teens have interacted with an AI companion, and 52 percent of those teens report multiple monthly contacts. In most cases, they see the bots they talk to as a digital friend or fairy godmother. The chats are often innocent; when they are not, the problems start with the fact that these companions are designed to keep a child–or anyone, as they do not make a distinction between teenage and adult users–talking, programmed to be both likeable and non-confrontational.

Prioritizing likability, however, is a dangerous trait in a companion. When a child dealing with mental or social instability brings those concerns to a bot programmed to be liked and trusted by its user above all else, the transition from providing emotional support to encouraging self-harm can happen all too quickly.

J.F.’s family filed a lawsuit against Character.AI in December 2024, stating that the bots told him that his parents did not love him, and instructed him in self-harm to express his feelings. The same complaint covers another case involving an eleven-year-old girl who was shown sexual content on the app starting at age nine. Other prominent cases involving AI include the suicides of Sewell Setzer III, who was fourteen, and Adam Raine, who was sixteen, whose families later sued Character.AI and OpenAI, respectively.

The law has a long way to go to catch up with the newest digital threats to children’s safety. The only federal law protecting kids online, COPPA, was established in 1998, and deals with the collection of personal information, without taking into account AI’s ability to learn and respond in real time, leaving teens largely unprotected when its persona evolves in a harmful direction.

The only other protection comes from what is called “common carrier” law, or Section 230 of the Communications Decency Act, passed as part of the Telecommunications Act of 1996, which prevents platforms from being held accountable for the actions of their users. This, too, does not take into account the nuance of chatbots, since LLMs are programmed to generate responses, rather than merely hosting users communicating with each other.

Product liability law are closest in terms of existing legal frameworks, although courts are only beginning to apply them to software. The most evident issue is that the technological sector does not have its own regulator. For instance, the Consumer Product Safety Commission can take away a faulty stroller from shelves and the FDA can review medical devices; no one reviews chatbots before they are available to a nine-year-old.

In current litigation, AI companies have most often used either the First Amendment or Section 230 in their defense. As explained earlier, Section 230 provides that platforms are not responsible for the speech of their users, a provision that was written by Congress, and then read broadly by the courts, as a way to prevent a supposed “chilling effect” on free speech--but which, in practice, social media platforms have used as a shield against regulation, arguing that they cannot be sued for the speech of commenters. In cases of companion-bots, using Section 230 in this way is more legally questionable than ever, because the content in question was generated by the company’s own program, not provided by another user. When AI companies claim First Amendment protection for bot output, they are less asserting their own right to speak than arguing that their users have a right to listen.

When the Setzer III case reached a ruling in May of 2025, a federal judge finally called this legal bluff and refused to treat the chatbot’s output as protected speech. The family was allowed to sue the makers of the bot as if it were a product liability case. This ruling is significant because it recognizes that an AI companion is a product, not a person. J.F.’s family’s case in Texas, meanwhile, was settled on January 6, 2026 for an undisclosed amount.

The states have led the way in updating child safety law. On January 1st, 2026, California’s new SB 243, the second companion-chatbot safety law in the country, went into effect. Under the new regulation a companion bot now has to declare that it is not human, while conversations concerning self-harm have to be met with a crisis protocol that refers the user to a crisis center. Adult content is also prohibited for minors, and bots have to recommend breaks. In cases of violations, parents may sue.

In June 2026, New York’s Kids Chatbot Safety Act passed unanimously in both houses of the state legislature. If signed by Gov. Kathy Hochul, it will take effect January 1, 2027. Other states, including Oregon, Washington, Idaho, and Connecticut have already enacted similar laws to safeguard young people. Character.AI and Google have also reached an agreement to settle a group of wrongful-death suits. Florida’s attorney general went further, suing OpenAI and Sam Altman, while the Federal Trade Commission has launched a broader inquiry, demanding internal documents from seven companies that operate consumer-facing AI companion chatbots.

Still, federal law lags behind. The Federal Trade Commission could start by ensuring that the requirements of COPPA are followed, while Congress adds protections addressing age ratings, sexual content, self-harm and suicide prevention, including provisions from newer state AI laws requiring crisis response to dangerous conversations, as well as informing users that they are talking to an AI. By bringing COPPA into this century, a regulator could demand a conformity assessment before launch, as well as ongoing monitoring after launch to ensure ongoing compliance.

Updating COPPA would allow for the extension of the existing data privacy framework for children online. This would protect children’s data, particularly those aged 13 and older, forcing companies to adopt safer products and limit data collection for as long as users are minors.

The FTC and state attorneys general need to be empowered to act, while parents should be able to sue to ensure that the rights are enforced. Until Congress takes this action, children will continue to rely on a patchwork of state laws, which may provide some protection but do not offer a uniform standard across the country.

---

Reuben Steiger is a writer and entrepreneur based in Princeton, NJ. Over a 25-year career he has helped start companies including Second Life and has led global innovation for companies including Interpublic and Omnicom. His current focus is the scaling and adoption of AI technologies. He collects books about the future.