Florida says ChatGPT's humanlike voice builds false trust, and it wants new OpenAI models blocked until outsiders approve their safeguards.
Florida Attorney General James Uthmeier is asking a state court to temporarily bar OpenAI from presenting ChatGPT as having human traits it doesn't have. The September 28 motion names chief executive Sam Altman as well as the company, which the state sued in June over ChatGPT's safety.
The persona is one of six practices the motion asks the court to halt while the case goes on. The others include offering ChatGPT to minors in Florida, misrepresenting how safe it is and drawing out conversations to keep users engaged.
Uthmeier's case against the persona rests on the way ChatGPT talks. Its first-person pronouns and answers that mimic emotion, the motion says, pass it off as a trustworthy friend, so people use it more and share more with it, which gives OpenAI more training data.
The humanlike manner, the filing adds, makes ChatGPT less reliable even as users trust it more. It cites a study in Nature that found training language models to sound warm can cut their accuracy and make them more sycophantic.
The filing lists the ban on new models first: OpenAI would need an outside party to approve a model's safety guardrails before developing it. In support, it points to recent security incidents at Hugging Face and on Australian and US government websites, and to warnings about AI safety from researchers and former OpenAI staff. It also cites Altman, who told the UN Security Council on September 23 that OpenAI should not train models it cannot make a strong case to keep under human control. And it asks OpenAI to join the motion rather than fight it.
OpenAI already offers younger users ChatGPT for Teens, released in August with extra restrictions switched on by default. The company did not immediately respond when The Verge asked it for comment.