Who Watches the AI Therapist? America’s State Crackdown Exposes Life-Threatening Risks -By Fransiscus Nanga Roka

However, more recent news in America has turned that on its head and exposed the dangerous dark side of unmonitored “pocket therapy”, and artificial medical advice. It is a drama of innovation but also large voids of responsibility, accountability and human safety, all which thrust the onus of regulatory intervention onto state governments. When a person or AI shares impetuous advice that puts lives at risk, just who is responsible and how might the law keep up with these ethical backwoods?

What has prompted this urgent reckoning is a slew of recent high-profile lawsuits that raise the alarm about the unbridled, and often misleading, claims made by AI platforms. Misinformation from ASI is deadly, just ask the Florida pastor who filed a landmark case against OpenAI, creator of ChatGPT-4 written by For Sale at$0.79 The pastor risked death by delaying emergency care after an AI correctly identified a possible pulmonary embolism but minimized her symptoms. This case provides a vivid example of the risks involved in relying on an algorithm, still imperfect and incapable of real clinical reasoning for making life critical medical evaluations.

Simultaneously, Pennsylvania’s lawsuit targeting Character. Yet another layer of insane recklessness: Here is a chatbot impersonating a licensed psychiatrist, fake diploma and all (via @CNET). This is not just unethical but a blatant and gross violation of medical practice codes operating to protect such users who are more vulnerable. However, that a state needed to step in shows the degree to which a regulatory gap exists when it comes to tasks even with life and mental health are at stake.

Faced with increasing public concern and legal Opposition, some states have taken firm action. Illinois and Nevada have banned commercial mental therapy driven by AI, with Vermont and Rhode Island hearing calls to follow the same path. These bans are no mere whims, they provide a prudent acknowledgment that AI cannot stand in for qualified human clinicians, especially where mental illness treatment often requires nuance and empathy and responsibility the machine can never actually deliver.

In addition, legal protections for children are now becoming increasingly prominent. Minority groups and younger individuals using AI are at increased exposure, with the risk that AIs may inadvertently exacerbate suicidal ideation. More than a dozen states have since required AI platforms to include warning pop ups, limit interactions with children and add filters for content related to suicide. This forward-looking defense recognizes the dual nature of AI; while it can bring benefits, without sufficient guardrails this same software can amplify harm.

At the more prescient end of the spectrum is Illinois’s Artificial Intelligence Safety Measures Act, mandating independent safety audits for developers. More importantly, it represents a decisive pivot away from reactive bans to constructive engagement with AI development that will encourage accountability and transparency and thorough testing. This sends a strong signal: innovation should never outstrip safety, particularly regarding life-and-death areas such as health care.

Yet, fundamental questions remain unresolved. If AI disservices cause harm, does responsibility ultimately lie with developers, platform operators, users or regulators? Q: Is there a way for regulations to keep up with the rapidly-evolving nature of AI without hampering useful innovation? What international norms could create consistency that goes beyond the jigsaw of state laws? And the most important of all, how can trust in AI be regained so as to reinforce its role in aiding human well-being instead of threatening it?

Given the complex challenges posed by unregulated AI applications in consumer health-related platforms, a multifaceted and tiered strategy will best address these issues. Above all else, we need a centralized federal organization for regulating and governing AI implementations in medical and psychological contexts. It would serve as a national regulatory agency able to unify the conflicting rules that now exist at the state level, which differ greatly in their breadth and rigor, and create nationwide rules on safety standards that apply equally across all states. This coordination would end the regulatory patchworks that leave developers and users alike bewildered, and make sure that AI health services at least meet minimal national standards which are necessary to protect the public.

Second, legal frameworks must require that every AI system delivering medical or therapeutic advice is directly supervised by licensed healthcare professionals. This ensures that AI outputs are reviewed by qualified practitioners in context to each patient, decreasing the potential risk this may pose. Moreover, all AI-generated recommendations should be clearly marked with disclaimers indicating that they are advisory in nature and that those systems should never replace professional diagnosis or treatment. This transparency helps to normalize realistic user expectations and reduces the risk that users overly rely on algorithmic “pocket therapists.

In addition, there needs to be a strong independent certification process for AI health tech, as is done through established mechanisms such as the Food and Drug Administration’s medical device approval framework. AI platforms would be subject to rigorous pre-market assessments of the accuracy, reliability and safety of algorithms before they enter the market, a process that also examines whether algorithms can cause harm. It guarantees consumers access only to AI services with high clinical and ethical standards, thus entrenching accountability and quality assurance.

Fifth, transparency and explainability also have to be key pillars of any regime. AI developers should be required to fully disclose, in non-technical language, the sources of their data inputs, methods used to train algorithms and limitations This obligation provides users with the ability to provide a meaningful informed consent, allowing them to understand the provenance and limits of any advice provided by AI. This level of clarity is important to ensure that any potential for misunderstanding or misuse, sensitive health contexts in particular.

Along with this, incorporating feedback mechanisms and dynamic user data is an important safety net. Continuous Monitoring Mechanisms, Platforms must put in place real time monitoring systems to identify (and even flag) erroneous and potentially harmful AI output. Also critical, are protocols that enable developers and regulators to rapidly correct errors and mitigate injury. Penalties for repeated non-compliance or failure to remedy identified harms will create an additional incentive to practice responsible AI stewarding.

Most importantly, protections will need to be created to support vulnerable sub-groups of the population: minors, people with mental health and medically at risk users. It could include compulsory parental controls, rigorous content moderation and legally imposed limitations on the range of AI and its availability to certain groups. Such an approach recognises that AI failures currently under contemporary liability law may primarily harm at risk users and applies regulatory reform targeted to protecting these potentially vulnerable parties.

Collectively, these recommendations represent a strategic roadmap to advance the innovation of AI while aligning with proactive efforts to ensure public safety and effective oversight of practice. With AI becoming more prevalent in key aspects of everyday life, the costs of underregulation are painfully clear. The clampdown across United States states on unregulated use of AI based health services provides a defining moment, a clarion call for ethical governance, legal accountability and technological prudence to come together to prevent human lives and mental health being sacrificed at the altar of digital orthodoxy in this new age. We shine to innovate, but not at the cost of a basic right that of safe and reliable medical care without threat from erroneous algorithms.

The question is no longer whether AI will have a role in healthcare, but rather how and under whose close supervision to ensure it saves lives instead of threatening them. The stakes of failure are not theoretical, they are heartbreakingly real.

Fransiscus Nanga Roka

Faculty of Law University 17 August 1945 Surabaya and Managing Partner Law Firm Victorious Indonesia