AI Can Pose Safely and Still Harm Patients -By Fransiscus Nanga Roka

California has done what it must do: draw the line somewhere, because a white coat does not fit an algorithm.

Governor Gavin Newsom signed Assembly Bill 489 on October 11,2025. Law, up to October 2023: The law focuses on developers and deployers if their AI systems use protected professional terms in advertising or as part of functionality that can reasonably be construed as suggesting care, reports or assessments based on the advice of a licensed human practitioner. Particular unauthorized use shall be deemed a separate breach. Injunctions or restraining orders may be sought by professional licensing boards.

Who does AB 489 regulate? Those that develop or utilize the AI or generative-AI systems.

What does it prohibit? False signals that an AI possesses a medical license or that a licensed natural person is providing its output.

Where and when does it matter? Across California’s rapidly expanding digital-health market, whenever AI enters advertising, patient communication or healthcare functionality.

Why was it necessary? Because medical authority changes behavior. A frightened patient may treat “Dr. AI” as more credible than an ordinary chatbot, delay emergency care, accept a fabricated diagnosis or follow dangerous treatment advice.

How will California enforce it? Through the professional board or enforcement agency responsible for the falsely invoked healthcare profession.

This is important—but dangerously incomplete.

AB 489 regulates the costume of artificial authority more clearly than the substance of algorithmic medicine. An AI may stop calling itself “Doctor” and still deliver clinically reckless advice. Removing “M.D.” from a chatbot’s name does not cure hallucinations, biased training data, fabricated citations, unsafe medication recommendations or failures to recognize emergencies.

The law also does not expressly ban every humanlike design choice. Its enacted language focuses on prohibited terms, letters and phrases that imply licensure. Claims that AB 489 categorically outlaws conversational tone, medical-looking icons or every phrase such as “doctor-level” overstate the statutory text. Deception can survive through avatars, voice design, testimonials, response style and carefully engineered ambiguity without using a protected title.

California’s earlier AB 3030 requires specified healthcare facilities and practices to disclose when generative AI creates patient communications concerning clinical information and to explain how patients can reach a human. Yet it exempts communications reviewed by a licensed or certified provider. California Legislature Disclosure is useful, but a disclaimer cannot transform unreliable output into safe medicine. Nor does nominal “human review” guarantee meaningful scrutiny when clinicians face automation bias and crushing workloads.

The central accountability question therefore remains unanswered: When an AI gives harmful advice, who bears the legal burden—the developer, deployer, healthcare institution, supervising clinician or all of them?

California should now build a second regulatory wall.

First, prohibit AI from independently diagnosing, prescribing, changing medication or managing psychiatric crises unless a licensed clinician meaningfully reviews the decision. Second, require conspicuous identity disclosure throughout every interaction, not merely in fine print. Third, mandate clinical validation across age, sex, race, disability and language groups before deployment.

Fourth, compel developers and providers to preserve audit logs, disclose material model limitations, report serious incidents and notify regulators when system updates alter clinical performance. Fifth, impose joint liability when developers conceal known risks or institutions deploy systems without adequate supervision. Mandatory insurance should ensure that injured patients receive compensation even when responsibility is disputed.

Finally, California should create independent testing, whistleblower protection and a private right of action for patients deceived or harmed by medical AI.

AB 489 correctly declares that software cannot impersonate a doctor. But the greater danger is not merely an algorithm lying about who it is. It is an algorithm giving medical advice without anyone being clearly responsible when that advice destroys a life.

Fransiscus Nanga Roka

Faculty of Law University 17 August 1945 Surabaya and Managing Partner Lw Firm Victorious Indonesia