When Medical AI Kills, Who Enters the Dock? -By Fransiscus Nanga Roka

Recommendation diagnosing, writing a medical record and eventually treatment is the capability of Artificial Intelligence. Yet when an error by the machine disables or kills a patient, the machine cannot be questioned and imprisoned, nor morally tried. The law, thus, has to resolve a combustible query: Who should be put in the dock- the doctor, hospital, developer, data vendor or company executive?

First, one factual correction matters. That commonly referenced 69% number does not show that AI scribes do all of their work in the same time 6.9 out of ten primary-care clinics. Doctors’ recording time is 69.1% shorter during simulated consultations facilitated by a 2026 study. Data on actual adoption show close to 40% of British and Australian general practitioners are currently using it. Hype hides a more serious problem: around the world, implementation is racing ahead of accountability in AI adoption.

What is happening? Increasingly, clinical decisions and medical records are determined using diagnostic algorithms and ambient AI scribes. They might leave out allergy data, create symptoms from whole cloth, skew conversations or spit back convincing but inaccurate findings. It is especially problematic because one hallucination can silently infect an entire treatment chain, as later physicians often depend on those records.

Who is legally responsible? With assistive AI, physicians mostly still must evaluate the patient independently. If a doctor were to blindly rely and it turns out that as a result would constitute malpractice because the reasonably competent doctor would have questioned an implausible output. Automation bias may provide some common sense justification for humans but it is not a basis for avoiding legal liability.

However, the characterization of AI as “merely a tool” is rapidly becoming an industry alibi. A stethoscope should not retrain itself, hide its reasoning or behave differently between racial groups. Lethal levels of harmful teachings from defective design, biased training data and inadequate warnings, unsafe updates or known performance decline must hold developers, deployers and healthcare institutions responsible when harm occurs due to such deficiencies.

But hospitals should also be responsible if they buy poorly validated systems, provide minimal or no training, create workloads in which human review is fictional or disregard warnings after deployment. This so-called “human-in-the-loop” is a fake one, as hundreds of machine-generated outputs simly get rubber-stamped by some doctor in time-pressured circumstances.

When and where does liability develop? It can come up during development, licensing, purchasing, clinical use and deployment to the clinic or after (in software updates during care) monitoring. The FDA already has an inventory of approved AI-powered medical devices, and the European Union categorizes many medical-device AI systems as high-risk requiring risk management, documentation & direct human oversight. But authorization is not the same as immunity: regulatory approval may protect from liability for development but not negligent design or unsafe use.

Existing Malpractice Law Is Not Readily Adequate Because causation is distributed. One company provides the data, another builds the model, one selects the system that qualifies for use by the hospital, another vendor alters the algorithm remotely and finally a doctor sees his patient. We end up with a game of blame amongst all the defendants and an injured patient left looking at a black box which is protected by trade secret status.

Criminal liability must remain fault-based. AI possesses no mens rea. Prosecutors ought to pursue those who knowingly conceal defects, deploy unvalidated systems, make misleading performance claims or keep using an unsafe product after serious safety warnings: human or corporate recklessness. An adverse event in itself should never become automatic criminal liability for a doctor.

The answer is pro-rata, accountable accountability. Two specific requirements that should be included are informed patient consent, immutable audit logs, transparently identifying versions used (including version changes to the models), representative clinical validation and constant identification of bias in such systems. Effective access to model documentation and incident records must be provided to patients and courts. Negligent clinical reliance triggers physician liability; negligent deployment, hospital accountability; defective systems and concealed risks warrant developer culpability.

Lastly, mandatory AI-malpractice insurance and no-fault compensation should protect (penalize) the patients whenever the ‘causation’ divorce is impossible.

Medical AI must not create the new accountability laundering in which corporations capture the profit, clinicians absorb the blame, and patients carry the damage. No one should be allowed to hide behind a machine if no one can explain its diagnosis.

Fransiscus Nanga Roka

Faculty of Law University 17 August 1945 Surabaya and Managing Partner Law Firm Victorious Indonesia