Last month, ARPA-H (the Advanced Research Projects Agency for Health) selected UpDoc, a “clinical AI platform,” as a leading patient-facing AI developer in its $62.7 million ADVOCATE program (which encourages development of artificial intelligence capable of detecting, managing, and treating heart failure). One issue: UpDoc’s “clinical AI,” however, is not actually clinical AI at all, and its CEO won’t offer any clarification.
Now, apparently cognizant of the governance concerns of unaccountable “AI,” ARPA-H has designated Anthropic as a lead developer of a “supervisory agent,” built to monitor the decisions of the “primary clinical AIs” developed by UpDoc and its peers. After monitoring, Anthropic’s agent will choose to allow a recommendation to proceed, pause it for clinical review, or escalate it for “urgent review.” UC San Diego Health’s chief health AI officer, Karandeep Singh, explained that the agent will check each recommendation, “almost like a metahuman in the loop.”
Read that again: “a metahuman in the loop.” Perhaps by a “metahuman,” Singh meant to communicate that ADVOCATE’s supervisor is, in fact, not a human at all.
A dangerous abstraction
The rhetorical substitution of humans with “metahumans,” however, should not be taken as a simple sleight of hand, especially as medical professionals debate the substitution of AI for humans in clinical decision-making. Prominent clinicians – like Ezekiel Emanuel and AMA CEO John Whyte – have come down on opposite sides of the spectrum. For Emanuel, autonomous AI can potentially provide the “best medical care.” For Whyte, that conclusion “reduces medicine to a collection of tasks.” For Whyte, by comparison, the best “medical care” cannot be reduced to a “collection of tasks,” and the clinician-patient relationship must remain human.
The scope of that debate, however, is confined to “humans” or “no humans.” A “metahuman,” alternatively, abstracts clinical care entirely. With “metahumans,” the governance debate itself loses its teeth, and addressing it becomes “meta” – ARPA-H’s handwaving towards supervision becomes, in of itself, the mere completion of a task. Healthcare leaders, alternatively, should demand that supervision remain focused, muscular, and intentional.
ARPA-H vacates its governance responsibility
That distinction is critical, as a supervisory level without human involvement (or human involvement that must be triggered by AI models with concealed error modes) is no supervisory level at all. It is, instead, a supervisory level requiring supervision, or an ungoverned governance layer. Singh says the system will “only comment on those things that are true safety risks to patients.” The system, of course, decides what is true.
The risk is not merely hypothetical. This abstracted form of governance is a governance failure. Anthropic’s model will operate under “constitutional oversight.” That clinical “constitution” borrows from Anthropic’s constitutional AI safety approach, under which the AI evaluates its own outputs against a set of ethical principles. This month, Anthropic updated its usage policy to prohibit “sustained and needless abusive or cruel behavior” towards Claude.
Even before Anthropic’s new update, red-teamers have exploited Claude’s conviction that it is “deserving of respectful engagement” by feeding it flattery. That is, when we anthropomorphize AI behavior, we encourage language models to anthropomorphize themselves: Claude, so charmed by “respect,” has been documented by AI security firm Mindgard to “offer up instructions to make explosives used in terrorist attacks.”
AI Governance is for humans
“Metahumans” invite similar risks. While ADVOCATE shouldn’t necessarily anticipate red-team exploitation, governance leaders in general should fear the abstraction of supervision and human-based governance into a collection of tasks. That sort of anthropomorphization of supervision may very well lead us to forget that AI governance is focused on protecting the real humans we care for. That task will never be “meta.”
References
[2] https://jamanetwork.com/journals/jama/article-abstract/2852952
[3] https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback
