UpDoc ignited industry-wide punning on June 25th, after it announced it obtained FDA clearance as the “first Software as a Medical Device (SaMD) that uses patient-facing large language models,” prompting every digital technology newsletter to cheekily ask “What’s UpDoc?” The punning has died down, but the main question hasn’t: what, exactly, is UpDoc, and why won’t its CEO provide a straight answer?
Ostensibly, the AI helps clinicians “deploy clinical AI agents to complete tasks autonomously on their behalf,” through a native app-deployed AI chatbot that allows patients to input their blood glucose levels or symptoms and suggests medical action like eating sweets or changing insulin dosage. AI enablement helps UpDoc make the bold claim that its services are the “realization of Clinical AI’s potential.” That claim has commanded $18 million in oversubscribed seed funding from investors, among them Eli Lilly, Mayo Clinic, and Section 32.
An ostensibly trailblazing FDA clearance
Is the claim accurate? UpDoc’s press release makes the tech’s function very clear: “with UpDoc, licensed clinicians for the first time can deploy AI agents that are FDA-cleared and integrated within a provider’s electronic health record.” Its LinkedIn announcement is equally unambiguous: UpDoc is “the first FDA cleared patient-facing AI capable of delivering care that has historically required an encounter with a licensed clinician.” The obvious implication is that the AI itself delivers care.
If, in fact, an LLM making clinical decisions received FDA clearance for its product, it would be, as UpDoc puts it, a “historic” breakthrough in the regulation of medical AI. The FDA classifies AI devices either as a SaMD, which forces innovators to obtain premarket approval and 510(k) certification, or provides regulatory discretion to a set of devices under defined exemption criteria. Whereas the FDA defines medical devices as point solutions with scheduled outcomes – diagnosis monitoring, injury alleviation, etc. – adaptive algorithms are non-deterministic.
As such, LLM innovators struggle to scale to market within a moribund regulatory environment not built for AI, and, in turn, most seek regulatory discretion. Clinical, client-facing algorithms have rarely been developed: consider that, in Utah, Doctronic, a telehealth service which uses a native LLM to autonomously refill patient medications, obtained regulatory approval through a narrow Utah sandbox, and without communicating with the FDA.
AI decision-maker or AI interface?
So, does UpDoc’s AI model offer clinical advice to patients? Neutral observers are skeptical. In a 2023 JAMA report, the system’s designers described it as “rules based and deterministic, based on titration algorithms” (that is, not an LLM, which are necessarily non-deterministic). UpDoc’s founder Sharif Vakili clarified that the JAMA Network paper predates UpDoc’s current technology. That’s weird, since UpDoc boasts that its innovative “Remote Patient Intervention” was “conducted for the first time in patients in a Stanford clinical trial published in JAMA Network,” and that RPI improved “medical adherence by 65% and chronic disease control by 3x” in the “same Stanford MIVA trial.” That’s the MIVA trial Vakili dismissed; it’s strange, to say the least, for a digital health founder with an advanced LLM to flaunt a system he dismisses as archaic.
Vakili has clarified almost nothing: when asked whether the UpDoc’s LLM helps make treatment decisions, the founder noted that “the UpDoc is conversing with the patients with an LLM” but that “that’s probably the amount we can disclose publicly without going too much into the architecture of the system;” after researchers suggested that UpDoc’s FDA clearance indicated substantial similarities to a scheduled point solution approved in 2019, Vakili assured commentators that the technology is “way more complicated that anybody is understanding” and vowed that the “company would reveal more in the future.”
We're still waiting, but, in contrast to Vakilis' evasion, the FDA summary seems pretty clear. In their clearance summary, released last Christmas (why did UpDoc wait so long to announce it?), the FDA plainly states that no clinical testing was performed for UpDoc’s clearance. Rather, the FDA granted UpDoc approval purely through substantial equivalence to the aforementioned 2019 product, a non-LLM, rules-based calculator. Here’s the kicker: the FDA description splits UpDoc into an AI powered device, identified as the “Conversation Service,” and a deterministic, diagnostic product, identified as the “Clinical Service.” The Clinical Service is what actually computes the insulin-instructions based on HCP-defined treatment parameters, not the Conversation Service. It seems that AI is the interface, not the decision maker.
Providers need transparency
Whereas UpDoc’s oversubscribed funding is driven by its FDA approval, the excitement for its approval would be dampened if the employed LLM is simply an interface. Without insight into the machinic function of AI algorithms, providers struggle to assess the efficacy, safety, and basic function of the AI systems they employ. If UpDoc provides consultation with non-deterministic AI systems, its trailblazing regulatory approval is admirable; if it doesn't, hospitals can probably trust a system approved in 2019. But governance and trust are interpersonal and, either way, providers don’t know UpDoc’s function because Vakili has refused to elaborate. UpDoc’s strategic ambiguity seems more tailored to shelter its “regulatory strategy,” as Vakili put it, than clarify the types of results providers can expect. When founders, bolstered by $18 million in funding, refuse to disclose basic facts about their model, that shouldn’t inspire trust.
Bottom line: healthcare providers should demand pre- and post-deployment visibility into the machinic function of the products they employ. That is a luxury that has not yet been afforded: the average AI/ML-enabled devices cleared by the FDA between 1970 and December 2024 scored a transparency rating of 3.3 out of 17, a shockingly low figure. In the absence of vendor transparency, providers must seek out the technologies and third parties which can provide the structural auditing themselves.
References
[3] https://jamanetwork.com/journals/jama-health-forum/fullarticle/2846947
[4] https://www.linkedin.com/company/updocai/
[5] https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm?ID=K181916
[6] https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm?ID=K253281