Commenters on FDA's AI Draft Most Often Asked How to Monitor Models
· Updated

The public comments on FDA's 2025 draft guidance on AI in drug development keep asking the same questions, and FDA has not answered them. The most common, from 21 of the 48 commenters whose submissions we could read, is how to monitor a model and when to revalidate it. Just as many asked what FDA expects of training data, and 20 asked how to score a model's risk. Nearly two years on, the guidance is still a draft, yet FDA has started applying it.
FDA published the draft in January 2025. It is the only FDA document that says how a sponsor should show that an AI model used in a trial can be trusted. Its docket holds 124 comments from 108 people and organizations. What the draft itself asks for, step by step, is in our walkthrough of its credibility framework.
The draft asks for a risk score without saying how to compute one
The draft makes the evidence a sponsor owes depend on a model's risk. It covers models whose output FDA will rely on for a decision about a drug's safety, effectiveness or quality (90 FR 1157, docket FDA-2024-D-4689). For each such model, the sponsor should establish its credibility for a defined use in seven steps, and the evidence required rises with the model's risk.
Risk, in the draft, depends on how much the model drives the decision and how bad a wrong decision would be. The draft does not say how to score either one. Its only clinical example is a model that sends low-risk trial participants home instead of keeping them for 24 hours of monitoring. FDA calls that model high risk, then stops without showing what its credibility assessment would contain. A sponsor therefore has no worked score to copy, which is why so many commenters asked for one.
The most common requests were monitoring, training data and risk scoring
124 comments from 108 submitters, 7 January 2025 to 6 June 202648 submitters whose asks we could read, each counted once74 of 74 ask quotes found word for word in the saved comment or letter
Open a request to see who asked for it, in their own words. The highlight marks the ask; each card links to the comment on Regulations.gov.
21 of 48 readable submitters
CommentSpecify metrics for acceptable performance thresholds and triggers for reassessment of AI models in a regulatory context.
Read the commentLetter, as posted on its websiteRecommendation 3: FDA should provide clear guidance on what constitutes a significant model change requiring revalidation or reassessment of model credibility, especially for models used in cGMP systems or quality control.
Read the commentLetter, as posted on its websiteWe ask the Agency to consider including examples of successfully and effectively implemented lifecycle monitoring programs in the final guidance, by sharing anonymized case studies of lifecycle monitoring approaches and plans it has had the opportunity to evaluate in regulatory submissions.
Read the comment
Every card is in the data (JSON), and every comment with its codes is in the CSV below.
The top requests came from trade groups, contract research organizations (CROs) and individuals alike, so they reflect a need shared across the field. Monitoring is the clearest case, because the draft's section on keeping a model reliable is written for manufacturing, in the language of ICH Q12, the international guideline on post-approval changes. Invaryant asked FDA to "specify metrics for acceptable performance thresholds and triggers for reassessment". The Consumer Healthcare Products Association (CHPA) asked what "constitutes a significant model change requiring revalidation".
The requests on data and risk ask for the same thing, a standard to work to. The American Society of Hematology wrote: "Just as clinical trials report subject demographics, AI models should disclose dataset composition". On risk, CHPA wanted "a tiered framework or decision-tree with illustrative examples", and George Evgrafov, an independent commenter, pointed out that "Different organizations may rate the same model's influence or uncertainty differently."
Two newer questions concern models the draft did not foresee. It never mentions large language models, and 12 readable commenters asked about them. Nine asked who answers for a vendor's model, since, as Evgrafov noted, vendors can change commercial models "at any time", and LUNGevity noted that a trial model is often built by a contractor "from whom methodological details may be difficult to obtain". Until FDA answers these questions, each sponsor has to answer them for itself.
Large drug companies are mostly missing from the readable record
These requests describe the questions the draft leaves open, but not what large sponsors want. Most companies and trade groups typed "see attached" and uploaded a letter, and Regulations.gov blocks automated downloads of attachments. For 57 of the 108 submitters we could not find the letter anywhere else. Of the 13 drug and biotech companies that commented, we could read only Eigen Bio and Dr. Reddy's, while AbbVie, Boehringer Ingelheim, EMD Serono, BioNTech and Otsuka filed letters that only FDA has seen. Pfizer, Roche, Novartis, Lilly, Sanofi, AstraZeneca, GSK, Merck, Johnson & Johnson and Bristol Myers Squibb did not file in their own names. For PhRMA and BIO, the largest industry groups, only a trade journalist's account was available. The counts above therefore show what smaller companies, CROs and individuals asked, and large sponsors may want something different.
FDA is already applying the draft
The open questions matter now because FDA has begun to use the draft. In April 2026 FDA asked for comments on a pilot of AI in early-phase trials, and the notice says FDA "will apply considerations previously outlined in draft guidance". For a sponsor in that program, the draft is the working standard. The notice asks about "both sponsor-developed and proprietary systems" but does not settle who answers for a vendor's model.
The only later FDA text on these questions is the FDA and EMA principles of January 2026. They ask for "scheduled monitoring and periodic re-evaluation" and for documented data provenance, but do not say how often, against what threshold, or what to report. A sponsor in the pilot will have to set those terms itself.
What a sponsor can do now
Until the guidance is final, a review team will apply the draft, so we would settle its gaps early and in writing.
- Score the model's risk yourself, write down why, and take it to an early meeting with FDA, which the draft encourages.
- Before the trial starts, decide what performance you will monitor, how often, and what result triggers revalidation.
- If a vendor built the model, agree in the contract what it will disclose to you and to FDA, including changes.
- If your letter is one of the 57 we could not read, publish it. Comments can still be filed at any time under 21 CFR 10.115(g)(5).
Data and method
We collected all 124 comments on the drug docket from Regulations.gov in November 2026. Where a comment was only an attachment, we used the letter from the organization's website if it named this guidance, which recovered 11 submitters. For eight organizations, including PhRMA and BIO, the only text was press coverage, so their requests are probably undercounted. We counted each submitter once and assigned commenter types ourselves. One person coded every comment, and a random reread of 22 found no clear errors.
Downloads: every comment with its coded requests (CSV) and counts by request (CSV).
Frequently asked questions
Has FDA finalized its AI guidance for drugs?
No. In November 2026 the January 2025 draft was still marked as a draft, and FDA's April 2026 pilot notice says it will apply the draft's considerations.
What did PhRMA say about the draft?
PhRMA's letter is an attachment we could not obtain. According to RAPS, PhRMA supported excluding AI that does not affect patient safety, drug quality or study reliability, and asked FDA to define "model".
Does the draft cover AI used to write regulatory documents?
It excludes AI used for "drafting/writing a regulatory submission" when that use does not affect patient safety, drug quality or the reliability of study results.