Key Takeaways:

• Contracts and records decide much of AI liability for now. Case law on clinical AI is still thin, so the allocation of responsibility in vendor agreements and the evidence of how your organization evaluated, deployed, and monitored a tool carry real weight when something goes wrong.
• Bias audits have to test outcomes, not just inputs. A widely used care management algorithm never used race as an input and still sharply reduced the number of Black patients flagged for extra care, which is why subgroup performance testing belongs in every procurement.
• FDA clearance doesn’t prove a tool works for your patients. Few cleared AI devices report prospective or randomized evidence, and a widely deployed sepsis model performed far below its vendor’s figures in independent testing, so local validation and ongoing monitoring are the health system’s job.

Health system CIOs, CMIOs, and digital health product leads evaluating AI vendors share a governance gap that boilerplate contracts won’t close. When an algorithm makes a mistake inside a clinical workflow, who’s responsible? What counts as an adequate bias audit before deployment? How much clinical evidence turns a vendor’s demo into a defensible deployment.

The questions aren’t limited to formal procurement. In October 2026, Johns Hopkins Bayview Medical Center began notifying 3,182 patients after a staff member uploaded a spreadsheet containing their names, medications, and medical record numbers to a public AI tool the hospital hadn’t approved. Johns Hopkins’ own AI guidelines had already told staff not to do that. A policy existed, and it wasn’t enough.

This guide translates liability allocation, bias auditing, clinical validation, and post-deployment monitoring into contract terms, internal policies, and a checklist you can use before the next deployment decision lands on your desk.

What Governs Health System AI in 2026

No single regulator governs the AI a health system buys. Several frameworks overlap, and knowing which ones apply shapes the rest of your program.

The Joint Commission and the Coalition for Health AI. In September 2025, the Joint Commission and CHAI released their Guidance on the Responsible Use of AI in Healthcare, organized around seven elements: AI policies and governance structures, patient privacy and transparency, data security and data use protections, ongoing quality monitoring, voluntary and blinded reporting of AI safety events, risk and bias assessment, and education and training. The guidance is voluntary, but the Joint Commission has said a voluntary certification program will follow, and CHAI has published governance playbooks to put it into practice. It’s the closest thing the industry has to a shared standard.

Federal nondiscrimination law. The 2024 Section 1557 rule at 45 CFR 92.210 prohibits covered entities from discriminating through “patient care decision support tools,” a category that explicitly includes AI. Since May 1, 2025, covered entities have had an ongoing duty to make reasonable efforts to identify tools that use race, color, national origin, sex, age, or disability as input variables and to mitigate the risk of discrimination. HHS’s Office for Civil Rights issued a January 2025 letter suggesting practical steps, including getting input-variable information from vendors and maintaining a method for evaluating tools. The current administration has said little publicly about enforcing the provision, but it remains in the regulation, so don’t treat that quiet as a safe harbor.

FDA. FDA regulates AI that meets the definition of a medical device, and its AI-enabled device list reached 1,614 entries by mid-2026. Many tools health systems buy fall outside that scope, including most ambient documentation, operational AI, and decision support that meets FDA’s non-device criteria. For those, your governance program is the only oversight they get.

States. Colorado’s AI Act treats health care decisions as a high-risk use and puts duties on deployers, including risk management programs, impact assessments, and patient notices. Its effective date was pushed back to June 30, 2026, and its status has since been unsettled amid a federal push to preempt state AI laws. Texas, Illinois, and Utah have also passed laws that reach AI used in patient care. Check current requirements in every state where you operate.

Who’s Liable When Clinical AI Gets It Wrong

The honest answer is that the law is still forming. When Michelle Mello and Neel Guha reviewed 803 court cases involving injuries from AI and other software for their 2024 New England Journal of Medicine analysis, they found the signals from case law still faint. In one case they discuss, a court dismissed a negligence claim against a software developer in part because the developer’s licensing agreement gave physicians final decision-making responsibility. Contract terms can shape who ends up liable.

In practice, responsibility for an AI-assisted decision is shared, and a sound governance program assigns each party what it can control:

• Developers are best placed to answer for design choices, training data, performance claims, known limitations, and disclosure of updates.
• Health systems are best placed to answer for procurement diligence, local validation, governance, security, workflow integration, and training.
• Clinicians are best placed to answer for using the tool as intended, checking its output in proportion to the stakes, and recording when they follow or set aside its recommendation.

Vendor agreements often don’t reflect that balance. Licensing terms commonly disclaim warranties about output accuracy, cap damages, and assign training and post-deployment monitoring to the health system, even though those tasks depend on information only the developer has. Mello and Guha recommend that providers push back on liability disclaimers, require developers to supply the information needed for risk assessment and monitoring, seek indemnification for errors in the model, and carry adequate insurance for tools built in-house.

Documentation tools deserve a specific note. When a clinician signs a note drafted by an ambient AI tool, they attest to its content, so review before signing is the safeguard. Build time for that review into the workflow, and make sure your audit trail shows that it happened.

What to Write Into Vendor Contracts

Turn those principles into enforceable terms before you sign.

Contract term	What to require	Why it matters
Intended use	The clinical use, population, and setting the tool was built and validated for	Defines what “working correctly” means and what’s out of scope
Performance commitments	Specified metrics the tool must meet, with remedies if it falls short	Gives you a contract claim when performance degrades
Subgroup performance	Results by race, ethnicity, sex, age, payer, and other relevant groups	The basis for your bias audit and your Section 1557 diligence
Data documentation	Training data sources, time periods, demographics, and known failure modes	Shows whether the model’s population resembles yours
Validation evidence	Studies completed, where, and with what design	Places the tool on the evidence ladder described below
Change control	Advance notice of model updates, performance data after each one, and the right to delay or reject an update	Prevents silent changes to a validated tool
Revalidation	Which party funds revalidation after updates, workflow changes, or population shifts	Keeps monitoring from becoming an unfunded mandate
Logging and data access	Audit logs, outputs, and inputs available to you for monitoring	You can’t monitor what you can’t see
Incident reporting	Notice of errors, safety events, and security incidents, with cooperation on investigations	Supports patient safety and breach response
Indemnification and caps	Indemnification for model errors and liability caps proportionate to the risk	Allocates financial responsibility where control sits
Data use	A business associate agreement, limits on secondary use, and no training on your PHI without consent	Protects patients and your HIPAA posture
Exit	Return or destruction of data and continued access to records at termination	Keeps you whole if the relationship ends

If you’re expected to run local validation, the contract should define what passing performance looks like and who pays for the analysis. If the vendor won’t commit to subgroup disclosure or change notification, treat that as information about the vendor.

Questions to Ask in the Vendor Demo

• What exactly does the model predict, and how was that outcome labeled in the training data?
• Which sites, time periods, and populations were used for training and validation, and how do they compare with ours?
• What are the results by race, ethnicity, sex, age, and payer, including calibration?
• What evidence exists beyond retrospective validation?
• How will you notify us of model, data, or prompt changes, and can we test them before they reach production?
• What logs and outputs can we access for our own monitoring?
• What happens to our data, including any use for training, and which subcontractors touch it?
• What do your terms say about liability for model errors?

Bias Auditing: From Checkbox to Contract Deliverable

The most cited case of algorithmic bias in health care shows why inputs alone don’t tell the story. A 2019 study in Science examined a widely used commercial algorithm that identified patients for extra care management. It predicted health care costs as a proxy for health needs, and because less is spent on Black patients with the same level of need, it concluded they were healthier than equally sick White patients. Race wasn’t an input. Correcting the bias would have raised the share of Black patients flagged for extra help from 17.7% to 46.5%.

That’s the lesson for any audit. Section 1557 directs attention to input variables, which is a necessary first step, but bias can enter through the choice of outcome label, through missing data, or through differences in how care is documented. Test what the model actually does for each group.

Vendor documentation rarely makes that easy. A 2024 review of 692 FDA-authorized AI devices found that only 3.6% reported the race or ethnicity of study participants, 99.1% gave no socioeconomic data, and 81.6% didn’t report participants’ ages.

How to Structure a Pre-Deployment Bias Audit

1. Confirm the outcome. Check that the model predicts what you actually care about, not a convenient proxy for it.
2. Set thresholds before the demo. Your governance committee should decide in advance what performance gap between subgroups is acceptable. If a model reaches 92% sensitivity for White patients and 78% for Black patients, that gap should block deployment until it’s resolved.
3. Measure by subgroup. Compare sensitivity, specificity, positive predictive value, and calibration across race, ethnicity, sex, age, payer, preferred language, and disability status where your data allow.
4. Look at missing data. Groups with less documentation, fewer visits, or different coding patterns can look healthier to a model than they are.
5. Run it on your own data. If the vendor can’t or won’t supply subgroup results, test a representative local sample in silent mode before go-live.
6. Decide and document. Fairness metrics can conflict, so you can’t optimize them all at once. Record which ones you chose, why, and what mitigation you applied, from adjusted thresholds to added workflow safeguards to not deploying at all.

Clinical Validation: How Much Evidence Is Enough

AI evidence tends to stop early. Many models are reported with strong discrimination on retrospective data and little on calibration, performance at other sites, or effect on patient outcomes. A practical way to evaluate a vendor is to ask where its tool sits on an evidence ladder:

Rung	What it shows	What it supports
Internal validation	Performance on held-out data from the development site	Technical feasibility
External validation	Performance at different sites, populations, or time periods	Generalizability
Local silent deployment	Performance on your patients and data, without affecting care	Readiness for your setting
Prospective evaluation	Effect on decisions and outcomes when used in care	Clinical value
Post-deployment monitoring	Sustained performance after go-live	Continued safety

Your contract should state which rungs the tool has cleared and who funds the next one.

External validation matters because performance often drops outside the development setting. When Michigan Medicine independently tested the Epic Sepsis Model across 38,455 hospitalizations, it found an area under the curve of 0.63, well below the 0.76 to 0.83 the vendor had reported. The model missed 67% of patients with sepsis while generating alerts on 18% of all hospitalized patients. Hundreds of hospitals had adopted it before that independent check.

The evidence behind FDA-cleared tools is often thin too. In a 2025 JAMA Health Forum analysis of 691 cleared AI devices, only 1.6% reported data from randomized trials and 7.7% from prospective studies. A separate study of 950 AI devices found that those without reported clinical validation were nearly twice as likely to be recalled, with recalls concentrated soon after clearance.

Before deployment, run the tool in silent mode on your own patients, compare its output with current practice, and measure calibration, alert burden, and the workflow it will actually live in. A model validated at an academic medical center can behave differently in a community hospital with a different patient mix, documentation style, or EHR configuration.

Post-Deployment Monitoring: The Step Vendors Often Skip

Validation describes a model at one moment, and models change. Patient populations shift, documentation practices evolve, scanners and lab assays get replaced, EHR upgrades alter data feeds, and vendors push updates. Few AI products arrive with a plan for any of it: the 2024 FDA device review found that only 9.0% included a prospective postmarket surveillance study.

A monitoring program needs owners, measures, a cadence, and triggers.

• Owners. Name the people responsible for each deployed tool, on both the clinical and technical side.
• Measures. Track performance against outcomes where you can observe them, calibration, alert volume, false positive rates, clinician override frequency, subgroup performance, and shifts in the input data.
• Cadence. Match it to risk. A model that influences diagnosis or triage warrants at least quarterly review and automated alerts between reviews, while a low-risk operational tool may need less.
• Triggers. Any vendor update, workflow redesign, EHR upgrade, device replacement, or measurable population shift should prompt revalidation, and a threshold breach should pause the tool until you’ve investigated.

The Joint Commission and CHAI guidance pairs ongoing quality monitoring with voluntary, blinded reporting of AI safety events, so incidents at one organization can inform others. Build an internal pathway for clinicians to report AI-related errors and near misses, and route them to the governance committee.

Governing Generative AI and Ambient Documentation

Generative tools need a few extra controls. Their output varies from run to run, their errors read as plausible, and the underlying model can change while the product name stays the same.

• Define the failure modes you’ll measure. For ambient documentation, that means fabricated content, omitted content, and content attributed to the wrong speaker or patient. Sample signed notes regularly and have clinicians score them against the source encounter.
• Treat model and prompt changes as material. A vendor switching foundation models, or changing the instructions that shape output, can alter behavior as much as retraining does. Require notice, and rerun your evaluation.
• Keep people on the consequential steps. Patient-facing messages, orders, and anything that changes care should pass through a person, with the review recorded.
• Watch for automation complacency. As clinicians come to trust a tool, edit rates fall. That can mean the tool improved or that review got lighter, so pair the metric with periodic audits.
• Trace the data path. Know where audio, transcripts, and prompts go, how long they’re kept, and whether the vendor or its model provider trains on them, and make sure your BAAs cover every party that touches them.

Shadow AI: The Governance Gap Outside Procurement

The Johns Hopkins Bayview incident didn’t involve a vendor contract at all. A staff member used a public AI tool during an internal review, and the hospital then had to ask the tool’s operator to delete the data, without public confirmation that it did. It’s not an isolated case: Loma Linda University Health reported in May 2026 that a file with limited patient information had been uploaded to an outside AI platform during a research study.

Policies are necessary but, as Hopkins showed, not sufficient. Pair them with:

• Sanctioned alternatives. Give staff approved AI tools covered by business associate agreements, so the fastest option is also the compliant one.
• Technical controls. Use data loss prevention and network controls to block or warn on uploads of PHI to unapproved AI services.
• A maintained list of approved tools, with clear guidance on what data each may receive.
• Training that uses real scenarios, like pasting a patient list into a chatbot to summarize it.
• An incident playbook for AI-related disclosures, covering containment, vendor deletion requests, HIPAA breach assessment, and patient notification.
• A reporting culture that encourages staff to flag their own mistakes quickly.

Dogtown’s HIPAA-compliant app development work covers the technical side of keeping PHI inside sanctioned systems.

Building Internal Accountability Structures

Oversight should scale with risk. A simple tiering scheme keeps low-risk tools moving while concentrating scrutiny where errors can reach patients.

Tier	Examples	Minimum oversight
Administrative	Scheduling, staff-reviewed coding suggestions, internal search	Approved-tool list, BAA where PHI is involved, periodic spot checks
Operational	Bed management, staffing forecasts, chart summaries for internal review	Data use review, local performance check, monitoring of key measures
Clinical decision support	Risk scores, alerts, triage suggestions, ambient documentation	Bias audit, silent local validation, at least quarterly monitoring, clinician training
High-impact clinical	Diagnostic outputs acted on directly, autonomous or near-autonomous functions	Everything above, plus prospective evaluation, hard stops when the model is uncertain, and committee review of every material change

Governance needs a body with real authority. Form a multidisciplinary AI governance committee that can approve, suspend, or retire any AI tool, with members from clinical informatics, legal, compliance, risk management, IT security, data science, nursing, and the frontline specialties affected. Give it these working parts:

• An AI inventory. Every tool in use, its owner, intended use, risk tier, validation status, and monitoring plan, including AI features embedded in EHR and vendor platforms.
• Risk tiers. Each tool is assigned a tier from the table above, which sets its reviews, its monitoring cadence, and whether it needs second reads or hard stops.
• Documented decisions. Who approved each tool, on what evidence, and with what conditions, which is also the record you’ll want if a deployment is ever challenged.
• Training and credentialing. If AI influences diagnosis, triage, or referral, AI literacy is a patient safety requirement for the people using it.
• Patient transparency. The Joint Commission and CHAI guidance recommends informing patients when AI directly affects their care, and some state laws now require it.

Keep the policy itself principles-based. AI changes faster than detailed rules can, so a clear set of commitments and a dependable review process will hold up better than a long list of specific prohibitions.

What a Governance Review Looks Like in Practice

Here’s how a hypothetical inpatient deterioration alert might move through a governance program.

Step	What happens	Record it leaves
Intake	A nursing leader requests a vendor’s deterioration model for medical-surgical units	Inventory entry with an owner and intended use
Tiering	The committee classifies it as clinical decision support	Tier assignment and required reviews
Diligence	The vendor supplies subgroup results, training data documentation, and validation studies	Evidence file with gaps noted
Contract	Change notice, revalidation funding, logging access, and incident reporting are negotiated	Signed terms and BAA
Bias audit	Local data show lower sensitivity for patients with limited English proficiency	Mitigation plan, such as an adjusted threshold and added nurse review for that group
Silent validation	The model runs for 90 days without alerting and is scored against actual deterioration events	Calibration, alert burden, and subgroup results
Go-live	The committee approves with conditions, including a monthly alert-volume review	Approval record with conditions
Monitoring	Alert volume jumps after an EHR upgrade changes a vital-signs feed	Tool paused, feed fixed, model revalidated, incident logged

Every step leaves a record. That record is what turns governance from a policy into evidence you can show a regulator, an accreditor, or a court.

A Governance Checklist for Your Next AI Deployment

• Intended use defined. The clinical use, population, and setting are documented and match the vendor’s validation.
• Regulatory status known. You know whether the tool is an FDA-regulated device and which state laws apply.
• Contract terms secured. Subgroup disclosure, change notification, revalidation funding, logging access, incident reporting, and indemnification are in writing.
• BAA in place. Data use limits cover any PHI the tool touches.
• Bias audit passed. Subgroup performance meets thresholds set before the demo.
• Local validation complete. Silent-mode results on your data support go-live.
• Monitoring assigned. Owners, measures, cadence, and triggers are documented.
• Clinicians trained. Users know the tool’s limits and how to report problems.
• Patients informed. Disclosure practices meet your policy and applicable state law.
• Committee approval recorded. The decision, evidence, and conditions are on file.

Why Partner With Dogtown Media

Dogtown Media builds AI-powered healthcare applications with governance built in: subgroup evaluation, audit logging, model documentation, monitoring dashboards, and change control that health systems increasingly require before they’ll deploy. Our AI compliance and risk mitigation practice helps teams plan validation and monitoring from the first architecture decision, and our AI-powered healthcare app development team builds the clinical decision support, remote monitoring, and workflow tools that have to pass those reviews.

If you’re preparing for a deployment decision or building an AI product health systems will scrutinize, request a free consultation to map your governance requirements.

Frequently Asked Questions

What’s the difference between internal and external validation in clinical AI?

Internal validation checks a model against data held back from its own development set, so the patients come from the same institutions and time period. External validation tests it on data from different sites, populations, or time periods. A model can perform well internally and poorly externally if its training data don’t represent other settings, as the Epic Sepsis Model’s independent evaluation showed. Ask for external validation evidence, then confirm performance on your own patients in silent mode before go-live.

Can we rely on FDA clearance as proof that an AI tool works for our patients?

No. Clearance means the device met FDA’s requirements for its submission, and most AI devices are cleared through 510(k), which often doesn’t require prospective clinical studies. Only 1.6% of the cleared AI devices in one 2025 analysis reported randomized trial data, and devices without reported clinical validation were nearly twice as likely to be recalled in another. Many AI tools aren’t FDA-regulated at all. Local validation and ongoing monitoring remain your responsibility either way.

Who’s responsible if an AI tool drafts a clinical note with an error and a physician signs it?

The physician who signs the note attests to its content, so the standard for clinical documentation hasn’t changed. The health system can share exposure through how it selected, configured, and trained staff on the tool, and vendor terms typically disclaim responsibility for output accuracy. That’s why review time belongs in the workflow, why the audit trail should show review happened, and why contracts should address the vendor’s obligations for known failure modes.

How often should we revalidate a deployed AI model?

Base it on risk. Models that influence diagnosis, triage, or treatment warrant at least quarterly performance review, with automated alerts between reviews. Any material change should trigger revalidation: a vendor update, a workflow redesign, an EHR upgrade, a device or assay replacement, or a measurable shift in your patient population. A breach of a predefined threshold, such as a spike in alert volume or a widening subgroup gap, should pause the tool until it’s been investigated.

What should we do if a vendor won’t provide subgroup performance data?

Treat it as a serious warning sign. Under Section 1557, covered entities have a duty to make reasonable efforts to identify discrimination risk in clinical decision support tools, and HHS’s guidance points to getting input-variable information from vendors as a starting point. If a vendor won’t share subgroup results, run your own audit on a representative sample of your data before deploying, or make vendor completion of one a condition of the contract. A vendor unwilling to show how its model performs across your patients isn’t ready for clinical accountability.