Key Takeaways:

• On-device inference takes the cloud provider out of the PHI chain. HHS treats a cloud provider that processes or stores ePHI as a business associate even when it can’t decrypt the data, so keeping inference on the device is the cleanest way to avoid that relationship, provided analytics, logs, and backups don’t send the data out anyway.
• Built-in models changed the build. Apple’s Foundation Models framework and Google’s Gemini Nano give apps free, offline language models, and open models like Gemma 4 and Phi-4-mini fill the gaps, but device support is uneven, so every patient-facing feature needs a fallback.
• Where a model runs doesn’t change its FDA status. Intended use decides whether a feature is a medical device, and the first FDA clearance of a patient-facing LLM device points to the pattern that works: a narrow task, the model as interface, and deterministic logic making the clinical decision.

If you’re a digital health product lead evaluating AI features for a patient-facing app, or a health system innovation director weighing clinical documentation tools, the question is less whether to use large language models than where to run them. Sending protected health information to a third-party cloud API brings business associate agreements, risk analyses, and the reality that every conversation leaves your institutional boundary. Running the model on the user’s phone or tablet keeps that data local.

The engineering challenge is fitting a capable language model on a phone without degrading its output to the point where it stops being useful. That challenge looks different in 2026 than it did even a year ago. Apple and Google now ship language models inside their operating systems, open small models have improved sharply, and FDA has cleared its first device built around a patient-facing LLM.

This guide covers why on-device deployment matters, how it compares with private cloud and vendor APIs, which models and techniques make it practical, where it pays off in healthcare, how data can still leak, what regulators care about, and how to build it.

Why On-Device Deployment Matters for Healthcare AI

HIPAA permits covered entities to use cloud services for electronic protected health information, provided they sign a business associate agreement with the provider and include the service in their risk analysis. HHS’s cloud computing guidance sets a high bar for avoiding that relationship: a cloud provider that creates, receives, maintains, or transmits ePHI is a business associate even if the data is encrypted and the provider holds no decryption key. Encrypting data before it reaches a cloud model doesn’t remove the need for a BAA.

On-device architecture changes the posture. If the model runs entirely on the patient’s phone or the clinician’s tablet and no PHI leaves the device, there’s no transmission to a model provider and no business associate relationship with that provider for the inference. The data never lands in a server log, never crosses a network boundary, and never sits in a vendor’s environment.

The same logic matters outside HIPAA. Many consumer health apps aren’t offered on behalf of a covered entity, so HIPAA doesn’t apply to them, but the FTC’s Health Breach Notification Rule does. Since July 29, 2024, the rule has explicitly covered health apps, and its definition of a breach includes unauthorized disclosures, such as sharing health data with an advertising or analytics vendor without the user’s authorization. State laws such as Washington’s My Health My Data Act add their own consent requirements. Processing on the device keeps sensitive inputs, and the model’s health-related inferences, out of reach of those disclosure risks.

The Architecture Spectrum: Device, Private Cloud, or Vendor API

On-device is one point on a spectrum, and many products combine more than one.

Factor	On-device	Self-hosted (your cloud or servers)	Vendor API under a BAA
Where PHI goes	Stays on the device	Your controlled environment	The vendor’s environment
BAA with a model provider	Not needed for inference	Needed with your cloud host if hosted	Required
Model capability	Small models, roughly 1 to 4 billion parameters	Open models of any size your hardware supports	Frontier models
Works offline	Yes	No	No
Per-query cost	None after build	Infrastructure	Usage fees
Model updates	Ship with app or OS releases	You control	Vendor controls
Best fit	Sensitive, narrow, latency-critical or offline tasks	Larger models with full data control	Complex reasoning across many data sources

Self-hosting sits in the middle. Running an open model on your own infrastructure keeps data inside your environment while allowing larger models than a phone can hold. In a January 2026 preprint, OpenAI’s open-weight gpt-oss models, which run on local workstations or servers, performed competitively with leading proprietary models on clinical decision support tasks.

Hybrid designs combine the tiers. An app can handle routine, sensitive tasks on the device and send harder requests to a self-hosted or BAA-covered model, with the user’s consent and, where possible, after removing identifiers locally. Local de-identification has strong evidence behind it. The LLM-Anonymizer, published in NEJM AI in 2025, removed 98.05% of identifying characters from 250 clinical letters using Llama-3 70B on local hardware, and the 2026 Multimodal Anonymizer reached 98.80% patient-level de-identification sensitivity with a local open model. Those systems run on servers, not phones, but the pattern of scrubbing data before it leaves a trusted environment applies at any tier.

The On-Device Options Have Changed

Until recently, on-device meant bundling an open model into your app. Now the operating system may already include one.

Apple’s Foundation Models framework. Since iOS 26, any app can call the on-device model behind Apple Intelligence through Apple’s Foundation Models framework. It’s a roughly 3-billion-parameter model quantized to 2 bits, it runs offline, and there are no per-token charges. Swift-native guided generation constrains output to typed structures, and tool calling lets the model fetch data from your app. Apple is clear about the limits: it’s a device-scale model meant for tasks like summarization, extraction, and short generation, not a general-knowledge chatbot, and in Apple’s own evaluations it performs competitively with open models such as Qwen-3-4B and Gemma-3-4B. It requires Apple Intelligence-capable hardware, and Apple’s larger server model isn’t available to third-party apps.

Google’s Gemini Nano. On Android, the ML Kit GenAI APIs expose Gemini Nano through AICore, a system service that runs the model entirely on the device and shares it across apps. Task-specific APIs cover summarization, proofreading, rewriting, and image description, and a Prompt API handles custom prompts. Support is limited to specific device models, and Google maintains the current list. Gemini Nano 4, announced at Google I/O 2026, needs at least 12 GB of RAM and a flagship chip, and one analysis estimates it reaches only 1% to 3% of active Android devices.

Open models you bundle yourself. When you need control over the model, cross-platform consistency, or medical fine-tuning, you ship your own. Google’s Gemma 4, released in April 2026, includes E2B and E4B models built for phones and moved to the Apache 2.0 license. Microsoft’s Phi-4-mini (3.8 billion parameters, MIT license) remains its current small model, Alibaba’s Qwen3.5 series spans 0.8 to 9 billion parameters, and Meta’s Llama 3.2 1B and 3B are still its smallest models, according to a September 2026 comparison. Check each license and acceptable use policy before building a clinical feature on it.

Device coverage is an equity question. Built-in models run only on recent hardware, and bundled models need enough memory to load. Patients on older or budget phones may not be able to run either. Detect device capability at runtime, and give every AI feature a fallback, whether that’s a BAA-covered server path the user opts into or a non-AI version of the feature, so the people with the least expensive phones aren’t the ones locked out.

Model Compression Techniques That Make Mobile Deployment Practical

Size is the core constraint. A 7-billion-parameter model stored at 32-bit precision needs about 28 GB of memory, and even at 16-bit precision it needs about 14 GB. Phones have a fraction of that, shared with the operating system and every other app, and the model also needs working memory for its context, which grows with prompt length.

Quantization reduces the numerical precision of the model’s weights. Moving from 16-bit to 4-bit precision cuts the weights to roughly a quarter of their size, so a 3-billion-parameter model fits in about 1.5 to 2 GB. Apple goes further, quantizing its on-device model to 2 bits. Lower precision can degrade performance on complex reasoning, so measure the quantized model against the full-precision version on your own task before committing.

Knowledge distillation trains a small model to imitate a larger one. NYU Langone’s MedMobile fine-tuned the 3.8-billion-parameter Phi-3-mini on medical data, including GPT-4’s reasoning, and scored 75.7% on USMLE-style MedQA questions, above the roughly 60% passing mark for physicians.

Medical fine-tuning often matters more than size. In the Stanford-led Medicine on the Edge benchmark, which tested on-device models on clinical cases on iPhones and iPads, medically fine-tuned models such as Med42 and Aloe reached the highest accuracy, while the general-purpose Phi-3 Mini struck the best balance of speed and accuracy. The researchers also found that older devices can run these models, with memory a bigger constraint than processing power.

Pruning removes weights that contribute little to the output, and it’s sometimes combined with quantization for further savings, at the cost of more validation work.

Once compressed, the model needs a runtime. Common choices include GGUF files for llama.cpp, ONNX Runtime, Apple’s Core ML and MLX, and Google’s AI Edge tooling, and the right one depends on your platforms and whether you need a single model across iOS and Android.

Performance Trade-Offs: Latency, Accuracy, Battery, and Offline Operation

A clinical tool that makes a clinician wait won’t get used, so latency has to fit the moment of use. On a phone, speed depends on the chip, available memory, hardware acceleration, context length, and heat. Sustained inference warms the device, and thermal throttling slows it down, which is why the Medicine on the Edge study tracked thermal state alongside accuracy. Test on the slowest device you intend to support, during long sessions, not only on a cool flagship.

Accuracy is the harder trade-off. Small models handle bounded tasks well: summarizing a discharge instruction, extracting fields from a form, or explaining when to take a medication based on a vetted source. They’re less reliable for open-ended clinical reasoning, such as building a differential diagnosis or interpreting ambiguous results. Apple’s guidance for its own model makes the same point, recommending that developers break work into smaller, well-defined tasks. Design the feature around what the model does well, and route the rest elsewhere.

Battery matters too. Inference is computationally intensive, so continuous background processing or long conversational sessions drain power quickly. Running inference on the device’s neural or graphics processors, batching work, and letting users control when the model runs all help.

The advantage of offline operation is unambiguous. A patient-reported outcomes app in a rural area, a paramedic’s protocol reference, or a home health visit without signal can’t depend on an API call. Dogtown Media saw this firsthand with the cardiovascular emergency protocols app it built with Minneapolis Heart Institute and Allina Health. The protocol data was hard-coded for offline use so rural physicians could reach it without a connection, and an app aimed at a few hundred physicians has been downloaded more than 4,000 times in seven countries. In one case, a rural physician who couldn’t find a paper protocol at their hospital used the app to start treatment and transfer the patient for emergency surgery.

Healthcare Use Cases Where Local Deployment Justifies the Engineering Cost

Clinical documentation and note assistance. Most ambient documentation tools send audio or text to the cloud. An on-device version pairs local speech recognition with a local model that drafts the note on the clinician’s device and sends nothing out until the clinician saves it to the EHR. Google’s ML Kit already includes an on-device speech recognition API on its newest Pixel devices. The open question is quality: a small model’s draft has to meet the documentation standard for billing and compliance, so clinician review stays in the workflow.

Patient education and symptom guidance. Apps that answer questions about a condition, medication, or care plan can run locally, interpreting the question alongside data from a connected wearable without sending that data anywhere. Ground answers in vetted content stored on the device, and use structured output to keep responses within approved boundaries.

Offline clinical protocols. An LLM layer could make an offline protocol app like the Minneapolis Heart Institute app searchable in plain language. Keep the vetted protocol set as the source of truth: retrieve the relevant passage, show it with its source, and let deterministic logic, never the model, calculate any dose.

Mental health support. Mental health apps handle some of the most sensitive conversations in healthcare, so keeping them on the device has obvious appeal. It also raises a duty-of-care question. If a local model detects signs of crisis, the app still has to connect the person to help, such as the 988 Suicide and Crisis Lifeline, without depending on a server. Several states, including Illinois, now restrict AI-delivered therapy, and FDA’s fiscal 2027 guidance agenda includes a planned draft on evidence for generative AI conversational devices for mental disorders. Dogtown’s guide to mental health app development covers those requirements.

De-identification at the edge. In a hybrid app, a local model can strip identifiers from a document or conversation before anything goes to a cloud model, narrowing what the cloud ever sees.

“On-Device” Doesn’t Mean Nothing Leaves the Device

Running the model locally only protects data if the rest of the app does too. These are the common leak paths:

• Analytics and telemetry SDKs that capture screen content, events, or model outputs. For consumer apps, sending health inferences to an analytics vendor without authorization can be a breach under the FTC rule.
• Crash reports and logs that include prompts, responses, or user data in their payloads.
• Cloud backups of app data, which can carry stored conversations to the platform’s backup service unless you exclude or encrypt them.
• Push notifications that put health details on a lock screen or route them through a notification service.
• Sync features that quietly move conversation history to a backend.
• The model update channel, which should verify that downloaded model files are authentic before loading them.

Device security carries the weight that server security used to. Encrypt stored inputs and outputs with the platform’s file protection, keep keys in the Keychain or Keystore, require biometric or passcode authentication for sensitive features, and decide how the app behaves on jailbroken or rooted devices. When a covered entity deploys the app, ePHI on the device is still subject to the Security Rule. Encryption is an addressable specification today, and HHS’s proposal to make it mandatory remained unfinalized as of mid-2026, but encrypting to HHS’s standards also means a lost device doesn’t count as a breach of unsecured PHI. Dogtown’s HIPAA-compliant app development work builds these controls in from the start.

What a Privacy-Preserving Feature Looks Like

Here’s how a hypothetical medication-question feature in a patient app might run entirely on the device.

Step	What happens	Where the data goes
Question	The patient asks whether they can take their blood pressure pill with grapefruit juice	Stays on the device
Scope check	The on-device model classifies the question as medication guidance within the feature’s approved scope	Stays on the device
Retrieval	The app pulls the relevant passage from a vetted, versioned medication guide stored locally	Stays on the device
Answer	The model writes a short answer in a structured format, citing the guide’s section and version	Stays on the device
Out-of-scope path	Questions about changing a dose or new symptoms trigger a fixed response that routes the patient to their care team, with no model-generated advice	Stays on the device unless the patient chooses to message the care team
Storage	The conversation is saved encrypted and excluded from cloud backups	Stays on the device
Measurement	With the patient’s opt-in, the app reports anonymous counts, such as how often questions fall out of scope	Aggregate metrics only

The model never makes a clinical decision on its own. It classifies, retrieves, and phrases, while fixed rules handle anything that could change care, a division of labor similar to the one analysts describe in FDA’s first LLM clearance.

Regulatory Considerations for On-Device Medical AI

Where a model runs doesn’t decide whether it’s regulated. FDA looks at intended use: a feature that informs diagnosis or treatment can be a device whether it runs in the cloud or on a phone, and a feature that stays within general wellness or meets the criteria for non-device clinical decision support generally isn’t. FDA updated both of those guidances in January 2026.

FDA’s AI-enabled device list reached 1,614 entries by mid-2026, about 76% of them in radiology, and the agency says it plans to identify devices with LLM-based functionality in future updates. The first clearance of a device built around a patient-facing LLM came in December 2025: UpDoc, prescription software for insulin management in adults with type 2 diabetes, cleared through a 510(k) with a dose-calculator predicate.

Analysts at Innolitics described the clearance as a cage built around the model: what FDA cleared is a dose calculator, and the LLM’s role appears to be conversation and fact extraction around it. That’s the right pattern for small on-device models too, since they’re strongest at bounded language tasks and weakest at open-ended clinical reasoning.

On-device deployment does change how you manage the device after launch:

• Updates. A bundled model is frozen at the version in your app, and new versions reach users only through app releases they install. If the feature is a device, a predetermined change control plan lets you describe planned model updates up front, and you’ll want a way to enforce a minimum app version when a fix matters.
• Built-in models. Apple and Google update their system models with operating system releases, on their own schedules. A regulated feature that depends on a model you don’t control needs re-validation each time it changes, which often makes a bundled model the safer choice for device functions.
• Postmarket monitoring. Without server-side logs, you’ll need privacy-preserving telemetry, such as opt-in, aggregate error reporting, to learn how the model performs in the field.
• Cybersecurity. An app that connects to the internet can still meet FDA’s cyber device definition, and the model runtime belongs in your software bill of materials.

FDA’s guidance agenda for fiscal 2027 puts finalizing its AI lifecycle guidance at the top of the list, so expect more specific expectations for AI-enabled devices within the year.

How to Decide Between On-Device and Cloud-Based AI

Work through these questions in order.

What data goes in? If the input contains PHI or sensitive consumer health data and the task doesn’t need external processing, on-device deserves a serious look. If the model needs lab results from one system, imaging from another, and medication history from a third, a server architecture is usually more practical.

How hard is the task? Bounded tasks such as summarizing, extracting, classifying, and explaining vetted content suit small models. Tasks that need frontier-level reasoning don’t fit on a phone.

Does it need to work offline? If connectivity is unreliable or latency would break the experience, local inference may be the only option.

Which devices do your users carry? If many users have older or budget phones, plan the fallback before committing to an on-device design.

What does it cost at your volume? Cloud inference costs scale with usage. Built-in models carry no per-query fees, and bundled models shift compute to the user’s device, at the cost of engineering effort, app size, and battery.

How often will the model change? If clinical content or model behavior needs frequent updates, keep the changing knowledge in an updatable local store the model reads from, or choose a server architecture you control.

How to Evaluate a Small Model for Clinical Use

Public benchmarks such as MedQA say little about how a model will handle your feature. Build an evaluation set from the questions or documents your users will actually produce, with clinician-approved reference outputs, and score every candidate model and quantization level against it.

Measure more than accuracy. Track how often the model states something its source doesn’t support, whether it correctly declines or escalates out-of-scope requests, whether its structured output parses every time, and how its answers shift between the full-precision and compressed versions. Include adversarial cases too: requests to change a dose, attempts to get the model to ignore its instructions, and inputs in every language your users speak.

Run the evaluation on real devices across the range you support, since memory pressure and thermal throttling can change behavior. Then keep it as a regression suite. Rerun it before every app release, and for built-in models, after every operating system update that changes the system model.

Implementation Path

1. Define a narrow task and an evaluation set. Write down exactly what the model should do and build a test set of real-world inputs with expected outputs, reviewed by clinicians where clinical content is involved.
2. Try the built-in models first. On Apple and Android devices that support them, the system models cost nothing to ship and stay off your app’s download size.
3. Choose an open model if you need one. Start with small models such as Gemma 4 E2B or E4B, Phi-4-mini, Qwen3.5, or Llama 3.2, and review each license.
4. Fine-tune if the base model falls short. Parameter-efficient methods such as LoRA keep fine-tuning affordable.
5. Compress and compare. Quantize, then run your evaluation set against the full-precision model to confirm the accuracy you’re keeping.
6. Package for your runtime. Pick GGUF, ONNX, Core ML, MLX, or Google’s AI Edge formats based on your platforms.
7. Test on the devices your users actually own. Measure memory, latency, heat, and battery during long sessions on your lowest supported device.
8. Close the leak paths. Audit analytics, logs, backups, notifications, and sync, and encrypt everything the app stores.
9. Build capability detection and a fallback. Decide what happens on devices that can’t run the model.
10. Pilot with real users and clinicians, measuring whether the experience fits the workflow along with whether the outputs are accurate.

An On-Device AI Readiness Checklist

Before launch, confirm each of these is in place.

• Task scoped. The feature does one bounded job, with deterministic logic handling any calculation that drives care.
• Evaluation passed. The compressed model meets your accuracy bar on a clinician-reviewed test set.
• Devices covered. Capability detection works, and unsupported devices get a defined fallback.
• Data flows audited. No PHI or health inferences reach analytics, crash reports, backups, or notifications.
• Storage encrypted. Inputs and outputs are encrypted at rest, with keys in the platform keystore.
• Access protected. Sensitive features require biometric or passcode authentication.
• Model integrity verified. Model files are checked before they load.
• Regulatory status decided. You know whether the feature is a device, wellness, or non-device decision support, with a change control plan if it’s a device.
• Monitoring planned. Opt-in, privacy-preserving telemetry shows how the model performs in the field.
• Escalation designed. Safety-critical signals, such as a mental health crisis, reach help without depending on a server.

Why Partner With Dogtown Media

Dogtown Media has built healthcare apps that work where connectivity doesn’t, from the offline cardiovascular protocols app trusted by physicians in seven countries to HIPAA-compliant patient apps across iOS and Android. Our AI-powered healthcare app development team evaluates built-in and open models, compresses and validates them on real devices, and designs the fallback paths patients on older phones need. For features that may be medical devices, our AI compliance and risk mitigation practice plans the regulatory path alongside the architecture.

If you’re weighing on-device AI for a healthcare app, request a free consultation to map your use case to the right architecture.

Frequently Asked Questions

What model sizes are practical for on-device deployment on current smartphones?

Models in the 1 to 4 billion parameter range, quantized to 4 bits or lower, are the practical range today. At 4 bits, a 3-billion-parameter model’s weights take roughly 1.5 to 2 GB, plus working memory that grows with context length. Apple’s built-in model is about 3 billion parameters at 2 bits. Larger models can run on flagship devices with plenty of memory, but speed drops as size grows, and older phones may not have the memory to load them at all.

Does on-device deployment eliminate the need for a business associate agreement?

It eliminates the need for one with a model provider for inference that stays on the device. Other parts of the app may still need BAAs. If the app syncs data to a backend, integrates with an EHR, or sends analytics, those flows can involve PHI. And if you’re building the app on behalf of a covered entity, your company may itself be a business associate. For consumer apps outside HIPAA, the FTC’s Health Breach Notification Rule and state health privacy laws apply instead. Evaluate the architecture end to end.

How do you keep an on-device model current with clinical guidelines?

Keep the changing knowledge out of the model’s weights where you can. Store guidelines in a local knowledge base the model retrieves from, and update that content independently of the model. Have the app check for content and model updates at launch, and display the version and date of the guidance behind each answer so clinicians can judge whether it’s current. When the model itself must change, ship it through an app update, and if the feature is a medical device, cover planned changes in a predetermined change control plan.

What are the battery and performance impacts of running an LLM on a mobile device?

Inference is computationally heavy, so long conversations or continuous background processing drain the battery faster than typical app use, and sustained use can trigger thermal throttling that slows responses. Hardware acceleration, quantization, batching, and letting users decide when the model runs all reduce the impact. Measure it on your lowest supported device, over realistic session lengths.

Can on-device models be fine-tuned or updated without releasing a new app?

Fine-tuning on the device is technically possible with methods such as LoRA, but it’s rare in production healthcare apps because it’s resource-intensive and opens the door to corrupted or manipulated training data. Most teams fine-tune centrally and ship the result. Content updates through a local knowledge base can happen without an app release. Federated learning, where devices contribute to a shared model without sending raw data, lets models improve over time, but it reintroduces coordination and privacy questions that on-device deployment was meant to avoid.