8 Generative & Frontier Models

The previous chapters described models that label images — assign a class, draw a box, outline an organ. Generative models learn the distribution of images well enough to produce new ones: to fill in missing tissue, translate between modalities, synthesize training data, or answer questions in natural language. This chapter covers that shift and the frontier models that now dominate the conversation: diffusion models, vision-language systems, medical foundation models, and the agentic workflows built on top of them.

For the engineer

Generative models trade a discriminative model’s clear loss landscape for a much harder evaluation problem: when the model creates the output, how do you know it is right? This chapter’s recurring theme is that faithfulness, not realism, is the property medical applications must demand.

For the clinician

The models described here are the ones that make headlines: image generators, chatbots, systems that “see and reason.” The practical questions are the same as for any other AI system — what was it trained on, what can it actually be trusted to do, and who checks its output — but the failure modes are subtler, because a fluent, confident, realistic answer can be wrong in ways that are hard to detect.

8.1 Diffusion models

Diffusion models learn to create images by learning to destroy them: start with a real image and progressively add noise until nothing recognizable remains; then train a network to reverse that process step by step. Once trained, the network can start from pure noise and walk backward to a realistic sample. Text conditioning — the mechanism behind image generators like Stable Diffusion — lets the reverse process be steered by a description.

In medicine, diffusion models are used for:

Synthesis — generating realistic images for data augmentation, privacy-preserving sharing, or balancing underrepresented classes.
Reconstruction — recovering a sharp image from undersampled MRI (accelerated acquisition), low-dose CT, or sparse-view CT. Here the generative model acts as a learned prior: it knows what anatomy should look like, and uses that knowledge to fill in what the scanner did not capture.
Anomaly detection — by learning what normal looks like and flagging pixels where the real image cannot be reconstructed from the learned distribution.

The central caveat: because diffusion models are generative, they can invent plausible detail. A reconstructed image may look convincing and contain tissue that was not there. Whether that is acceptable depends entirely on the downstream use, and is the recurring question of this chapter.

8.2 GANs: history and remaining niches

Before diffusion, generative adversarial networks (GANs) were the dominant approach: a generator produces images and a discriminator judges real versus fake; training pushes each to beat the other, and the generator learns to produce samples the discriminator cannot distinguish from real data. GANs powered the first wave of medical image synthesis, super-resolution, and modality translation (CycleGAN, pix2pix, and their many descendants).

Diffusion has largely displaced GANs for general image generation because it is more stable to train and produces higher-quality, more diverse samples. GANs retain niches where their strengths matter: extremely fast one-step generation, precise paired translation tasks, and legacy systems still in production. The field’s current center of gravity, though, is diffusion and its successors.

8.3 Vision-language models

Vision-language models (VLMs) combine a vision encoder with a language model so the system can reason about images and text in the same space. Feed it an X-ray and the question “is there a pneumothorax?” and it can answer in natural language, grounded in what the vision encoder extracted.

In medical imaging, VLMs power:

Report generation and editing — draft findings from an image, or revise an existing report.
Visual question answering — “where is the abnormality?” “is this nodule larger than last time?”
Zero-shot classification — asking whether an image shows a finding the model was never explicitly trained to detect, by comparing its visual embedding to a text description.

Their promise and their danger are the same property: fluency. A VLM’s output is generated text, and its faithfulness to the image must be evaluated explicitly, not assumed — a theme that carries into the report-generation discussion in Chapter 6.

8.4 LLMs in imaging

Large language models touch imaging in three distinct roles:

Report generation and structured extraction. A VLM drafts findings from the image; a plain LLM turns dictated reports into structured fields, extracts measurements, or rewrites a report at a reading level appropriate for a patient.
Retrieval-augmented generation (RAG). Instead of answering from parametric memory alone, the model first retrieves relevant documents — UpToDate entries, departmental protocols, the hospital’s own guidelines — and grounds its answer in them. For medicine, where the answer must be traceable and current, RAG is less an enhancement than a requirement.
Reasoning and agentic use. Chain-of-thought reasoning, tool use, and agentic orchestration turn an LLM from a text generator into a system that can call a segmentation model, query the EHR, and compose a workflow. That is the subject of Chapter 9.

RLHF and instruction tuning are what make these models usable: after pre-training, human feedback teaches the model to follow instructions, prefer honest answers, and refuse when it does not know — imperfectly, and with hallucination still a live failure mode.

8.5 Medical foundation models

A foundation model is a large model pre-trained on broad data (self-supervised on unlabeled medical archives, or broadly supervised across tasks) that can then be adapted — fine-tuned, prompted, or distilled — to many downstream tasks. Medical imaging has produced several families:

General segmentation — MedSAM, MedSAM2, BiomedParse, TotalSegmentator: promptable or task-flexible segmentation across organs and modalities. The SAM family’s medical descendants (Chapter 7) belong here.
Modality-specific encoders — CheXzero, CheXagent, RadImageNet: self-supervised or supervised models pre-trained on large radiology archives, ready to be fine-tuned on downstream tasks with far less labeled data than training from scratch.
Multimodal medical LLMs — MedGemma, LLaVA-Med, and similar models that accept images and text, aiming to serve as the conversational layer over imaging.
Model-as-infrastructure — TotalSegmentator’s organ segmentation, nnU-Net’s auto-configuration, open model zoos that treat a model as a reusable building block rather than a one-off project.

The strategic question for any organization is whether to fine-tune a foundation model or train from scratch. Foundation models win when data is scarce, tasks are diverse, and the pre-training distribution is close enough to the target; bespoke models still win when the task is narrow, the data is abundant, and the foundation model’s priors are wrong for the modality.

8.6 Agentic workflows

The models above answer questions when asked. An agentic system decides which questions to ask, calls the tools needed to answer them, and composes the results — a shift from “model as function” to “model as orchestrator.” In imaging, that means an agent that receives a study, decides which segmentation model to call, queries the prior comparison, reads the measurement tool’s output, and drafts a report — all under human review.

That capability is significant enough to deserve its own chapter: Chapter 9 examines the harness, the tools, the orchestration patterns, and the safety implications in detail.