24 Challenges, Benchmarks & Your First Research Project
A medical-imaging challenge gives a newcomer a bounded task: specified inputs, a reference standard, an evaluation protocol, and often a baseline. It can teach image geometry, data curation, model development, and reproducibility in one experiment. It also makes shortcuts easier to expose because the question and scoring procedure are written down.
The phrase open challenges in earlier chapters refers mainly to unsolved research problems. This chapter names organized competitions, benchmarks, and dataset exercises you can use to begin working on those problems. The directory is a selected, source-linked snapshot reviewed on 2026-10-11; it is not a list of competitions guaranteed to accept submissions today.
24.1 Three different ways a challenge can remain useful
An active competition has a current timeline and submission phase. A continuous benchmark may accept new evaluations after the original event. An archived challenge can still offer training data, reference labels, evaluation code, and papers for local reproduction. These states are not interchangeable. A functioning homepage or an old registration link does not establish that submissions are open.
For example, the ACDC organizers explicitly say the challenge is closed while the data and ground truth remain available. The PANDA homepage also marks its competition closed. Both can still motivate carefully scoped learning projects, subject to current data access. In contrast, SynthRAD2025 describes a post-event leaderboard continuing through 2030. Confirm the actual submission interface and rules before planning an entry.
Challenge families change. Learn2Reg describes 2026 PSMA PET/CT and breathing-related registration tasks; earlier editions have different inputs and references. autoPET V emphasizes interactive clinician-in-the-loop PET/CT lesion segmentation. A paper saying only “Learn2Reg” or “autoPET” is not a sufficiently reproducible dataset specification.
24.2 Choose the project before choosing the model
Begin with one output you can inspect. Two-dimensional lesion segmentation or a single-organ volume task is often easier to debug than whole-slide grading, multimodal longitudinal analysis, or real-time video. This is a starting-scope recommendation, not a ranking of clinical importance.
Ask five practical questions:
- Can I access the data under its terms? Read registration, institutional requirements, permitted use, redistribution, and citation rules. “Downloadable” does not mean unrestricted. fastMRI requires an access application and a data-sharing agreement.
- What is the prediction unit? A pixel, lesion, image, volume, slide, patient, or procedure changes splitting and evaluation. Keep patients and their derivatives together.
- What does the reference actually mean? A contour, radiologist-marked opacity, grade, or survival outcome supports a different claim. Benchmark labels do not automatically represent a diagnosis or a treatment decision.
- Can I afford a complete baseline? Estimate download size, decompression, preprocessing, GPU memory, and training time. A reduced local experiment must be labeled as such rather than compared as if it used the official full protocol.
- Can I reproduce the score? Identify the official implementation, version, aggregation rules, missing-output policy, label conventions, and submission format before optimizing anything.
24.3 A directory of starting points
The machine-readable directory is data/challenges.csv. Each entry records the edition or scope, task, access action, evaluation focus, beginner exercise, major pitfall, source, and status qualification. Evaluation focus is a learning aid; the linked organizers’ code defines the actual ranking. Status text deliberately says when a current submission window was not established.
| Challenge or resource / scope | Task and evaluation focus | Access and status | Starter exercise and pitfall |
|---|---|---|---|
| RSNA Pneumonia Detection — 2018 (Chest radiography) | Localize radiologist-marked pulmonary opacities. Evaluation: Official box matching and detection scoring. | Archived edition; current competition entry not established. Access: Use RSNA/Kaggle access instructions and dataset terms. Reviewed 2026-10-11. | Reproduce a box-localization baseline and inspect coordinates. Pitfall: Opacity labels do not establish microbiological pneumonia. |
| Medical Segmentation Decathlon — 2018 benchmark; choose one of ten tasks (CT/MRI) | Task-specific anatomical or tumor segmentation. Evaluation: Official task labels and ranking; overlap and boundary assessment. | Historical benchmark; current submission window not established. Access: Read task-specific downloads and license/citation requirements. Reviewed 2026-10-11. | Start with one organ task and source-space overlays. Pitfall: A subset experiment is not the full Decathlon protocol. |
| ACDC — 2017 (Cardiac cine MRI) | Ventricular/myocardial segmentation at defined phases; examination classification. Evaluation: Task-specific segmentation and classification evaluation. | Closed; organizers explicitly retain data and ground truth. Access: Follow the official dataset download and citation instructions. Reviewed 2026-10-11. | Segment documented phases then inspect volume conventions. Pitfall: Phase or contour-convention errors change measurements. |
| BraTS — 2019 segmentation track (Multi-sequence brain MRI) | Segment edition-defined brain tumor regions. Evaluation: Dice and 95th-percentile Hausdorff distance per defined region. | Archived edition; do not infer current family participation from this page. Access: Use official edition data-request instructions and terms. Reviewed 2026-10-11. | Reproduce region masks and explicit missing-sequence handling. Pitfall: Region definitions and preprocessing differ across editions. |
| KiTS23 — 2023 (Contrast-enhanced kidney CT) | Segment kidney, renal tumor, and cyst. Evaluation: Dice and surface Dice on hierarchical evaluation classes; official rank aggregation. | 2023 benchmark; current submission window not established. Access: Official GitHub/download instructions; CC BY-NC-SA 4.0 terms. Reviewed 2026-10-11. | Reproduce one segmentation baseline with HEC scoring. Pitfall: Replacing class unions with a custom class average changes the benchmark. |
| LUNA16 — 2016 (Chest CT) | Pulmonary nodule detection or false-positive reduction. Evaluation: FROC sensitivity versus false positives per scan; official hit/exclusion rules. | Historical benchmark; verify current evaluation access. Access: Read official data and submission instructions. Reviewed 2026-10-11. | Evaluate supplied candidates before attempting full detection. Pitfall: World coordinates and excluded findings affect hit counts. |
| fastMRI — Choose exact dataset and historical challenge track (Raw MRI k-space) | Accelerated MRI reconstruction. Evaluation: Official reconstruction metrics and data conventions. | Dataset/baseline resource; current competition window not established. Access: NYU application and data-sharing agreement; research/education restrictions. Reviewed 2026-10-11. | Reproduce a documented reconstruction baseline. Pitfall: Pixel similarity does not establish diagnostic fidelity. |
| ISIC — 2018 Task 1: lesion boundary segmentation (Dermoscopy) | Skin-lesion boundary segmentation. Evaluation: Selected task’s official mask scoring. | Historical task in challenge archive; check phase/access before entry. Access: Read task data instructions and collection-specific rights. Reviewed 2026-10-11. | Build a small 2D segmentation baseline with grouped splits. Pitfall: Repeated images of one lesion can leak across partitions. |
| REFUGE — 2018 (Fundus photographs) | Glaucoma classification and optic disc/cup segmentation. Evaluation: Official classification and disc/cup task evaluation. | Archived edition; current submission window not established. Access: Read official challenge data-access and use rules. Reviewed 2026-10-11. | Start with disc/cup overlays before disease classification. Pitfall: Image labels and dataset prevalence do not establish a screening pathway. |
| CAMELYON17 — 2017 (Whole-slide lymph-node histology) | Metastasis analysis and patient-level nodal staging. Evaluation: Official endpoint and slide/patient aggregation rules. | Historical benchmark; verify current evaluation access. Access: Use organizer registration and data instructions. Reviewed 2026-10-11. | Inspect tissue tiles and their slide/case provenance. Pitfall: Tile-level accuracy cannot substitute for patient-level staging. |
| PANDA — 2020 (Prostate biopsy whole-slide histology) | Gleason/ISUP grade assessment. Evaluation: Official ordinal grading score and label protocol. | Closed on organizer homepage; data access checked separately. Access: Follow linked Kaggle dataset access and terms. Reviewed 2026-10-11. | Reproduce slide-level grading with a fixed tiling baseline. Pitfall: Stain/site shifts and slide-to-case relationships affect evaluation. |
| EndoVis Robotic Instrument Segmentation — 2017 (Endoscopic surgical video) | Binary, instrument-part, and instrument-type segmentation. Evaluation: Task-specific segmentation protocol in organizer report. | Archived edition; access and submission must be rechecked. Access: Consult the original challenge report and dataset instructions. Reviewed 2026-10-11. | Reproduce frame masks while splitting complete procedures. Pitfall: Adjacent frames and future context can inflate real-time claims. |
| Learn2Reg — 2026 PSMAReg / Learn2Breath; select one (PET/CT and respiratory CT) | Longitudinal or breathing-related image registration. Evaluation: Selected task’s accuracy, deformation plausibility, and runtime checks. | 2026 tasks described; current submission phase not established. Access: Follow organizer links to selected CodaBench task and rules. Reviewed 2026-10-11. | Start with identity transforms and landmark/geometry checks. Pitfall: Displacement direction, units, and resampling conventions matter. |
| HECKTOR — 2022 (FDG PET/CT plus clinical data) | Head/neck primary and nodal tumor segmentation; recurrence-free survival prediction. Evaluation: Task-specific segmentation or survival-risk scoring. | Archived 2022 edition; portal points to a newer edition. Access: Read official data agreement and edition-specific submission instructions. Reviewed 2026-10-11. | Begin with segmentation and source-space fusion review. Pitfall: Tumor masks and censored survival endpoints are different tasks. |
| autoPET V — Fifth edition (Whole-body PET/CT) | Interactive clinician-in-the-loop lesion segmentation. Evaluation: Official prompt/interaction budget and lesion evaluation. | Fifth-edition task described; current submission phase not established. Access: Read current timeline, data instructions, and interaction rules. Reviewed 2026-10-11. | Compare a baseline under one fixed simulated interaction protocol. Pitfall: Oracle clicks and clinician interaction are different conditions. |
| SynthRAD — 2025 (MRI or CBCT paired with CT) | Synthetic CT generation for radiotherapy research. Evaluation: Official image and radiotherapy-oriented evaluation. | Organizer states post-event leaderboard remains open through 2030; confirm interface. Access: Follow organizer dataset registration and task-specific terms. Reviewed 2026-10-11. | Reproduce one anatomical-site baseline with paired geometry review. Pitfall: Image fidelity alone does not validate treatment planning. |
| OpenKBP — 2020 dose-prediction benchmark (Contoured planning CT) | Predict spatial radiotherapy dose. Evaluation: Official dose and dose-volume-histogram evaluation. | Archived edition documented in organizer paper; current submission not established. Access: Consult organizer paper and linked data/evaluation release. Reviewed 2026-10-11. | Reproduce dose prediction separately from plan optimization. Pitfall: Predicted dose is not automatically a deliverable treatment plan. |
| ToothFairy — 2023 first edition (Dental CBCT) | Inferior alveolar canal segmentation. Evaluation: Selected edition’s official segmentation scoring. | Archived first edition; later targets and access require separate checks. Access: Read organizer data-access and label-protocol instructions. Reviewed 2026-10-11. | Inspect thin-structure labels and physical-space connectivity. Pitfall: Canal annotations must not be treated as direct validated nerve anatomy. |
| SNMMI Lu-177 Dosimetry Challenge — 2021 tasks; variability study (Quantitative SPECT/CT and planar imaging) | Compare stages of internal-dosimetry calculations. Evaluation: Organ and lesion volumes, time-integrated activity, and dose-estimate variability. | Historical study; current submission phase not established. Access: Follow original paper to University of Michigan Deep Blue Data; verify task-specific files and terms. Reviewed 2026-10-11. | Compare two stated curve-integration methods on the same supplied measurements. Pitfall: The patient-data study examines variability and does not supply absolute dose-accuracy ground truth. |
| BBBC047 Cell Painting — Version 1; dataset exercise (Multiplexed fluorescence microscopy) | Build and compare morphological profiles of compound perturbations. Evaluation: Prespecified replicate agreement and profile retrieval on grouped held-out data. | Published research dataset; not a live competition. Access: Follow BBBC links to images and profiles; retain required attribution and inspect metadata. Reviewed 2026-10-11. | Reproduce a small profile-retrieval baseline before training an image encoder. Pitfall: Related wells and plate effects can inflate apparent biological generalization. |
| JUMP Cell Painting — Choose named dataset release and perturbation set (Multiplexed fluorescence microscopy) | Compare chemical and genetic perturbation profiles. Evaluation: Define a retrieval or generalization protocol with controls and batch-aware splits. | Public resource family; not a single ranked competition. Access: Use JUMP Hub to select a documented release; inspect individual data terms and storage requirements. Reviewed 2026-10-11. | Compare replicate agreement and retrieval across one declared held-out batch. Pitfall: Profile similarity generates a hypothesis and does not establish mechanism or therapeutic benefit. |
| QIN-PROSTATE-REPEATABILITY — TCIA version 3; test-retest exercise (Multiparametric prostate MRI) | Study repeated image-derived measurements using paired scans and annotations. Evaluation: Prespecified paired differences, bias and repeatability with units and denominators. | Published test-retest collection; not a live competition. Access: Check collection documentation, CC BY 4.0 attribution and TCIA policies; retain paired-study identity. Reviewed 2026-10-11. | Reproduce a source-linked paired measurement report and inspect exclusions. Pitfall: Test-retest data do not measure treatment efficacy or establish an imaging surrogate endpoint. |
24.3.1 Start small: annotation, cardiac contours, or opacity localization
An archived ISIC lesion-boundary task provides a two-dimensional mask problem, ACDC provides a phase-specific cardiac segmentation problem, and the RSNA 2018 challenge provides a chest-radiograph localization problem. These make different learning exercises: preserve lesion grouping for skin images, phase and physical geometry for cardiac volumes, and image coordinates for detection boxes. Do not compare their metrics as if they measure the same capability.
For a first experiment, inspect a small permitted sample, render source and reference overlays, reproduce the simplest documented baseline, and produce an error report. For ACDC, evaluate contours before calculating downstream cardiac quantities. For chest radiography, keep the distinction between pulmonary-opacity annotations and a confirmed pneumonia diagnosis. RSNA’s dataset documentation explains the annotation setting.
24.3.2 Move to volumes: anatomy and tumor segmentation
Medical Segmentation Decathlon offers multiple segmentation tasks; choose one named task and keep its units and labels explicit. KiTS23 distinguishes kidney, tumor, and cyst labels and uses hierarchical evaluation classes. Replacing the official class unions with a familiar three-class average changes the benchmark.
BraTS provides a multi-sequence tumor setting, but editions and tasks have different label definitions and rules. Keep a label dictionary, evaluate each defined region, and inspect small or absent regions. A model that produces a good average mask can still fail on a clinically important focus. A new clinical claim requires an appropriate reference and validation beyond the leaderboard.
24.3.3 Then tackle the pipeline: registration, reconstruction, pathology, and video
Registration requires transform conventions and spatial validation, not just an image similarity loss. Start with an identity transform and known landmarks, then reproduce the selected Learn2Reg task’s checks. MRI reconstruction needs raw-data and sampling conventions; start from a documented fastMRI baseline and do not equate pixel fidelity with diagnostic fidelity.
Whole-slide pathology needs physical scale, tile coordinates, slide/case grouping, and an aggregation rule. CAMELYON17 and PANDA concern different endpoints. Video challenges need procedure-level splits and an explicit temporal context: a method using future frames cannot be described as real-time merely because inference is fast.
For radiotherapy, separate contour quality, synthetic-CT fidelity, dose prediction, and deliverable planning. SynthRAD and OpenKBP address different outputs. Dental CBCT projects such as ToothFairy require label definitions and careful inspection of thin anatomy; do not infer that a later edition has the first edition’s target set.
24.3.4 RPT, discovery, and trials: choose a quantitative research question
The SNMMI Lu-177 Dosimetry Challenge studied variability across internal-dosimetry steps. Its paper links the data repository at the University of Michigan. It is a historical comparison exercise, not an established current entry window or an absolute dose-accuracy reference. Retain acquisition times, units, region definitions, curve assumptions, and intermediate outputs. Agreement between software packages is not sufficient proof that either estimate is accurate.
BBBC047 and JUMP provide discovery-oriented Cell Painting resources rather than a single live competition. Begin with quality checks and replicate/profile comparison. Define whether your test holds out compounds, plates, or laboratories; the independence unit differs from a patient-based clinical-imaging split.
For trial-oriented biomarker work, the QIN-PROSTATE-REPEATABILITY collection offers a test–retest setting. A repeatability exercise asks whether measurements remain consistent, not whether a treatment works. Use a separate, appropriately designed study for treatment-effect questions. An imaging-charter and audit-workflow simulation can be a useful engineering project even without a leaderboard.
See Section 23.4, Section 23.5, and Section 23.6 for the clinical or experimental contracts behind these exercises.
24.4 A four-week research exercise
This is an illustrative schedule for a small local project, not a guarantee of competition readiness.
| Week | Work | Reviewable deliverable |
|---|---|---|
| 1 | Read task and terms; inventory data; inspect overlays; freeze grouped split | Data card, label dictionary, source citations, split manifest |
| 2 | Reproduce a baseline; verify metric code and inverse mapping | Versioned configuration, checkpoint provenance, source-space predictions |
| 3 | Examine failures; try one justified change; record compute cost | Baseline-versus-change comparison on the same validation cases |
| 4 | Freeze the method; run untouched local test or permitted official submission | Reproduction instructions, metrics with denominators, error gallery, limitations |
Use a tiny subset for loader and geometry checks, then decide whether to run the full experiment. An initial smoke test should detect swapped axes, label interpolation, incorrect units, empty outputs, and accidental training/test overlap before an expensive run.
24.5 Evaluate what the task actually asks
Segmentation. Overlap and boundary measures answer different questions. Small structures, absent classes, disconnected predictions, and spacing can affect interpretation. Use the official empty-class and aggregation rules. KiTS23’s documented Dice and surface-Dice ranking is not interchangeable with a custom metric summary.
Detection. A box or candidate is evaluated under a matching rule. LUNA16 measures sensitivity against false positives per scan with a defined nodule-hit criterion. Image-level AUROC cannot replace that localization evaluation.
Classification and grading. Ranking, thresholded performance, calibration, and ordinal agreement are different views of a model. Match the official endpoint, then add clearly labeled analysis relevant to your research question. A high challenge score does not identify a usable clinical threshold.
Registration and reconstruction. Inspect geometry, plausibility, runtime, and task consequences alongside numerical metrics. A transform can align one structure while damaging another; a reconstruction can look smooth while deleting a feature.
Interactive methods. Specify who supplies prompts, what information they see, and the click or editing budget. A simulated oracle prompt and a real user’s action are different evaluation conditions. Preserve failed and abandoned cases in the denominator.
24.6 Avoid turning the leaderboard into a training set
Choose hyperparameters on training and validation material, freeze decisions, and use the test set for the stated final comparison. Repeated leaderboard submissions provide feedback about the hidden test set, even without revealing its labels. Respect submission limits and disclose how many variants you evaluated.
Watch for repeated patients, lesions, eyes, visits, slides, adjacent frames, and reformatted volumes across partitions. New filenames do not create independent observations. Public data used for pretraining can overlap a benchmark; report known overlap and uncertainty instead of assuming “foundation model” means a clean test.
A benchmark result should state edition, task, data split, permissible extra data, preprocessing, model and checkpoint versions, metric implementation, inference budget, missing-output count, and submission history. Distinguish reproduced official scores from local scores on a custom split. Keep uncertainty and error review alongside the average.
24.7 Find the next challenge without relying on stale dates
Use the Grand Challenge directory, RSNA challenge hub, and the named organizer portals to locate tasks. Read the current edition’s rules, access instructions, compute constraints, external-data policy, and submission phase. Some portals retain old “ongoing” labels; verify the dated timeline and working submission page.
The directory deliberately omits prize promises and unverified deadline dates. Update its reviewed_on and status evidence when rechecking an entry. Data-access and submission availability are separate fields: a closed competition can have accessible data, while an announced task can require approval before any download.
24.8 Connect the exercise to a specialty workflow
Use Chapter 23 to identify who will inspect the result and how it might be used. A segmentation challenge is a strong place to learn source-space overlays and annotation review. It is not, by itself, an evaluation of longitudinal response, radiation treatment safety, a surgical plan, or an autonomous report.
In MedAI OS, use the currently verified viewer features to inspect research images and reference outputs. Add optional inference or annotation only after checking the application version and prerequisites. A successful challenge submission and a successful viewer round trip are complementary pieces of engineering evidence; neither constitutes clinical authorization.