How to Build High-Quality Dermatology Image Datasets for AI
A dermatology AI model can score well in a paper and still struggle the moment it meets a patient who doesn’t resemble the training set. That’s rarely a modeling failure. Most of the time it’s a data problem, and it usually gets baked in months before anyone trained a model.
Building a dermatology image dataset that holds up outside a benchmark takes more than pulling images from a public archive and attaching labels. It takes the same habits a careful clinician uses: know what you’re looking for, know what’s missing, and check the work more than once. Below is what that process actually involves, in the order most teams need to work through it.
Define the AI Application First
Before collecting a single image, get specific about what the model needs to do. Is it a screening tool or a diagnostic aid. Is it meant for primary care triage or a specialist second opinion. Is it detecting melanoma specifically or classifying lesions across a broad range of conditions.
This decision shapes almost everything that follows: which conditions to include, whether dermoscopic images are required or clinical photos are enough, how many cases are needed per class, and which regulatory pathway the project is likely to end up in. A model built to flag “possible malignancy, refer to specialist” needs a different dataset than one meant to distinguish between a dozen inflammatory skin conditions in a primary care setting. Teams that rush past this step often end up with datasets that look impressive on paper and don’t map to any real clinical decision.
Select the Skin Conditions to Cover
With the use case defined, the next task is choosing which conditions belong in the dataset, and which ones don’t. Most dermatology AI projects concentrate on a focused set of high-impact categories: melanoma, basal cell carcinoma, squamous cell carcinoma, benign nevi, and common inflammatory conditions such as eczema, psoriasis, and acne.
Depth beats breadth here. A dataset spread thin across fifty diagnoses is usually less useful than one built deep across ten. Rare conditions still deserve representation, but not at the cost of the statistical power needed for the categories the model will encounter most often once it’s actually in use.
Clinical Image Collection
Clinical photographs, taken with a standard camera or smartphone under normal lighting, form the backbone of most real-world dermatology datasets because they reflect how patients and primary care providers actually capture and share images. The catch is that “just take a photo” hides a lot of technical detail.
Consistency matters more than almost anything else at this stage: fixed distance and angle, controlled and documented lighting, a consistent background where possible, and color calibration references given how much diagnosis in dermatology depends on color and texture. Inconsistent acquisition introduces noise that no amount of cleaning later on can fully undo.
Dermoscopic Image Collection
Dermoscopic images, captured with a dermatoscope that magnifies and illuminates the skin surface, reveal structures the naked eye can’t pick up and remain the standard for pigmented lesion analysis. The International Skin Imaging Collaboration (ISIC) has spent nearly a decade assembling one of the largest publicly available collections of quality-controlled dermoscopic images, sourced from clinical centers worldwide using a range of devices, with every incoming image screened for privacy and quality before it’s added to the archive. Anyone building or benchmarking a dermoscopy model would do well to study that screening process before designing their own.
The practical lesson from that work: device variability is a real source of error. When pooling images across sites or devices, acquisition settings need to be documented closely enough that a model learns the clinical signal, not the fingerprint of a particular camera or lighting setup.
Patient Diversity Isn’t Optional
This is the part most teams underweight, and it’s usually the one that causes the most damage after deployment. Dermatology AI has a documented skin tone problem. The research team behind the Diverse Dermatology Images dataset found that existing public datasets, including ISIC, largely lack clinical images and skin tone diversity, and that models trained on that data show a marked drop in performance when tested against a more representative benchmark. A broader survey of skin tone assessment practices in clinical research reached a similar conclusion, noting that noninvasive diagnostic tools, AI skin diagnosis benchmarks included, tend to show reduced accuracy in patients with darker skin tones, and that regulators are actively looking at how device evaluation should account for this.
In practice, that means sourcing images across the full range of Fitzpatrick skin types, or a finer scale like the Monk Skin Tone scale, tracking that distribution as a metric rather than assuming it will even out on its own, and treating underrepresented skin tones as a collection priority rather than something to patch in later with synthetic data.
How Many Images Do You Actually Need
There’s no single number that applies across projects, and anyone offering one without asking about the use case first is guessing. Per-class sample size, image quality, and diversity of presentation matter more than raw volume.
As a rough reference point, published dermoscopic datasets used in melanoma classification research often run from the low thousands to well over ten thousand images for major classes, and that’s before accounting for the diversity requirements above. Rare conditions or subtle differentials usually need targeted, proactive collection rather than reliance on whatever happens to be sitting in a retrospective archive, since naturally occurring cases of an uncommon presentation are unlikely to accumulate to a useful volume on their own. Projects that depend on rare categories should budget time and resources for deliberate collection from the outset rather than assuming the numbers will work themselves out.
Dataset Preparation for Machine Learning
Even well-captured images aren’t ready for training as-is. Preparation typically covers:
- De-identification. Removing metadata, visible identifiers, and any protected health information embedded in the image itself, in line with HIPAA and GDPR.
- Standardization. Normalizing resolution, color calibration, and file format across sources so the model isn’t inadvertently learning to tell one hospital’s equipment apart from another’s instead of learning the pathology.
- Splitting. Dividing data into training, validation, and test sets at the patient level rather than the image level, to prevent the same patient’s images from leaking across splits.
- Metadata structuring. Attaching clinical context, such as age, anatomical site, skin type, and how the diagnosis was confirmed, in a consistent, machine-readable schema.
Skip any of these and the effect usually shows up later as an unexplained accuracy problem that’s really a data engineering issue wearing a disguise.
Annotation: Where Precision Matters Most
Annotation is where clinical expertise and AI development meet directly. Bounding boxes are a reasonable choice for fast, lower-cost localization of a finding. Segmentation, which traces the exact boundary of a lesion, takes longer and costs more, but it’s necessary when a model needs to understand shape, border irregularity, or precise area, all of which carry diagnostic weight in melanoma assessment. Multi-annotator workflows, where more than one qualified reviewer labels the same image, have become standard practice for anything intended for clinical use, since one reviewer’s call on an ambiguous border can noticeably shift what the model ends up learning.
Whichever annotation method is used, the qualifications of the annotator matter as much as the tooling. A well-built labeling interface doesn’t compensate for a reviewer who doesn’t understand what they’re looking at.
Medical Quality Assurance
A dataset needs QA that goes beyond checking whether files opened correctly. That means clinical review of a representative sample of labels, a clear process for resolving disagreement between annotators, and documentation of how each diagnosis was confirmed. Biopsy-confirmed ground truth carries different weight than a single clinician’s visual impression, and a well-documented dataset is honest about which one it’s using.
This stage is also where diversity gaps, labeling drift, and acquisition inconsistencies tend to show up, provided someone is actually looking for them. Building QA in as an ongoing checkpoint rather than a one-time audit at the end catches problems while they’re still inexpensive to fix.
Dataset Balancing
Even a carefully collected dataset is rarely balanced from the start. Malignant cases are naturally rarer than benign ones, and common conditions vastly outnumber rare ones. Balancing techniques, including targeted oversampling of underrepresented classes, weighted loss functions during training, or simply prioritizing collection toward known gaps, help keep a model from learning to predict the majority class and calling that accuracy.
The goal isn’t perfectly even representation across every category. That’s rarely realistic clinically. It’s having enough signal in each class for the model to learn real distinguishing features instead of shortcuts.
Regulatory Considerations
For a model headed toward clinical use in the US, FDA expectations about the underlying data are getting more specific. The FDA’s AI/ML-based Software as a Medical Device Action Plan and related good machine learning practice guidance call for manufacturers to use unbiased, representative datasets and stay transparent with end-users about algorithm changes over time. The agency has also moved toward a total product life cycle approach, treating dataset composition as something to document and monitor continuously rather than certify once and forget. Building that documentation habit early in a project, tracking where the data came from, how it was diversified, and how labels were confirmed, is far less painful than reconstructing it after the fact for a submission.
Data privacy compliance, HIPAA in the US and GDPR in the EU, runs alongside these device-specific requirements and needs to be part of the collection workflow itself rather than something added at the end.
The same underlying discipline shows up across medical imaging AI generally, not just dermatology. Acquisition protocols, annotation rigor, and de-identification carry over in similar form to other modalities, and we’ve walked through how these considerations play out in practice in this piece on dermatology data collection for AI. For teams weighing where dermatology fits into a broader data strategy, dermatology image data collection tends to follow the same basic sequence as other imaging domains: collection, annotation, and medical QA, even when the clinical specifics differ.
Frequently Asked Questions
How do you build a dermatology AI dataset? Start by defining the model’s intended clinical use, then select the conditions it needs to cover, collect clinical and/or dermoscopic images under standardized protocols, ensure diversity across skin tones and presentations, and run the data through de-identification, annotation, and medical QA before splitting it for training and testing.
What images should be collected? Both clinical photographs and dermoscopic images, depending on the use case. Clinical images suit general screening and triage applications, while dermoscopic images are better for pigmented lesion analysis where fine structural detail matters. Each needs its own consistent acquisition protocol.
How many images are needed? It depends on the number of classes, their prevalence, and the performance bar the project is aiming for. Per-class sample sizes in the low thousands are a common starting point for major categories in published research, while rare conditions usually need dedicated, proactive collection rather than passive accumulation.
How can datasets be prepared for machine learning? Through de-identification, image standardization across resolution, color, and format, patient-level train/validation/test splitting to prevent data leakage, and structured metadata that captures clinical context alongside each image.
How is patient privacy protected in dermatology imaging datasets? Through de-identification workflows that strip metadata and visible identifiers, informed consent processes aligned with HIPAA and GDPR, and additional anonymization steps for identifiable regions such as the face before the data is used in model development.
What’s the difference between a research dataset and a clinically deployable one? A research dataset needs to show a concept works. A clinically deployable one needs documented, reproducible collection protocols, diversity across the populations it will actually encounter, confirmed ground truth where possible, and the kind of audit trail regulators expect to see behind a submission.
