Apps Have Low Accuracy and Sensitivity for Detecting Melanoma

The accuracy and sensitivity of dermatology smartphone applications for melanoma detection are highly variable and overall low, investigators reported in a study published the British Journal of Dermatology.

The study included images from 35 patients (mean age, 54 years [range, 23-87]; 60% women). A total of 15 clinical images of histologically proven invasive melanoma (pT1a-pT2b) and 15 histologically proven benign nevi, all from patients with lighter skin phototypes, as well as 5 images of benign nevi in patients with skin of color, were assessed.

The researchers identified all publicly available smartphone and Web-based dermatology apps that offered artificial intelligence diagnostics, and the images were uploaded to each app. For apps that provided 1 or more diagnoses, the accuracy was calculated according to the presence or absence of melanoma in the top 1 or top 3 diagnoses.

Among the 43 apps that were identified, 25 purported  to identify melanoma and were functional. Of this group, 10 did not allow upload from camera roll and 8 required metadata. In addition, 4 apps rejected at 1 or more images, 15 of the 25 apps returned diagnoses, 12 of the 25 provided risk categories, 2 of the 25 provided risk scores, and 3 apps provided more than 1 output type.

Regarding top 1 measures among all the apps, the mean sensitivity was 0.28 (95% CI, 0.17-0.39), the mean specificity was 0.81 (95% CI, 0.71-0.91), and the mean accuracy was 0.59 (95% CI, 0.55-0.62). For the diagnosis-based, risk category-based, and score-based apps, the mean accuracy was 0.56, 0.60, and 0.64, respectively.

Among the 10 apps that provided at least 3 ranked diagnoses, the mean top 1 accuracy (0.56) was higher compared with the mean top 3 accuracy (0.41). The mean accuracy decreased from 0.63 to 0.61 in 4 apps when rejected images were classified as benign output. In all, 8 apps did not identify 1 melanoma in their top 1 ranking, 4 apps did not include melanoma in their top 3, and 3 apps had a specificity <.05.

SkinScan and SkinVision (which had the highest accuracy of 0.81) were the only apps that had a European conformity (CE) marking, although published studies assessed their performance as “low quality,” and none had US Food and Drug Administration approval, according to the investigators.

The findings are limited by the heterogenous output of apps, which did not allow direct comparison, noted the researchers. Other limitations include the retrospective nature of the clinical images and lack of melanoma in participants with skin of color.

“Regardless of disclaimers identifying apps as educational or research tools, this poses potential risks to the public,” stated the investigators. “Clinicians should be aware of app limitations and their widespread accessibility to lay users.”

Disclosure: Several of the study authors declared affiliations with medical device and/or dermatology services companies. Please see the original reference for a full list of authors’ disclosures.

Reference

Sun MD, Kentley J, Mehta P, Duzsa S, Halpern AC, Rotemberg V. Accuracy of commercially available smartphone applications for the detection of melanoma. Br J Dermatol. Published online November 22, 2021. doi:10.1111/bjd.20903