We're unable to sign you in at this time. Please try again in a few minutes.
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Original Investigation |

Detection Accuracy of Collective Intelligence Assessments for Skin Cancer Diagnosis

Ralf H. J. M. Kurvers, PhD1,2; Jens Krause, PhD1,3; Giuseppe Argenziano, MD4; Iris Zalaudek, MD5; Max Wolf, PhD1
[+] Author Affiliations
1Department of Biology and Ecology of Fishes, Leibniz-Institute of Freshwater Ecology and Inland Fisheries, Berlin, Germany
2Center for Adaptive Rationality, Max Planck Institute for Human Development, Berlin, Germany
3Faculty of Life Sciences, Humboldt-University of Berlin, Berlin, Germany
4Department of Dermatology, Second University of Naples, Naples, Italy
5Department of Dermatology and Venereology, Medical University of Graz, Graz, Austria
JAMA Dermatol. 2015;151(12):1346-1353. doi:10.1001/jamadermatol.2015.3149.
Text Size: A A A
Published online

Importance  Incidence rates of skin cancer are increasing globally, and the correct classification of skin lesions (SLs) into benign and malignant tissue remains a continuous challenge. A collective intelligence approach to skin cancer detection may improve accuracy.

Objective  To evaluate the performance of 2 well-known collective intelligence rules (majority rule and quorum rule) that combine the independent conclusions of multiple decision makers into a single decision.

Design, Setting, and Participants  Evaluations were obtained from 2 large and independent data sets. The first data set consisted of 40 experienced dermoscopists, each of whom independently evaluated 108 images of SLs during the Consensus Net Meeting of 2000. The second data set consisted of 82 medical professionals with varying degrees of dermatology experience, each of whom evaluated a minimum of 110 SLs. All SLs were evaluated via the Internet. Image selection of SLs was based on high image quality and the presence of histopathologic information. Data were collected from July through October 2000 for study 1 and from February 2003 through January 2004 for study 2 and evaluated from January 5 through August 7, 2015.

Main Outcomes and Measures  For both collective intelligence rules, we determined the true-positive rate (ie, the hit rate or specificity) and the false-positive rate (ie, the false-alarm rate or 1 − sensitivity) and compared these rates with the performance of single decision makers. Furthermore, we evaluated the effect of group size on true- and false-positive rates.

Results  One hundred twenty-two medical professionals performed 16 029 evaluations. Use of either collective intelligence rule consistently outperformed single decision makers. The groups achieved an increased true-positive rate and a decreased false-positive rate. For example, individual decision makers in study 1, using the pattern analysis as diagnostic algorithm, achieved a true-positive rate of 0.83 and a false-positive rate of 0.17. Groups of 3 individuals achieved a true-positive rate of 0.91 and a false-positive rate of 0.14. These improvements increased with increasing group size.

Conclusions and Relevance  Collective intelligence might be a viable approach to increase diagnostic accuracy in skin cancer and reduce skin cancer–related mortality.

Figures in this Article

Sign in

Purchase Options

• Buy this article
• Subscribe to the journal
• Rent this article ?


Place holder to copy figure label and caption
Figure 1.
Diagrams of the General Procedures for Participants in Studies 1 and 2

After the first-step diagnostic algorithm, the 3387 SLs evaluated as melanocytic underwent the next 4 diagnostic algorithms. ABCD indicates asymmetry, border, color, and diameter; SLs, skin lesions.

aMelanocytic vs nonmelanocytic SLs.

bMelanoma vs benign melanocytic SLs.

cMalignant vs benign SLs.

Graphic Jump Location
Place holder to copy figure label and caption
Figure 2.
Effects of Group Size on True- and False-Positive Rates Using the Majority and the Quorum Rules

The first-step algorithm (study 1) aimed to differentiate between melanocytic and nonmelanocytic lesions. The pattern analysis algorithm (study 1) aimed to differentiate between melanoma and benign melanocytic lesions. The 3-point checklist algorithm (study 2), aimed to differentiate between malignant (melanoma and basal cell carcinoma) and benign skin lesions. With increasing group size, the true-positive rate increased and the false-positive rate decreased for each combination of collective intelligence approach and each diagnostic algorithm. Data are expressed as mean values. The dashed lines represent the mean individual true- and false-positive rates (ie, group size of 1).

Graphic Jump Location
Place holder to copy figure label and caption
Figure 3.
Receiver Operating Characteristics Curves Using the Quorum Rule When the Quorum Threshold Is Varied

Each point is obtained by setting a different quorum threshold, starting at 0, with increments of 0.05 to 1. Data are shown for the first-step, pattern analysis, and 3-point checklist diagnostic algorithms. Data are based on a group size of 11.

Graphic Jump Location




Also Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
Please click the checkbox indicating that you have read the full article in order to submit your answers.
Your answers have been saved for later.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.


Some tools below are only available to our subscribers or users with an online account.

2 Citations

Sign in

Purchase Options

• Buy this article
• Subscribe to the journal
• Rent this article ?

Related Content

Customize your page view by dragging & repositioning the boxes below.

Articles Related By Topic
Related Collections
PubMed Articles