Reviewing the Medical Literature: Title and subTitle BreakStatistics Alone Are Not Enough

THE STUDY titled "Oral Treatment for Toenail Onychomycosis" by Crawford et al¹ in this issue of the ARCHIVES has the objective "to identify and synthesize the evidence for the efficacy and cost-effectiveness of oral treatments for fungal infections of the toenails." Their ambitious goal is to examine "all available data from evaluations of these therapies" and include "a statistical summary of their clinical and cost-effectiveness." This very thorough search included 11 databases, a hand search of 3 podiatry journals, searching the bibliographies of all review articles, and contacting all podiatry schools in the United Kingdom as well as pharmaceutical firms to identify unpublished or unlisted trials. Rigorous selection criteria were applied by 4 workers working independently in pairs. All reviewers independently summarized the trials using 12 quality criteria.

As their therapeutic end point, the authors chose a mycologically negative nail. Mycologic results of various studies were directly compared and subjected to statistical analysis. This assumes that mycologic examination is a standardized, reproducible technique that does not vary from laboratory to laboratory. If only this were the case! The mycologic examination is not a standardized laboratory procedure such as blood glucose or hemoglobin determinations. The 3 steps of sampling, direct microscopy, and culture are all highly sensitive to individual variation. The dermatologic and mycologic literature is replete with studies comparing the accuracy of various techniques of mycologic examination of the nail, which is attested by recent reports.^{2 - 4} Mycologic cure rates are only worthy of very rough comparison unless all mycology tests were performed in the same laboratory, preferably with blinding.

There is another problem with the choice of a mycologically negative nail as an end point. Toenail onychomycosis is a clinical problem: patients desire a normal-appearing nail. There is the implicit assumption that the mycologic cure has a defined relationship to the desired clinical end point, a normal-appearing nail. This is not the case. In almost all studies providing data on both mycology and nail appearance, the mycologic success rate exceeds—by a variable margin—the percentage of normal nails achieved. In an extreme example, Faergemann et al⁵ found griseofulvin produced a mycologic cure in 45%, but only 2% achieved a normal-appearing nail.

Crawford and colleagues addressed the matter of clinical cure, but were frustrated by the lack of clear definitions of clinical outcomes. They found only 2 trials that clearly defined clinical cure and presented data for these outcomes. The authors did not attempt to use a clinical end point, but instead chose an end point that had the appearance of precision and accuracy, the mycologically negative nail.

The authors included griseofulvin, ketoconazole, and fluconazole in their detailed treatment analyses. Why include drugs any knowledgeable dermatologist would consider obsolete in the treatment of toenail onychomycosis? The answer to this question, and the other puzzling aspects of this study became apparent when I looked at the authors' affiliations. There were 2 research fellows, a lecturer in biostatistics, a professor in health economics, 2 professors in health sciences, a general practice research registrar, and a research podiatrist. No dermatologist or mycologist.

When designing their study, a dermatologic consultant might have made the following 2 suggestions: First, there is no point in studying drugs no longer used in treating toenail onychomycosis such as griseofulvin, ketoconazole, and fluconazole. Griseofulvin produced such dismal results in toenail therapy that it has been long abandoned. Ketoconazole showed somewhat better results than griseofulvin, but was clearly inferior to what itraconazole or terbinafine can accomplish. Furthermore, our dermatologist would point out that the occasional, but nevertheless potentially serious, hepatotoxicity of this compound makes it unsuitable for the lengthy treatment required in treating toenail onychomycosis. Fluconazole is sometimes effective, and, for a short time, was the treatment of choice until itraconazole and terbinafine made their appearance. Even the limited number of studies involving fluconazole showed it to be clearly inferior to itraconazole and terbinafine.⁶ Second, this leaves itraconazole and terbinafine as the 2 drugs currently in use in the treatment of toenail onychomycosis. Two blinded head-to-head comparisons between itraconazole and terbinafine showed a slightly better success rate with terbinafine that was statistically significant.^{7 - 8} These 2 studies were noteworthy for using both nail morphology and mycology as end points. Two other studies showed a slight superiority of terbinafine over itraconazole that was not statistically significant.^{9 - 10} The conclusion of this clinically focused approach to the literature is that terbinafine has the best cure rate among currently available treatments. A 1998 literature review came to the same conclusion.⁶

In their systematic review, Crawford and colleagues conclude that "There is good evidence that a continuous regimen of terbinafine (250 mg/d) for 3 months is the most effective oral treatment for fungally infected toenails." Was their exhaustive search of the literature really necessary to reach their conclusion?

A comprehensive review of clinical studies requires knowledge of the accuracy of procedures, the relevance of end points, the reliability of data, and overall knowledge of what is involved in patient care. This can only be obtained from someone with firsthand experience; either a knowledgeable dermatologist or mycologist would have enabled the authors to eliminate much unnecessary work. For meaningful evaluations of clinical studies, statisticians need the advice and help of clinicians, just as we clinicians depend on statisticians for our confidence limits, probability numbers, and other statistical necessities.