Martin Frické [HREF1], Associate Professor, School of Information Resources and Library Science [HREF2], University of Arizona [HREF3], 1515 East First Street, Tucson, AZ 85719, USA. mfricke@u.arizona.edu
Don Fallis [HREF4], Assistant Professor, School of Information Resources and Library Science, University of Arizona. fallis@u.arizona.edu
The Internet has become an important source of health information. However, several empirical studies indicate that there is a significant amount of inaccurate health information on the Internet. Thus, it is important that Internet users be able to distinguish the accurate information from the inaccurate information. Information professionals have developed checklists for evaluating the quality of health information to assist Internet users in this regard. However, such checklists can only be effective if the proposed indicators of accuracy really do indicate accuracy. This article reports on two empirical studies that implement an effective technique for identifying such indicators of accuracy. In particular, one of these studies indicates that the HONcode logo [HREF5] is more likely to be displayed on web sites that contain accurate health information. However, many commonly proposed indicators of accuracy (e.g., the author having medical credentials, currency, lack of advertising) were not found to be correlated with accuracy.
More and more people are using the Internet to find information. In particular, more and more people are using the Internet to find information about medical topics (see Fox and Rainie 2001). However, while the Internet has many advantages as an information resource (e.g., it provides fast access to information on a wide range of topics), it has one major drawback. Since almost anyone can publish almost anything on the Internet, there is wide variation in the quality of information found there.
This is an especially serious problem when it comes to health information. If Internet users end up believing and acting upon inaccurate health information, it can have dire consequences. Several empirical studies have now established that there is a significant amount of inaccurate health information on the Internet (Impicciatore et al. 1997, Biermann et al. 1997, Sandvik 1999, Griffiths and Christensen 2000, Berland et al. 2001).
Information professionals have taken the lead in dealing with the problem of inaccurate information on the Internet. In particular, they have taken steps to help Internet users to avoid inaccurate information. For example, they have developed portals (such as the Internet Scout Project [HREF6] and Blue Web'n [HREF7]), which provide links to Internet sources that have been reviewed for quality. (Ip et al. 2001 look at how to promote collaboration between such portals.) In addition, information professionals teach Internet users how to evaluate the quality of Internet sources for themselves.
Toward this end, information professionals have typically chosen to provide Internet users with checklists for evaluating information on the Internet. Such checklists contain a fairly standard list of items that are supposed to be indicators of accuracy. For example, the Internet user is advised to check that the author of the web site is identified, that the author is an authority on the topic, that the web site is current, that there is no advertising on the web site, etc. In the case of health information in particular, the Internet user is also advised to check that the author has medical credentials, that peer-reviewed medical literature is cited, etc.
Such checklists can be found on a large number of web sites on evaluating information on the Internet (Oliver 1998, Smith 2002) and on evaluating health information in particular (Ambre et al. 1997, Silberg et al. 1997). Also, there are at least two new books on evaluating information on the Internet that emphasize this checklist strategy (Alexander and Tate 1999, Cooke 1999). In addition, several researchers have developed software tools that partially automate the process of using these checklists (Middleton et al. 1999, Price and Hersh 1999). Finally, such checklists have been widely used in academic libraries to teach students how to evaluate information on the Internet (see Vincent and Norman 1999).
Several worries, however, have recently been expressed about the checklist strategy. For example, Steve Black (1999) suggests that "lists of criteria may impede student understanding because it is better to have them grapple with the meaning of quality on their own." However, there is an even more serious problem with the checklist strategy. Namely, it is not clear that the proposed indicators are even correlated with accuracy.
Checklists can only help Internet users to distinguish accurate from inaccurate information if the proposed indicators are correlated with accuracy (or if the absence of the indicators is correlated with inaccuracy). However, there is currently no empirical evidence that such a correlation exists. These checklists are typically generated based only on what "web experts" believe to be indicators of accuracy. As a result, according to Alejandro Jadad and Anna Gagliardi (1998), "many incompletely developed instruments to evaluate health information exist on the Internet. It is unclear, however, whether they should exist in the first place, whether they measure what they claim to measure, or whether they lead to more good than harm."
The present article reports on two empirical studies that attempt to determine whether several proposed indicators are correlated with accuracy. Both studies focus on information on a particular medical topic. The first study looks at information on the Internet on treating children with fever. (Impicciatore et al. 1997 studied the accuracy of web sites on this topic. For further details on the fever study reported here, see Fallis and Frické 2002.) The second study looks at information on the Internet on Ewing's Sarcoma (a type of cancer). (Biermann et al. 1997 studied the accuracy of web sites on this topic.) These studies are part of a larger project to help Internet users distinguish accurate from inaccurate information on the Internet (see Fallis 2000).
The two studies described in this article used the very same methodology. First, web sites on a particular medical topic were collected using major Internet search engines (Yahoo [HREF8], AltaVista [HREF9], Google [HREF10]). Search terms were used that an average person who needed information on this topic would be likely to try (e.g., "fever AND treatment AND child").
Second, an accuracy instrument was used to evaluate the accuracy of web sites on this topic. This instrument was developed by consulting authoritative sources on the topic (El-Radhi and Carroll 1994, DeVita et al. 1997, Berkow 1997). In addition, the instrument was checked by a medical doctor. This is the same technique that has been used by previous studies to evaluate the accuracy of health information on the Internet (see, e.g., Impicciatore et al. 1997).
This instrument consisted of a comprehensive list of questions on the particular medical topic under investigation. For each question, a web site received a score of 1 for a correct answer, a score of 0.5 for no answer, and a score of 0 for an incorrect answer (cf. Connell and Tipple 1999). These scores were then combined to give each web site an overall accuracy score between 0 and 1. Web sites with an overall accuracy score on or above the median were deemed to be accurate. Web sites with an overall accuracy score below the median were deemed to be inaccurate.
Third, an indicator instrument was used determine whether several proposed indicators of accuracy were present on the web sites. The proposed indicators were taken from published checklists for evaluating the quality of health information on the Internet (Ambre et al. 1997, Silberg et al. 1997). In particular, each web site was checked for the domain of the URL, currency, the HONcode logo (see below), advertising, author identification and qualifications, copyright, contact information, spelling errors, exclamation points, references to peer-reviewed medical literature, and the number of inlinks (see below).
A couple of these proposed indicators may require further comment. The HONcode logo is displayed on medical web sites that voluntarily agree to abide by the Health on the Net Foundation's Code of Conduct [HREF11] for publishing medical information on the Internet. The number of inlinks is the number of other web sites that have hyperlinks to the web site in question.
Finally, for each of the proposed indicators, a contingency table was constructed:
| Web site is accurate. | Web site is inaccurate. | |
| Indicator X is present | # of web sites | # of web sites |
| Indicator X is not present | # of web sites | # of web sites |
A chi-square test for independence was then used to determine if there was a correlation between the proposed indicator and the accuracy of the web sites (or between the absence of the indicator and inaccuracy). A proposed indicator was deemed to be correlated with accuracy if the chi-square probability was less than 0.05 (and the expected cell frequencies were greater than 5). In order to confirm such a result, the test was then repeated using all possible dividing lines between accurate and inaccurate web sites.
100 web sites on the treatment of fever in children were found. The accuracy scores for these web sites ranged from 0.55 to 0.75 with a median of 0.65. (Accuracy scores have been rounded to two decimal places.) 50 web sites had overall accuracy scores on or above the median and were deemed to be accurate. 50 web sites had overall accuracy scores below the median and were deemed to be inaccurate.
In the fever study, only three proposed indicators were found to be correlated with accuracy. When the chi-square test for independence was repeated using different dividing lines, these three indicators were always found to be correlated with accuracy.
Web sites that displayed the HONcode logo were more likely to be accurate than inaccurate. In fact, the HONcode logo was almost four times more likely to be displayed on an accurate web site than on an inaccurate web site. The probability that the HONcode logo is not correlated with accuracy (i.e., the chi-square probability) is only 0.021.
| Web site is accurate. | Web site is inaccurate. | |
| HONcode logo is present | 11 web sites | 3 web sites |
| HONcode logo is not present | 39 web sites | 47 web sites |
Web sites with an organization domain (e.g., kidshealth.org) were more likely to be accurate than inaccurate. The chi-square probability for organization domain is 0.047.
| Web site is accurate. | Web site is inaccurate. | |
| Organization domain | 19 web sites | 10 web sites |
| Other | 31 web sites | 40 web sites |
Web sites that claimed copyright were more likely to be accurate than inaccurate (and those that do not claim copyright were more likely to be inaccurate than accurate). The chi-square probability for copyright is 0.008.
| Web site is accurate. | Web site is inaccurate. | |
| Copyright | 37 web sites | 24 web sites |
| No copyright | 13 web sites | 26 web sites |
Many commonly proposed indicators of accuracy, however, were not found to be correlated with accuracy. For example, the fact that the author has medical credentials, that the web site is current, that the web site has a large number of inlinks, or that there is no advertising on the web site were not found to be correlated with accuracy. (A web site was deemed to be current if the date of the last update was given and this date was within the last year.)
| Web site is accurate. | Web site is inaccurate. | |
| Author is an MD | 12 web sites | 10 web sites |
| Other | 38 web sites | 40 web sites |
| Web site is accurate. | Web site is inaccurate. | |
| Current | 9 web sites | 7 web sites |
| Not current | 41 web sites | 43 web sites |
| Web site is accurate. | Web site is inaccurate. | |
| No advertising | 34 web sites | 32 web sites |
| Advertising | 16 web sites | 18 web sites |
47 web sites on Ewing's Sarcoma were found. The accuracy scores for these sites ranged from 0.38 to 1.0 with a median of 0.81. 24 web sites had overall accuracy scores on or above the median and were deemed to be accurate. 23 web sites had overall accuracy scores below the median and were deemed to be inaccurate.
In the cancer study, only one proposed indicator was found to be correlated with accuracy. When the chi-square test for independence was repeated using different dividing lines, this indicator was always found to be correlated with accuracy.
Web sites with a commercial domain (e.g., drkoop.com) were more likely to be accurate than inaccurate. The chi-square probability for commercial domain is 0.027.
| Web site is accurate. | Web site is inaccurate. | |
| Commercial domain | 15 web sites | 7 web sites |
| Other | 9 web sites | 16 web sites |
Checklists can only help Internet users to distinguish accurate from inaccurate information if the proposed indicators are correlated with accuracy. The two empirical studies described in this article sought to determine whether several proposed indicators—taken from published checklists for evaluating the quality of health information on the Internet—are correlated with accuracy. In the case of most of these indicators (e.g., the author having medical credentials, currency, lack of advertising), the two studies failed to find a correlation with accuracy.
This result has been confirmed by the relatively few studies that have also undertaken to empirically test these published checklists. For example, in a study on the accuracy of information on depression, Kathleen Griffiths and Helen Christensen (2000) found that "currently popular criteria for evaluating the quality of websites were not indicators of content quality." (This study combined several proposed indicators of accuracy into a single index and checked for a correlation between this index and content quality.) Also, in a study on the accuracy of information on incontinence, Hogne Sandvik (1999) found that "the number of links to a site was not an indication of the quality of its contents." (This study only looked for a correlation between the number of inlinks and content quality.)
Even so, it must be noted that the failure to find a correlation does not conclusively establish that these indicators are not correlated with accuracy. In the cancer study especially, it is likely that the failure to find a correlation was partly due to the small size of the population. (Only 47 web sites on the fairly specialized topic of Ewing's Sarcoma could be found. As a result, for most proposed indicators, expected cell frequencies were less than 5.) Also, several proposed indicators appeared so infrequently on web sites (e.g., only 4 out of the 100 web sites on treating children with fever cited peer-reviewed medical literature) that no conclusions could really be drawn.
Of course, a few proposed indicators were found to be correlated with accuracy in the two empirical studies described in this article. In particular, web sites on treating fever in children that display the HONcode logo are more likely to be accurate than inaccurate. This suggests that compliance with the Health on the Net Foundation's Code of Conduct does tend to increase the accuracy of a web site.
While checklists can only help Internet users to distinguish accurate from inaccurate information if the proposed indicators are correlated with accuracy, it is important to point out that such a correlation is not sufficient to ensure that checklists will "lead to more good than harm." There are several other conditions that proposed indicators of accuracy must satisfy.
First, it must be difficult for authors of web sites with inaccurate information to display the proposed indicator. As James Perry (1986) notes, "when an evaluation technique has been developed and tested, it will become familiar to the evaluatees as well as the evaluators. Thus, its utility will decrease as evaluatees change their practices to gain higher evaluations." For example, it would be a simple matter for the author of a web site with inaccurate information to claim copyright in order to make the web site appear to be accurate. (It is not that difficult to get a URL with an organization or a commercial domain either.) By contrast, it would be difficult for the author of a web site with inaccurate health information to display the HONcode logo. The Health on the Net Foundation has procedures in place [HREF12] that are designed to prevent web sites that do not comply with their Code of Conduct from displaying the HONcode logo.
Second, it must be fairly easy for Internet users to determine whether the proposed indicator is present on web sites. Checklists will not help Internet users to distinguish accurate from inaccurate information on the Internet if they are too troublesome to use. For example, doing an extensive background check on the author of a web site might be a good way for Internet users to determine whether they should rely on the information, but it would be prohibitively expensive. By contrast, the HONcode logo has the advantage of being easy for Internet users to check for.
Third, and perhaps most importantly, Internet users have to put the right amount of faith in these indicators of accuracy. If Internet users overestimate the reliability of indicators of accuracy, checklists may do more harm than good. Since many commonly proposed indicators do not appear to be indicators of accuracy, it is quite likely that the checklist strategy is promoting such overestimation. In fact, such overestimation could be a problem even in the case of the HONcode logo. Internet users might not be aware that, while the fact that a web site displays the HONcode logo is some evidence that the web site is accurate, it is not a guarantee.
This final condition on indicators of accuracy provides further impetus for empirical studies of the sort described in this article. Checklists for evaluating web sites should not just tell Internet users which indicators of accuracy to check for; they should also tell Internet users exactly how much faith to place in these indicators. But in order to do this, information professionals need to empirically determine exactly how reliable these indicators of accuracy are (i.e., how much more likely they are to be displayed on accurate web sites than on inaccurate web sites).
Finally, more important than the specific results of these two studies is the fact that these studies implement an effective technique for identifying indicators of accuracy. In fact, this Indicators of Accuracy technique can be used to identify indicators of accuracy for any type of information on the Internet, not just health information (see Fallis 2000). In order to identify indicators of accuracy for other types of information (e.g., news, investment information, information about politics, information about consumer products), further empirical studies using this technique need to be carried out. Disseminating the results of such studies has the potential to greatly improve people's ability to evaluate the quality of Internet sources.
It is important that Internet users be able to distinguish accurate from inaccurate health information. Toward this end, information professionals have developed checklists for evaluating the quality of health information. Such checklists can only help Internet users if the proposed indicators of accuracy really do indicate accuracy. However, there is an effective technique for identifying indicators of accuracy. One study that implemented this Indicators of Accuracy technique indicates that the HONcode logo is more likely to be displayed on web sites that contain accurate health information. Many commonly proposed indicators of accuracy (e.g., the author having medical credentials, currency, lack of advertising), however, were not found to be correlated with accuracy.
This research was funded by a Research Grant Award from the Association for Library and Information Science Education [HREF13] and a Faculty Research Grant from the University of Arizona. Clayton Curtis, MD, PhD, checked the accuracy instruments. Graduate assistants in the School of Information Resources and Library Science at the University of Arizona (Tracy Cook, Virginia Cullen, Michelle Drumm, Alba Fernandez, Robert Frasier, Elizabeth Gouwens, Melinda Hardman, Andrew Kaplan, and Ping Situ) helped to collect the data. Anita Coleman, Margaret Higgins, Gerrard Liddell, and Kay Mathiesen provided helpful suggestions.
Alexander, J. E. and Tate, M. A. (1999). Web Wisdom. Lawrence Erlbaum Associates, Mahwah, New Jersey.
Ambre, J., Guard, R., Perveiler, F. M., Renner, J., and Rippen, H. (1997). "Criteria for Assessing the Quality of Health Information on the Internet." Available online [HREF14].
Berkow, R., Beers, M. H., and Fletcher, A. J. (1997). The Merck Manual of Medical Information. Pocket Books, New York.
Berland, G. K. et al. (2001). "Health Information on the Internet: Accessibility, Quality, and Readability in English and Spanish." in JAMA v.285 p.2612-21.
Biermann, J. S., Golladay, G. J., Greenfield, M. L. V. H., and Baker, L. H. (1999). "Evaluation of Cancer Information on the Internet." in Cancer v.86 p.381-90.
Black, S. (1999). "Obstacles to Teaching Novice Researchers How to Evaluate the Quality of World Wide Web Resources." in Vincent and Norman (eds.) "All That Glitters": Prospecting for Information in the Changing Library World p.17-31.
Connell, T. H., and Tipple, J. E. (1999). "Testing the Accuracy of Information on the World Wide Web Using the AltaVista Search Engine." in Reference & User Services Quarterly v.38 p.360-68.
Cooke, A. (1999). Neal-Schuman Authoritative Guide to Evaluating Information on the Internet. Neal-Schuman, New York.
DeVita, V. T., Hellman, S., and Roseberg, S. A., (eds.) (1997). Cancer: Principles and Practice of Oncology. JB Lippincott, Philadelphia.
El-Radhi, A. S. and Carroll, J. (eds.) (1994). Fever in Pediatric Practice. Blackwell Scientific, Oxford.
Fallis, D. (2000). "Veritistic Social Epistemology and Information Science." in Social Epistemology v.14 p.305-16.
Fallis, D. and Frické, M. (2002). "Indicators of Accuracy of Consumer Health Information on the Internet." in the Journal of the American Medical Informatics Association v.9 p.73-79.
Fox, S. and Rainie, L. "The Online Healthcare Revolution: How the Web Helps Americans Take Better Care of Themselves." Available online [HREF15].
Griffiths K. M. and Christensen, H. (2000). "Quality of Web Based Information on Treatment of Depression: Cross Sectional Survey." in the British Medical Journal v.321 p.1511-15.
Impicciatore, P., Pandolfini, C., Casella, N., and Bonati, M. (1997). "Reliability of Health Information for the Public on the World Wide Web: Systematic Survey of Advice on Managing Fever in Children at Home." in the British Medical Journal v.314 p.1875-79.
Ip, A., Currie, M., and Morrison, I.(2001). "Resource Synergy: Improving the Discovery of Quality Resources to Support an Information-Dense Learning Environment." in the Proceedings of AusWeb01, Available online [HREF16].
Jadad, A. R. and Gagliardi, A. (1998). "Rating Health Information on the Internet: Navigating to Knowledge or to Babel?" in JAMA v.279 p.611-14.
Middleton, M., Edwards, S. L., and Collins, J.(1999). "Internet Site Evaluation Tool: Students Professional Practice Experience Works Towards the Development of an Internet Tool." in Education for Library and Information Services: Australia v.16 n.2 p.5-17.
Oliver, K. (1998). "Evaluating the Quality of Internet Information Sources." Available online [HREF17].
Perry, J. A. (1986) "Titular Colonicity: Further Developments." in JASIS v.37 p.177-78.
Price, S. L. and Hersh, W. R. (1999). "Filtering Web Pages for Quality Indicators: An Empirical Approach to Finding High Quality Consumer Health Information on the World Wide Web." in the Proceedings of the American Medical Informatics Association Annual Symposium p.911-915.
Sandvik, H. (1999). "Health Information and Interaction on the Internet: A Survey of Female Urinary Incontinence." in the British Medical Journal v.319 p.29-32.
Silberg, W. M., Lundberg, G. D., and Musacchio, R. A. (1997). "Assessing, Controlling, and Assuring the Quality of Medical Information on the Internet." in JAMA v.277 p.1244-45.
Smith, A. (2002). "Evaluation of Information Sources." in The Information Quality WWW Virtual Library. Available online [HREF18].
Vincent, S. and Norman, S. K. (eds.). (1999). "All That Glitters": Prospecting for Information in the Changing Library World. JAI Press, Stamford, Connecticut.
Martin Frické, Don Fallis, © 2002. The authors assign to Southern Cross University and other educational and non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The authors also grant a non-exclusive licence to Southern Cross University to publish this document in full on the World Wide Web and on CD-ROM and in printed form with the conference papers and for the document to be published on mirrors on the World Wide Web.