Daniel Rep, Honours student in 1999 in the School of Computing and Mathematics, Deakin University
Dr Andrew Treloar [HREF1], Manager Web and Internet Facilities, Information Technology Services [HREF2], PO Box 28C, Monash University [HREF3], Victoria, 3800. Andrew.Treloar@its.monash.edu.au
This paper examines the effect the various types of Web site navigation systems have on Web users today and their preferences towards them. More specifically, it considers whether there are any differences between the five systems tested and whether users prefer one system over another. An experiment was developed, testing 25 university students for their ability to answer online questions using the different navigation systems and gaining their responses towards the systems via paper based questionnaires. The experiment results suggest that there are major differences in the time subjects took to answer the online questions using the various systems and from their responses, they have a preference for one system over another.
Given the current state of web technology, most user interactions with Web pages involve navigating hypertext links between and within documents. The main interface problem in Web sites is the lack of a sense of where you are within the local document set. Clear, consistent icons, graphic identity schemes, and graphic or text-based overview and summary screen can give the user confidence that they can find what they are looking for without wasting time. These features can be summarised as mechanisms to support navigation. If one looks at the theoretical underpinnings for such mechanisms, they lie in the disciplines of Usability and Human-Computer Interaction.
In fact it is possible to envisage a series of gradually decreasing concentric circles, moving from general to specific. On the outside is the area of Usability, related to computer and non-computer artifacts. Contained within that is Human Computer Interaction (HCI); this deals with the interaction between users and computer systems. A sub discipline within HCI is Web Usability; usability and HCI techniques as applied to Web document sets and applications. Web Navigation is one aspect of Web Usability.
Usability and HCI are well established disciplines with a rich literature that significantly predate the arrival of the Web. The application of existing HCI theory to the Web is much more recent, and there is very little formal work on Web Navigation.
Before presenting the research design and results, it may be worth briefly revisiting some of the the main themes in the research literature.
Nielsen (1990) has defined usability as being associated with five parameters:
From a user's perspective usability is important because it can make the difference between performing a task accurately or not, completing a job or not, between enjoying a process or being frustrated. From a developer's perspective usability is important because it can mean the difference between success and failure. Given a choice, people will tend to buy systems that are more usable. From a management point of view, software with poor usability can reduce a productive workforce to a level of performance worse than without computers at all.
In all cases, lack of usability can cost time and effort, and can greatly determine success or failure of a system.
HCI is a particular application of usability principles to the design and construction of computer systems that interact with users. When properly applied, HCI requires knowledge about:
For those interested in reading further, an extensive HCI bibliography is available online [HREF4].
It is possible to apply Nielsen's overall usability parameters to hypertext systems such as the Web:
Before discussing navigation support systems, it is important to distinguish between the two forms of information seeking: navigation and browsing. The main distinction between navigation and browsing is based on the user goals. In browsing, users explore the available hypertext to get a general idea about one or several topics. In navigation, users have a specific goal in mind. Navigation or browsing is often difficult. In a complex hypertext system; the structural complexity may be overwhelming for the user and impeded effective navigation. Users can quickly become disorientated and navigation and visualization tools are often needed (Valdez et. al. 1988). Some of these tools include the ability to backtrack, history lists, bookmark facilities, and overview diagrams or maps (Nielsen, 1995).
According to Fleming (1998) using the Web is all about verbs: searching, choosing, shopping, chatting, downloading, backing up, charging ahead. What we do on the Web is based on action. In general, most people need to know some basic things in order to orient themselves, to navigate around the virtual spaces (both document spaces and application spaces) within which they find themselves. This is true whether they're navigating a building or a Web site. These basic navigation needs to include answers to the following questions:
Navigation that works should therefore:
This research sought to investigate the effectiveness of different navigation schemes for a particular corpus of hypertext. This research is interesting in that it seeks to test the usability of these navigation schemes using a range of measures and thus try to validate some of the assertions commonly made about Web navigation. The research sought to focus on two different measures: differences between the navigation systems in the time users took to answer the online questions, and determining if the users had a preference for one system over the other. While there is quite a lot of research into usability and HCI, there is much less on Web usability and even less on Web navigation. What there is tends to talk in general terms. There is little in the literature dealing with the analysis of user behaviour in the use of navigation systems. One motivation for this research was to plug this gap.
Another intention was to see if there was a difference between user ratings for a system and the time taken to answer questions. Intuitively one might expect users to prefer the system that allowed them to locate information most quickly. Would this be the case?
A third motivation was to provide a model for others to use in similar research, and to try to identify deficiencies in the way this study sought to answer these questions.
Finally, this study sought to provide some quantitative and qualitative data on use of particular Web navigation systems.
The research design, in overview, consisted of the following phases:
The first task was to identify five different navigation systems to compare. Those chosen were Non-Structured (as a minimal position to be used as a basis of comparison), Sequence of pages, Two Level Non-Framed, Two Level Framed and a Graphical Site Map. Each of these can be found in sites on the Web. As far as one can tell without performing significant additional data collection they would appear to cover most of the navigation options commonly presented by web sites.
The Non Structured System was a single Web page with all the content sectioned by title. The scroll bar was the subjects' only method of browsing the information of the site. This is the equivalent of a printed page (technically, it is closer to a scroll , the leading edge information presentation technology around the destruction of the library of Alexandria, but never mind).
The Sequence of Pages System had each title section as a separate Web page linked to the next in order by previous and next text links with a start page link taking users back to the beginning page. Links were provided both at the top and bottom of the page.
The Two Level Non Framed System incorporated a table structure for the site with first level headings appearing at the top of the screen as green graphical button links. Upon selecting a section, a second level of heading appear on the left hand side of the screen as blue graphical button links. The content relevant to each section appears in the main section of the page. A start page link was also provided taking users back to the beginning page. Scrolling was required to reach content further down the page with the table structure having the navigation fixed at the top of the page.
The Two Level Framed System is an exact copy of the Two Level Non Framed System, however, it incorporates frames to allow the graphical button links to remain visible to users at all times.
The Graphical Site Map System uses an image map to list all levels of headings with the user having to click on the desired heading to enter the content. Once the user has entered, a series of previous, start and next text links are incorporated to allow for sequential movement.
Each of the above navigation systems was then implemented for a corpus of hypertext. Each implementation contained the same content and the information was placed in the same order to ensure valid comparisons. The content used was the manual for the popular Point of View (POV)-Ray graphical rendering software. This was chosen because it was:
In fact, it proved only necessary to use one section of POV-Ray's help documentation, the Beginning Tutorial. This section had three levels of information within it.
The five different web sites were placed on a dedicated server isolated from the rest of the network. This ensured that access delays were not a factor in navigation times and also meant that careful logging of access times was possible. Access privileges were restricted and subjects could not view the Web sites in advance or even access the server.
Subjects were selected at random from students enrolled in the first year unit SCC110: Multimedia Technology offered at Deakin University. One of the researchers was involved in demonstrating for this unit. In order that no coercion could be imputed, only students not taught by that researcher were approached. Students were invited via email to take part. No inducement was offered. After truly heroic recruitment efforts a total of 25 students responded.
In order to get the subjects to exercise each of the navigation systems, they were asked to locate the answers to a series of questions. The questions asked were taken primarily from the third level of the information structure. Each question was worded differently yet contained a form of consistency by asking the users in each case to find an answer rather than fill in the blanks. Topics covered in the Beginning Tutorial section of the POV-Ray manual were Our First Image, Simple Shapes, CSG Objects, Advanced Shapes, The Light Source, Simple Texture Options, Advanced texture Options, Using the Camera and Using Atmospheric Effects. Questions were distributed across these topics. A significant amount of proofreading was carried out in order to determine if the content was relevant and that users had a realistic chance of reaching the answers in a reasonable amount of time.
Users were given five questions to answer, one to be answered using each navigation system (five questions = five systems). To begin with they clicked on a link to the navigation system assigned to a question. Once they had located the answer they clicked on another link. The time taken by subjects browsing the Web sites was recorded by the Web server and stored in its log files. The server actually recorded the time and date pages were accessed. It was simply a matter of subtracting one time from the other to obtain an overall navigation (plus reading) time for that question.
In order to assess how the subjects responded to the navigation systems, they were asked a series of qualitative questions for each system. These included 5-point Likert scale assessments of:
In addition, they were given space to comment on what made the Web site easy or difficult to navigate, as well as making other comments or suggestions.
in order to ensure that the data from this experiment was as useful as possible, a number of careful design decisions were made. These took some time to devise and may be useful for other researchers.
Many (most?) user interfaces become easier to operate after some exposure. While the systems being investigated are relatively common in a generic sense, we wanted to try to remove any bias associated with encountering the navigation interfaces for the firs time. The subjects were therefore given two sets of questions to answer: the initial set of five (one for each system) and then a second set. The expectation was that they would find it easier to use each navigation system the second time through because they had now been exposed to it. The questions were different in each set, ensuring that memorising the location for an answer would be of no use.
It is possible that a particular navigation system might rate poorly simply because subjects were always assigned difficult questions to answer using that system. In order to avoid this, questions were randomly assigned to different navigation systems across the entire subject pool. This means that a given question could be associated with any of the navigation systems and might occur in the first set of questions or the repeat set.
In total there were 14 males and 11 females. The majority of subjects were between 18 and 25. Only 1 student was between 26 and 35 years of age. Eleven subjects were studying towards a Bachelor of Computing (Information Management) degree with 7 a Bachelor of Computing (Applied Computing). There was 1 response from each student studying towards an Arts/Science - Indonesian/Multimedia, Bachelor of Arts (Media Arts), Bachelor of Teaching/Bachelor of Computing, Secondary Teaching/Applied Science, Bachelor of Arts/Bachelor of Science and Multimedia degrees. There was only 1 student who had achieved other qualifications that were applicable and that included a Diploma of Information Technology.
It was important to determine the subjects' level of familiarity with computers in general. Most subjects had been using computers for some time. Only one had used a computer for less than a year, with over half (14 out of 25) the subjects having used a computer for four or more years.
Because use of the Internet would imply exposure to Web sites (and hence their navigation systems) subjects were also asked how long they had used the Internet for. Here the average length of time was shorter than for computer use as a whole. Nearly half (11 out of 25) had used the Internet for two years or less. The longest reported period was four years. Subjects were also asked to self-assess the level of Internet expertise. None admitted to being Beginners, 16 said they were Intermediate, 3 said they were Intermediate/Expert, and 6 said they were Expert.
Before analyzing the online results it is important to note that 10 different questions were asked with varying ease and difficulty for the users in finding the answers as some appeared on the first level of content and others in the third level. Therefore time taken to find various answers to these questions differed considerably to the extent that the results would be considered irrelevant. In order to summarise a large amount of data, the results have been averaged across all subjects and questions.
For the first exposure to the navigation systems, the quickest navigation system was the Graphical Site Map with an average time to locate the answer of 1.13 minutes. Next came the Two-Level Framed version with 1.36 minutes, and the Two-Level Non-Framed with 2.176 minutes. A long way behind were the Sequence of Pages with an average time of 4.87 minutes and the Non-Structured site with an average of 6.09 minutes.
When the subjects were exposed to the same interfaces again with a different set of questions, the answers were somewhat different. The fastest interface on average was still the Graphical Site Map with an average time of 1.24. Next came the two-level variants but in the reverse order: Two-Level Non-Framed version with 1.49 minutes and the Two-Level Framed version with 2.04 minutes. The last two sites had also swapped order: the Non-Structured version took an average of 3.51 minutes and the Sequence of Pages took 3.67 minutes.
Subjects were also asked to subjectively rate each navigation system. The questionnaires for each of the systems was identical. The systems will be discussed starting with the most favoured.
Consistent with the retrieval times, the majority of subjects gave the Graphical Site Map system the highest rating for most questions, in stark contrast to the responses given for some of the alternatives.
Next subjects preferred one of the two variants on the Two Level (Framed and Non Framed) systems. responses indicate that the subjects find this system easier to use than the two previously mentioned systems. There was a slight preference for the Non-Framed version.
Ratings given by subjects for the Sequence of Pages system show that they found it difficult to navigate and find what they need using the system and rated the elements of site navigation, site design, and information organization as not very effective. However, the ratings seem to indicate that they prefer this system a little more over the Non Structured system with a few more subjects finding the system somewhat effective and somewhat easy.
Lastly, the majority of subjects found the Non Structured system difficult to navigate and rated elements within the site very poorly.
For the Non Structured system, typical subject comments were:
"information overload made it difficult to find what was needed."
"It was difficult to navigate because you had to read a large amount of information to find what you were looking for."
"It was just too much on one page. You had to go through every thing from the start to end so that you don't miss out on the information you require."
For the Sequence of Pages system, subjects said things like:
"once again the navigation system made it difficult to find what you needed first."
"You still had to start from the beginning and scroll to the end...yet it is a little better than the non structured."
"it was easy in that I could see what the next page was and follow the link until I found the general subject area I was looking for."
For the Two Level Non Framed system, comments were:
"I could easily pinpoint the subject I was looking for which made it easier to get to!"
"The menu/ sub menu system allowed the information to be there in front of you quickly."
"It is much better than previous systems as you can simply read the question and then navigate your way through the site to find your answer."
As for the Two Level Framed system
one subject was quoted as saying that "the graphic interface made finding the subject much easier."
Another said they "could go to a particular section easily,"
another said that "framed options made it more accessible in less time."
For the Graphical Site Map system,
"information was easy to find. One could search through for the relevant key word and then find the appropriate page."
"This was easy to navigate as you had a content page which showed you where everything was in topics,"
"the best navigation system of all"
Shneiderman (1997) compared paper and hypertext versions of historical articles. Although the results favored paper based text, hypertext was helpful when users had to "jump around" in the information. McKnight, Dillon, and Richardson (1991) presented the same text in four formats: two hypertext and two linear. There were no differences in task completion time. However, users answered questions faster and more accurately with linear formats. Users spent surprisingly little time following hypertext links, but a great deal of time jumping back and forth between text and indices. These two studies favored linear presentation of information. Experiments with SuperBook, however, showed that hypertext combined with text search capabilities was superior to conventional paper documentation (Egan et al., 1989). It is important to note that in Egan et al.'s (1989) comparative study, it was the third revision of SuperBook, designed iteratively based on formative evaluation with the users, showed improvement in performance and user satisfaction. This study clearly demonstrated the impact of understanding the nature of user tasks and user involvement on the usability of hypertext.
In addition to understanding of user tasks, the effectiveness of hypertext also depends on its user interface. This importance of user interface design is evident in the work by Leventhal et al. (1993). In the design of HyperHolmes, the user interface was revised based on usability testing. The changes included the following: (1) tiled windows instead of overlapping windows, (2) removing "outgoing links," (3) simplifying search features, and (4) providing direct access to an overview page. These seemingly minor changes led to substantial improvement in subjects' search performance and emphasized the importance of the user interface in the success of hypertext systems.
Hypertext, however, is not always a good choice. As suggested by Whalley (1993), hypertext is "fundamentally flawed" for expository text, where the main concern for the reader is coherence and cohesion. The hypertext format would fragment the presentation.
Several studies have compared different implementations of hypertext documents to observe the effects of various organizing principles or access mechanisms on performance as is stated in Hypertext in Context. This is an important area. The term hypertext does not refer to a unitary concept. When comparisons are said to be made between hypertext and paper documents they are really being made between certain implementations of hypertext and standard versions of paper texts. Each implementation consists of one designer's ideas about how to build the interface between users and information. To make general claims or draw conclusions about the wider relevance of hypertext in such circumstances is problematic. However, studies comparing varying implementations can shed some light on what constitutes good hypertext.
Simpson and McKnight (1990) created eight versions of a hypertext document on plants, manipulating the contents list (hierarchical or alphabetic), presence or absence of a current position indicator and presence or absence of typographic cues. Subjects, made up of researchers and subjects, were required to read the text until they felt confident they had seen it all and were then required to perform 10 information location tasks before attempting to construct a map of the document structure with the cards. Results showed that readers using a hierarchical contents list navigated through the text more effectively and produced more accurate maps of its structure than readers using an alphabetic index. The current position indicator and additional typographic cues were of limited utility.
Wright and Lickorish (1990) compared two types of navigation system for two different hypertexts. The navigation systems were termed Index navigation where the reader needed to return to a separate listing to specify where to move next, and Page navigation where the reader could jump directly to other pages from the current display. The two text were on house plants and supermarket prices. Twenty-four subjects read both hypertexts, 12 per navigation system, answering multiple choice questions with the plants text and a variety of `GoTo', compared and compute tasks with the supermarket text.
From their results they concluded that each navigation system had certain advantages in particular situations. For example, the paging navigation system may appear burdensome but was found to be beneficial with the house plants text as it coupled navigation decisions with an overview of the text's structure. However, such activity with the tasks performed on the supermarket text (where decisions about where to go were not an issue because the question provided such information) turned out to be an extra load on working memory. As Wright and Lickorish state, "authors need to bear in mind both the structure inherent in the content material and the tasks readers will be seeking to accomplish when they are designing navigation systems for hypertext.
The results of this experiment provided answers to the research questions presented at the start of this paper and they allow for some interesting and meaningful conclusions to be made.
Question 1: Is there a difference between the navigation systems in the time subjects take to answer the online questions?
Results from the questions asked show that there is a difference between navigation systems in the time it took subjects to answer the questions with the Graphical Site Map system producing, on average, the lowest times. In six out of the ten questions, the Graphical site map produced the lowest time. Results from the comparison of Web site navigation systems based on the questions asked, certainly indicate that there are considerable differences between the different systems. This is primarily the case between the Non Structured and Sequence of Pages system against the Two Level Non Framed, Framed and Graphical Site Map systems.
Question 2: Do the ratings given by subjects indicate a preference for one system?
Based on ratings given by subjects for a number of questions regarding the ease of navigating and effectiveness of the Web sites, the Graphical Site Map system produced the highest rating. This was closely followed by the Two Level Framed system, Two Level Non Framed, Sequence of Pages and the Non Structured systems.
Future research that could be attempted in this same field and based on the same experiment would be to try and recruit a larger sample of subjects. This would produce a higher number of responses in order to perform statistical calculations such as cross tabulations to more specifically analyze the various Web site navigation systems and preferences towards them. Another idea would be the designing of more navigation systems to test, as although the ones in this experiment are some of the most common on the Web today, others exist that could also provide interesting results. A reduction in the length of time subjects took in answering a question would greatly improve the experiment as would the design of online questionnaires rather than the paper based questionnaires the subjects had to complete. All these factors could provide for future research involving the same experiment as well as act as improvements to the experiment that was completed.
I suspect some will question the value of this research. In fact, one counter-argument is that we only need to watch the process of Darwinian evolution on the Web to see which interfaces succeed and which fail.
<RESEARCH ACADEMIC POLEMIC>However, unless we do things like this (more sophisticated perhaps, with a wider audience perhaps, with more subjects perhaps) we will have no hard data on which to base usability decisions. Much of the current writing on Web usability (what there is) is based on research over a decade old. We need to start doing the basic research now to build on in the future! </RESEARCH ACADEMIC POLEMIC>
Fleming, J. (1998). Web Navigation: Designing the User Experience. OÍReilly, Beijing.
Egan, D. E., Remde, J. R., Gomez, L. M., Landauer, T. K., Eberhardt, J. and Lochbaum, C. C (1989)., "Formative Design-Evaluation of SuperBook Research Contributions" in ACM Transactions on Information Systems v.7 n.1 p.30-57.
Leventhal, L. M., Teasley, B. M., Instone, K., Rohlman, D. S., and Farhat, J. (1993). "Sleuthing in HyperHolmes: An Evaluation of Using Hypertext vs. a Book to Answer Questions" in Behaviour and Information Technology 1993 v.12 n.3 p.149-164.
McKnight, C., Dillon, A. & Richardson, J. (1991). Hypertext in Context. Cambridge University Press, Cambridge.
Nielsen, J. (1995). "The Future of Hypertext" in interactions 1995 v.2 n.2 p.66-78. Available online [HREF5].
Shneiderman, B. (1997). Designing the User Interface: Strategies for Effective Human-Computer Interaction. Addison Wesley Logman, Reading, Massachusetts.
Simpson, A. and McKnight, C. "Navigation in Hypertext: Structural Cues and Mental Maps" in HYPERTEXT II: State of the Art p.73-83
Valdez, F., Chignell, M., and Glenn, B. "Browsing Models for Hypermedia Databases" in Proceedings of the Human Factors Society 32nd Annual Meeting 1988 v.1 p.318-322.
Whalley, P (1993) "An Alternative Rhetoric for Hypertext" In McKnight, C, Dillon, A & Richardson, J (Eds.) Hypertext: A Psychological Perspective Chichester: Ellis Horwood Ltd.
Wright, P. and Lickorish, A. (1990), "An Empirical Comparison of Two Navigation Systems for Two Hypertexts" in HYPERTEXT II: State of the Art p.84-93.
Daniel Rep and Andrew Treloar, © 2000. The authors assign to Southern Cross University and other educational and non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The authors also grant a non-exclusive licence to Southern Cross University to publish this document in full on the World Wide Web and on CD-ROM and in printed form with the conference papers and for the document to be published on mirrors on the World Wide Web.
[ Proceedings ]