Karey Patterson, School of Social Science, Southern Cross University, Lismore, NSW, Australia, 2480. Email: kpatte10@scu.edu.au
Allan Ellis, Senior Lecturer, School of Social Science, Southern Cross University, Lismore, NSW, Australia, 2480. Email: aellis@scu.edu.au
The research discussed in this paper involved analysis of 134 user tests that were conducted on Australian university Web Sites (N=44). Between 3 and 5 users tests for each individual university Web site were performed. The analysis identified that there were some significant relationships between usability scores and Web site features such as search function, clearly targeted information for research, past students and feedback facility areas. The analysis also reinforced the relationship between Web site accessibility and usability and raises important issues for the management of university Web sites.
Over the last decade, there has been considerable research effort directed towards the understanding the effective utilisation of Web sites as a means of providing and disseminating information as well as in the area of e-business and service provision (Conhaim 1992; Cole 1998; Evans 1999; Downey 2000; Faden 2000; Fisher 2000; Khirallah 2000; Kirkwood 2000; Christner 2001; Cohen 2001; Dragan & Behr 2001; Ellis 2001; Collin, Dorbolo & Vandresse 2002) .
The massive growth and scale of the Internet and the amount of online resources has led to an information 'market' overwhelming users with choice (Nielson 2000; Snyder, Molich & Farrell 2000) . As a result usability has assumed a great deal more importance for Web site developers and managers in terms of satisfying users needs (Spool,et al. 1999; Nielson 2000; Snyder, Molich & Farrell 2000) . Universities that are not able to satisfy a users need for information or services, because of poor Web site navigation and design, may experience problems in turning Web site users into enrolled students and hence may lose competitive advantage to other institutions that provide a more usable Web site and relatively easier access to key information and services.
The impact of the Web on Australian and international universities has been significant (Ellsworth 1994; Bushell 1999; Patterson, Ellis & Brice 2000; Kyer 2002) . Universities have adopted the medium as a delivery vehicle, a publishing vehicle, a tool for displaying corporate image and a marketing communication tool (Axelson 2000; Barbian 2002; Deans 2003; Yates 2003) . There is considerable existing research data and discussion relating to online learning in the higher education sector (Ambrose 1996; Abrams & Haefner 1998; Arenson 1998; Berry 1998; Aldrich 2000; Barron 2000; Berry 2000; Abernathy 2001; Altschuler 2001; Arnone 2002) . There are also extensive resources and literature relating to Web based marketing (Carmichael 1998; Economy 1998; Guernsey 2000) there are however there are few published articles and research papers that relate to the provision of marketing or other materials via a university's Web site (Cubbison 1997; Bushell 1999; Carr 2000a; Hipwell 2000) .
There is a considerable body of existing usability literature that offers prescriptive advice for Web site developers (Kantner 1994; Nielsen 1996; Nielsen 1997; Buur & Bagger 1999; Feldman 1999; Spool et al. 1999; Nielson 2000; McCann & Mead 2002) . Usability research and testing have been seen as a difficult area by developers and managers of Web sites (Feldman 1999) . In the hectic and time pressured environment many university Web site managers feel that testing and research are time consuming exercises (Tselios et al. 2001) that while important are given a low priority. This is contrary to the advice available in the literature (Kantner 1994; Nielsen 1997; Spool, Scanlon et al. 1999; Nielson 2000) but nonetheless a reflection of the university Web site managers' busy work environment (Patterson & Ellis 2003) .
The research questions emerged from survey responses that were part of previous research conducted by the authors who interviewed Australian university Web site managers and conducted a pilot study investigating Web site features (Patterson & Ellis 2002; Patterson & Ellis 2003).
A relationship between Web site usability and the presence of certain Web site features and Web site management practices is important to the managers and users of Australian university Web sites. The research conducted in this paper analysed the demographic characteristics as well as computer knowledge and the Internet usage of the Web site usability subjects in order to identify if these variables influenced the usability results.
This paper does not examine in detail concepts relating to Web site accessibility in general. Nielson (2000) states that in order to make accessible Web sites designers should really only have to implement HTML as it was originally intended, to encode meaning and not just appearance. By following the advice of Nielson, designers would be able to design Web sites that are accessible by alternative Web browsers and to facilitate the use of the Internet by disabled users. Alexander (2003) has researched the Web sites of Australian universities with regard to their compliance with W3C accessibility guidelines. She concluded that Australian university Web sites presented barriers to disabled user groups and that even though many of the studied universities had policies governing Web site accessibility, the policies were not enforced or seen to be effective. The reasons for this were unclear but may have been due to poor understanding of accessibility requirements and procedures that assist in design accessible Web sites among designers and poor Web site quality assurance processes (Alexander 2003) .
The surveyed population (N=44) consisted of Australian universities (n=39) and other higher degree granting institutions such as The Australian Film, Television and Radio School, Australian Maritime College, Australian Defence Force Academy (n=3). A list of the studied population is available from the Australian Vice Chancellors Committee Web site [HREF 1]. Two Web sites (The Batchelor Institute of Indigenous Tertiary Education and Christian Heritage College) that were not initially listed on the AVCC site were also included in the study as they were degree-granting institutions and were listed on the Australian Commonwealth Department of Education, Science and Training Web site [HREF 2].
The users were asked to self rate their computer, Internet and informational retrieval proficiency and then were assigned an Australian university Web site from which they were to retrieve a set of required information in a "scavenger hunt" fashion (Spool et al. 1999). The scavenger hunt activities were based on typical student Web site usage. Students were asked to find specific information relating to enrolment dates, international student enrolment procedures, undertaking research at the university. The degree to which they were able to successfully find the required information was indicative of the degree of Web site usability on the tested Web site.
Usability tests were conducted by current, former and potential university students. Questions and tasks related to the users ability to source information such as the facsimile number of the university marketing department, enrolment cut off dates for certain courses, the name of the vice chancellor of the studied university, how many campuses made up the university, what was the international phone number of the switch board of the university. Users had to find answers to the information required (such as the cut off date for 2004 enrolments and procedures for cancelling current enrolment) from the university Web site they were allocated.
At the completion of forty-five minute usability test the users were asked to fill in a post-test that asked them about;
• Their level of user fatigue.
• Their feelings about using the studied site.
• What they would do and what alternative methods they would undertake if they were unable to find the required information via the Web site they just used.
Earlier research (Patterson & Ellis 2002; Patterson & Ellis 2003) collected data through a series of interviews with the Web site managers of the studied population was analysed in conjunction with the usability data. The research project investigating Autsralian university Web site management was split into four phases. This paper discusses the results of phase four (usability tests) however analysis was conducted on this data in conjunction with the data from phase two (interviews with Web site managers) and phase three (assessment of Web site characteristics). During the second phase of research data collection Web site managers were asked questions relating to the following areas:
In addition to the interview data collected the university Web sites were, as part of phase three, assessed with regard to the presence of the following Web site characteristics:
The data from earlier research (Patterson & Ellis 2002; Patterson & Ellis 2003) relating to Web site feature data and Web site manager interview data, were then combined and a sequence of analyses, general linear regression and factor analysis, were conducted in SPSS [HREF 3].
Using linear regression, relationships between variables from the three research phases were adjusted to remove individual differences, tested and several variables emerged as having a significant relationship to usability at the 90% confidence level. Stronger relationships were identified at the 95% confidence level for other studied variables. These findings do provide evidence that in this study university Web site usability is associated with certain Web site characteristics and the Web site management practices of individual universities.
In order to determine whether or not it was legitimate to group the measures of self-rated confidence or to keep them separate, reliability tests were conducted on the data. The reliability test was conducted and the coefficient alpha value .96, was indicative of an extremely positive relationship between each of the self rated competency items and showed that there was a strong correlation between any one of the self rated competency items. This high correlation showed that the five variables were really describing one characteristic and that they were essentially testing the one variable, user self rated competency.
Initially the usability data was tested against the Web site manager interview data. A factor analysis was then conducted on the Web site characteristics and then usability data were tested against the Web site characteristics, resulting in quite high "R" levels, which indicated relationships between the variables. The data were recorded again to gain the additional scale that reflected the scaled data rather than the load data. The effects of individual differences were removed to develop a usability index.
Presence of targeted research information, targeted information for past students, Intranet, contact information, an international phone number and high contrast design on the reviewed Web sites resulted in a significant effect upon the adjusted usability results at 95 % confidence interval.
The search function, feedback function and use of text or images of text variables were strongly associated with usability scores and constitute evidence of impact upon the adjusted usability index.
At the 90% confidence level the variables developed for the Web site manager interviews, overall Web site responsibility , who the Web site managers report to and whether or not the Web site managers use Web site statistics are significant in their relationship with adjusted Web site usability.
Using the general linear model univariate approach in SPSS, all significant variables were then combined into one model to assess, which among the significant variables were significant simultaneously. To achieve this the variables were combined and the least significant variable was removed, one at a time, until the model consisted of only variables that were significant.
The user self-rating section of the usability test identified a correlation between the five tested variables, which were combined to create the one variable, user self-rated competency. User self-rated competency was shown to be a good predictor of users ability to navigate a university Web site and retrieve information.
The relationship between search function, research information, targeted information for past students and an Intranet may be indicative of easy to follow paths for targeted users and improved usability. The presence of contact information and an international phone number may be indicative of considerate design for potential student enquiries and a desire to create trust with users resulting in improved usability. Presence of high contrast (black on white colours used for example) design on the reviewed Web sites and use of text over images of text resulted in a significant effect upon the adjusted usability results at 95 % confidence interval and as such reinforces the importance and concern for high contrast and accessible design with regard to Web site improved Web site usability. This result emphasises the importance of Alexander's (2003) research that discussed the low priority accessible design currently receives in the Australian higher education sector.
At the 90% confidence level the variables developed for the Web site manager assessment, overall Web site responsibility and who the Web site managers report to are due to improved levels of support and resourcing. Whether or not the Web site managers use Web site statistics also has a significant relationship with adjusted Web site usability and may be due to a research ethic and desire to follow user behaviour that has the added effect of consciously or unconsciously leading to development of more usable university Web sites.
This research investigated the relationship between Web site characteristics and features, as well as the Web site management practices of those who are responsible for the university Web site and found that the presence of certain Web site features as well as Web site management practices have a significant relationship to Web site usability.
The implications of the research results support a relationship between accessibility and usability and highlight the importance of developing accessible university Web sites. The research also identified the need to assess Web site management practices and the relationship with Web site usability. It is believed that the research results will have widespread implications for Web site managers that will affect the management of individual Australian university Web sites.
Universities interested in improving the usability of their Web site as well as those considering conducting Web site usability research are advised to focus upon the study of the significant characteristics identified in this research (feedback facility, search function, targeted research and past student information, contact information, high contrast design and international phone number) with regard to their Web sites and to assess their standing with regard to use of Web site statistics and the overall responsibility of their Web site team.
Due to the user tests being undertaken by students universities are also encouraged to include a wide sample of Web site users, including staff and other yet to be identified groups in their usability tests. To ignore the usability results that may be generated by other user groups, such as parents, employers and staff, when conducting a usability study is to jeopardise the validity of any result being interpreted in any way other than indication of student usability.
The results of this study have shown that there are differences between university Web sites based upon their site characteristics, features and how they are managed. The research also identified that user self-rated competency was a good predictor of the ability of users to find and retrieve important Web site content from Australian university Web sites. It identified the risks of not gaining a full understanding of the issues associated with usable Web site development by only examining one area of a Web site when testing usability, technical features and characteristics. The research highlighted the importance of collecting self-rated ability assessment from users when conducting usability research and in examining the usability data in conjunction with that relating to Web site characteristics, features as well as the dynamics and practices of the Web site management team and the relationship between these activities and Web site usability.
The research confirms the relationship between usability and accessible design issues and will hopefully stimulate more interest in and consideration of these areas of university Web site design and management.
Abbey, B. (2000). Instructional and cognitive impacts of Web-based education . Hershey, PA, Idea Group Pub.
Abernathy, D. J. (2000). "E-Learning Enchilada To Go." Training & Development 54 (11): 20.
Abernathy, D. J. (2001). "User-Friendly E-Learning." Training & Development 55 (3): 19.
Abrams, G. & J. Haefner (1998). "S.H.O.W.M.E.: spear-heading online work in mathematics education." T H E Journal (Technological Horizons In Education) 25 (10): 53.
Aldrich, C. (2000). "Customer-Focused E-Learning: The Drivers." Training & Development 54 (8): 34.
Alexander, D. (2003). How Accessible are Australian University Web Sites? AusWeb03, Hyatt Regency Sanctuary Cove, Australia, Southern Cross University Press. ISBN 1 863844 93 7
Altschuler, G. C. (2001). The e-learning curve; online education is constantly evolving, but some things can be sorted out right now. For one, you can't do American pop-culture from Singapore. The New York Times : 13.
Ambrose, D. L. M. (1996). "The campus neighborhood and all its attractions star in World-Wide Web site at U. of Baltimore." The Chronicle of Higher Education 42 (39): A22.
Arenson, K. W. (1998). More colleges plunging into uncharted waters of on-line courses. The New York Times : A16.
Ariew, S. A. (2001). "Learners online." CHOICE: Current Reviews for Academic Libraries 38 (Special Issue): 151.
Arnone, M. (2002). "Navy redesigns course-writing software to enhance online education." The Chronicle of Higher Education 48 (21): A30.
Axelson, M. (2000). "Education portal market takes shape with trios of SIS, e-learning and website integration." Internet Strategies for Education Markets: The Heller Report 5 (12): 1.
Barbian, J. (2001). "Approaches to the Management of E-Learning." Training 38 (5): 34.
Barbian, J. (2002). "E-learning For Dummies. (Web-Based Branding)." Training 39 (11): 20.
Barron, T. (2000). "Customer-Focused E-Learning: The Industry." Training & Development 54 (8): 30.
Bates, A. W. (1997). "The impact of technological change on open and distance learning." Distance Education 18 (1): 93-110.
Berry, J. (1998). Degree Programs Allow Students To Master The Web. InternetWeek : 65.
Berry, J. (2000). Traditional Training Fades In Favor Of E-Learning -- Internet Economy Demands A More Flexible Training Approach. InternetWeek : 33.
Birchard, K. (2001). "European nations promote online education." The Chronicle of Higher Education 47 (33): A46.
Bolan, S. (2001). "Pure E-Learning Solution Leaves Out Student : "You can't expect a person to change how they interact with another human being by putting them in front of a computer"." Computing Canada 27 (22): 32.
Bolch, M. (2002). "E-learning comes of age." Training 39 (9): 114.
Boshier, R., Brand, S., Dabiri, A., Fujitsuka, T., Chiao, Y.J.T. (2001). "Virtual universities revealed: not just a pretty interface." Distance Education 22 (2): 212-232.
Broadbent, B. (2000). "Staying Afloat in the E-Learning Sea." Training & Development 54 (5): 98.
Broadbent, B. (2001). "E-Learning: Strategies for Delivering Knowledge in the Digital Age." T&D 55 (8): 82.
Brown, M. (2000). "Web-based Education: Learning on the Job - Online learning offers advantages to employees who need to acquire on-the-job computer skills." Computing Canada 26 (5): 30.
Bryant, E. (2001). "E-Learning Unveils Distance Education Future." Library Journal 126 (10): 24.
Bushell, S. (1999). World wide ed. The Bulletin with Newsweek . 117: 90.
Buur, J., K. Bagger (1999). "Replacing Usability, Testing with USER DIALOGUE." Communications of the ACM 42 (5): 63.
Carlson, R. D., J. Repman (2002). "Activating web-based e-learning. (Chalk Dust from the Virtual Classroom)." International Journal on E-Learning 1 (2): 7.
Carmichael, M. (1998). "College towns see lower Web costs: developers offer service with less overhead." Business Marketing 83 (9): 38.
Carnevale, D. (2001). "As online education surges, some colleges remain untouched." The Chronicle of Higher Education 47 (24): A41.
Carr, S. (2000). "A new Web site aims to direct students to online courses, but price is steep." The Chronicle of Higher Education 46 (18): A54.
Carr, S. (2000a). "Online learning's 'yellow pages' are bracing for a shakeout." The Chronicle of Higher Education 46 (32): A57.
Carr, S. (2001). "With a national e-university, Britain gets in the online-education game." The Chronicle of Higher Education 47 (49): A27.
Chapnick, S. (2001). "Scotland Does E-Learning." T&D 55 (8): 42.
Charp, S. (2001). "E-Learning." T H E Journal (Technological Horizons In Education) 28 (9): 10.
Christner, T. (2001). "Dollars, Distance, and Online Education: The New Economics of College Teaching and Learning." Library Journal 126 (1): 128.
Coffman, S. (2001). "Distance Education and Virtual Reference: Where Are We Headed?" Computers in Libraries 21 (4): 20.
Cohen, A. (2001). Corporate E-ducation. PC Magazine : 137.
Cole, G. (1998). "Biggest classroom in the world." Times Educational Supplement (4272): B4.
Collin, P., D. Dorbolo, & Vandresse, L. (2002). "E-learning @ Belgacom: here's how one European firm makes e-learning work. (Pass Port)." T&D 56 (8): 54.
Conhaim, W. W. (1992). "Online education graduates to the mainstream." Link-Up 9 (4): 22.
Cubbison, L. (1997). "A heuristic for defining the purpose of a client's WWW site." Business Communication Quarterly 60 (4): 95.
Deans, K. R., von Allman, S. (2003). Are you being served? Full proceedings of AusWeb03, Hyatt Regency Sanctuary Cove, Southern Cross University. ISBN 1 863844 93 7
Downey, G. (2000). "E-learning Is Not Training's 'Silver Bullet' : Companies say cost, time and content are barriers to online education integration." Computing Canada 26 (25): 25.
Dragan, R. V. & M. Behr (2001). E-Learning - Help employees stay knowledgeable through virtual education. PC Magazine : 172.
Economy, P. (1998). "How to attract business via a website." Training & Development 52 (5): 97.
Ellis, R. K. (2001). "E-Learning a la Carte." Training & Development 55 (3): 79.
Ellsworth, J. H. (1994). Education on the Internet . Indianapolis, Indiana, Sams Publishing.
Evans, R. (1999). "Serving Modern Students in a Modern Society at the Community College: Incorporating Basic Technological Literacy." T H E Journal (Technological Horizons In Education) 27 (3): 102.
Faden, M. (2000). IBM Takes E-Business Learning Into Its Own Hands. InformationWeek : 316.
Feldman, S. (1999). "The Key to Online Catalogs That work? Testing One, Two, Three." Computers in Libraries 19 (5): 16.
Fisher, D. (2000). E-learning alliances gain momentum. eWeek : 18.
Galagan, P. A. (2001). "Top 14 things CEOs should know about E-learning. (Executive management)." T&D 55 (11): 69.
George, T. (2002). All Aboard E-Learning At Air, Rail Companies; American Airlines and Union Pacific see cost efficiencies in online training. InformationWeek : NA.
Gilhooly, K. (2001). "Making E-Learning Effective." Computerworld July 16, 2001 : 52.
Guernsey, L. (2000). Education: Web's new come-on; e-commerce sites see online courses as a novel way to lure shoppers. The New York Times : D1.
Hipwell, W. (2000). "Promoting Your E-Learning Investment." Training & Development 54 (9): 18.
Kantner, L. (1994). "Techniques for managing a usability test." IEEE Transactions on Professional Communication 37 (3): 143.
Khirallah, D. R. (2000). A New Way To Learn? -- As E-commerce accelerates business, companies turn to online learning to cut teaching costs. InformationWeek : 22.
Kirkwood, A. (2000). "Learning at home with information and communication technologies." Distance Education 21 (2): 248-259.
Kurzel, F. & Slay, J. (2003). Towards a Personalised Learning Environment . AusWeb03, Sanctuary Cove, Gold Coast, Southern Cross University Press.
Kyer, M. (2002). "Taking insurance training online." Canadian Underwriter 69 (9): 55-58.
McCann, D. & N. Mead (2002). "A case study in the usability and design challenges of e-learning: this article presents our thoughts on the key design principles of e-learning, aimed at giving the user a self-paced learning experience with a high level of support on both training and technical issues. (Training Supplement)." Journal of Banking and Financial Services 116 (1): 44.
Mills, M. (2003). Mainlining Online Learning in NSW Health . AusWeb03, Sanctuary Cove, Gold Coast, Southern Cross University Press.
Nielsen, J. (1996). "Putting the user in user-interface testing." IEEE Software 13 (3): 89.
Nielsen, J. (1997). "Something is better than nothing." IEEE Software 14 (4): 27.
Nielson, J. (2000). Designing Web Usability: The Practice of Simplicity . New York, New Riders Publishing.
Patterson, K. & Ellis, A. (2003). The World of the University Web site Manager - Issues and Environment . Proceedings of AusWeb03, Hyatt Regency Sanctuary Cove, Southern Cross University. ISBN 1 863844 93 7
Patterson, K., Ellis, A. & Brice, D. (2000). Client Versus Browser: A Case Study from Southern Cross University . Proceedings of the Moving Online Conference, Gold Coast, Southern Cross University Press.
Patterson, K. & Ellis, A. (2002) Design and Navigation: A Survey of Australian University Homepages , Proceedings of AusWeb02, Southern Cross University Press, Lismore, ISBN 1 863844 53 8
Spool, J., Scanlon, T., Schroeder, W., Snyder, C., & DeAngelo, T. (1999). Web Site Usability: A Desigers Guide . San Francisco, Morgan Kauffman Publishers.
Tselios, N. K. Avouris, N. M. Dimitracopoulou, A. & Daskalaki, S. (2001). "Evaluation of distance-learning environments: Impact of usability on student performance." International Journal of Educational Telecommunications 7 (4): 355.
Yates, R. (2003). Heterogenous University Web Collectives . Proceedings of AusWeb03 Changing the Way We Work, Hyatt Regency Sanctuary Cove, Southern Cross University. ISBN 1 863844 93 7