Steve Goschnick and Sandrine Balbo, Dept. of Information Systems, University of Melbourne, Vic, 3010, Australia
Derek Tong, Cathy Glen, Tony Chiang, Wendy So and Chi-May Ooi, Department of Computer Science, University of Melbourne, Vic, 3010, Australia
Web site instrumentation. Log file analysis. Usability evaluation.
This poster presents an innovative process used for supporting usability evaluations of web sites. We propose a process that combines the automatic capture of user’s actions and the linking of this capture to a representation of the User Interface, or, following Ivory and Hearst’s taxonomy: a method for which the automation supports capture and analyzing [1]. To our knowledge, no other method nor tool provides this combination with this degree of automation.
Using this process involves several chronological steps: the automatic instrumentation of the web site to be tested so that client-side events generated by a test subject are accurately recorded to a sophisticated log file; the capture of one to four digital videos (DV format) of the usability test in progress; the capture of the ‘points of interest’ of the test facilitator who is monitoring the test as it happens; analyzing the usability test after the event by combining all log file information and video footage within the purpose- built Web Evaluation Path Navigator (WEPN) program. In short, there are three phases of activity – logging, monitoring and analyzing. The time sequence covering the whole usability test can be navigated with several ‘views’ upon the data. There are essentially three different views of the usability test, interconnected and viewable in WEPN at the one time: the video window; the network-graph representing the web pages as nodes and hyperlink paths as lines or arcs joining the nodes; and the time-line with tags along it which represent the ‘points of interest’ added by the test facilitator which are then annotated by the analyst. The video view consists of one to four video images of the test-in-progress from different angles, including a screen-shot of the web site being navigated by the test subject.
The four videos are independently recorded in separate mini DV camcorders, and transferred to hard disk on the analysis machine after the test. WEPN includes a synchronization process in which the analyst synchronizes the four videos with the log file. Once all the files are on hard disk and within the WEPN system, the analyst can randomly access any part of the test instantly, or move through the test much faster than real time, as needed. This takes a lot of tedium out of the analysis of large amounts of video footage, commonly associated with u sability tests.
At the base of the WEPN is a sophisticated log file representing client-side events – keystrokes, mouse clicks, mouse movements in and out of web page components, etc. The log file we use is an extension of an Open Standard log file called a FLUD file, developed by the National Institute of Standards and Technology (NIST) [2].
The web site being tested is copied to a local web-server and ‘instrumented’ to capture the sophisticated client-side event log. NIST provide a program called WebVIP [2] for instrumenting a web site, which we also use. It inserts appropriate JavaScript code into the .html pages of the copied site, to capture the client-side events needed by the WEPN analysis system.
The monitoring system is a simple program which we also developed, which the test facilitator uses both before and during the actual usability test. The monitoring program allows the use of the ten keyboard function keys to be loaded with test-specific meaning, eg. ‘F1 – A terminology or labeling problem’. The test facilitator can then use the F1 to F10 functions keys to quickly add ten specific types of ‘points of interest’ during the tests progress. This is done on the test facilitator’s computer, and happens usually behind the one-way mirror of the usability laboratory.
The strength of this process is its ability to link the automatically generated site map with the four videos, the logging of the end-user’s actions, and the recorded ‘ points of interest’.
At present, WebVIP limits us to web sites done in standard html. To be able to instrument sites generated by content management systems, we are currently looking into developing our own program to capture client-side events.
[ Proceedings ] [ AusWeb Home Page ]