ViSION: Visualizing Student Interactions Online

Judy Sheard [HREF1], School of Computer Science and Software Engineering [HREF2] , Monash University [HREF3], Victoria, 3145. judy.sheard@csse.monash.edu.au

David Albrecht [HREF4], School of Computer Science and Software Engineering [HREF2] , Monash University [HREF3], Victoria, 3800. david.albrect@csse.monash.edu.au

Eli Butbul, School of Computer Science and Software Engineering [HREF2] , Monash University [HREF3], Victoria, 3145.

Abstract

An important consideration in the provision of an online learning environment is ensuring that the environment can be used by students in a way that is effective and efficient for their learning. From the educator point of view this means that the instructional model presented is appropriate for their students. A difficulty is that students mostly use online learning environments away from the classroom and out of sight of their educators. Without the monitoring that occurs in face-to-face teaching it is difficult for educators to know how their students are using and responding to these environments. This paper presents a tool which will allow educators to gain information about students’ usage of an online learning environment through visualization of their navigation. The tool provides options for the selection of paths to be visualized and for customization of the interface. The development of the tool and the pedagogical theory underpinning the design is described. This is the first stage of the development of a tool which will enable visualization of both static and dynamic online behaviour.

Introduction

With the vast amount of resources and effort that are put into the provision of Web-based learning environments within the tertiary sector it is important that they are used effectively. An advantage of the Web medium is that it can be used to develop learning environments that are accessible, flexible and adaptable to the learning needs of a diverse group of students. Various studies have shown that the design of an online learning environment and the way it is used within an educational program can encourage or impede learning processes (MacGregor 1999). It is therefore imperative that consideration is given to pedagogical issues when using Web technology. However, studies have also shown that Web technology is not always used with a pedagogical foundation in an educational setting (Greening 1998). This suggests that there is a need for systematic and appropriate research and evaluation into the educational use of this technology (Bates 1997).

An important consideration for educators is that the design and organization of teaching programs which incorporate Web resources are very different to those which are based solely on a traditional paper-based, face-to-face situation. Carr-Chellman and Duchasel (2000) argue that effective instruction online has specific and different design requirements and that simply transposing traditional course material onto the Web does not use the medium to its best advantage. However, Mitchell, Dipetta, and Kerr (2001) claim that many of the models for developing, integrating, and using online course delivery systems tend to be based on past experiences with older technologies or without technology.

Designing an effective Web-based learning environment requires an understanding of students’ learning behaviour in the dynamic Web medium. Determining student use of a Web-based learning environment can inform decisions about the appropriate pedagogical models to use when incorporating these environments into teaching programs (Peled and Rashty 1999). However, determining learner behaviour in electronic media is a complex problem. A difficulty is that students mostly use these environments away from the classroom and out of sight of their educators. Without the informal monitoring that occurs in face-to-face teaching it is difficult for educators to know how their students are using and responding to these environments. Educators have had to seek new ways of obtaining information about the learning patterns of their students. This requires the development of effective methods of determining and evaluating learner behaviour in electronic environments.

These issues have provided the impetus for the development of a visualization tool, ViSION (Visualization of Student Interactions Online), that can be used to explore learner behaviour in educational Web-based environments. ViSION will enable educators and instructional designers to gain insights into students’ learning strategies and resource usage by graphical representations of their interactions within the environment. This tool can also be used by researchers to inform the modelling of learner behaviour in electronic environments. This paper describes the design and development of the ViSION tool.

Investigating Website Usage

Many studies have investigated students using Web-based educational resources and reported findings of aspects of learner behaviour. A search of the literature showed that most studies have used data collected via surveys, interviews or from observing students as they work. Observations may be audio or video recorded. A concern with using these methods to collect data on website usage is that they may be too insensitive or imprecise for the evaluation task (Lu, Zhu et al. 2000). Furthermore, survey and interview methods rely on self-reporting and may be biased due to selective recall or inaccurate estimation (Yi and Hwang 2003). This problem was highlighted in a study of student use of a Web tutorial in biology by Cann (1999). Correlations of responses from an on-line questionnaire incorporated into the tutorial showed that only 17% of students were able to report the length of their sessions accurately. Another issue is that most studies found had been conducted in experimental situations and do not necessarily give a realistic view of actual learner behaviour. As Laurillard (2002) maintains, “The only real test of any learning material is its use under normal course conditions” (p.205).

Electronic environments provide new opportunities for obtaining information about students in a learning situation, allowing the collection and storing of data of interactions in log files. Data recorded on log files has the advantage of being an objective measure of actual website usage and does not suffer from biases due to self-report methods (Nachmias and Segev 2003). Furthermore, the information is accumulated automatically in a way that does not interfere with normal learning activities. Interactions recorded on a log file enables frequent or continuous monitoring of an on-line learning environment. Log file data provides a means of evaluating a system that is in constant change and is therefore suited to the dynamic Web environment. From a pedagogical perspective, continuous collection of learner interactions provides an opportunity to adjust a teaching program or modify the learning environment during the delivery of a course. This is in line with a culture of reflective practice and blurs the view of evaluation as discrete formative and summative practices.

Log files typically contain a huge volume of data formatted in a way that is difficult to interpret. The interactions stored on log files form a complex set of data on learning strategies and website usage. The challenge of using log file data to investigate learner behaviour becomes one of using an analysis which provides information at an appropriate level of detail. In order to be useful to educators and website designers the analysis needs to produce results that are comprehensible, but not lacking in detail through oversimplification. Results should also be presented in a way that provides useful and meaningful information. To enable this, educators and site designers need facilities which will enable them to undertake their own explorations and be able to easily interpret the results of these explorations.

Understanding Learner Behaviour

Records of learner interactions stored on log files provide a rich source of data about learning behaviour in Web-based learning environments. They can give insights into the strategies that the learners employ and difficulties they may encounter. Furthermore, information on accesses to different parts of a website can indicate the usefulness of various resources provided or highlight problems with site design. Some examples of questions that may be answered by analysis of log file data are: These questions illustrate the range of information that may be gained from analysis of website data. This information can be used in a variety of ways by educators, website designers and educational researchers to gain insights into learner behaviour. Educators often provide resources with expectations that students use them in particular ways and at particular times. Knowing which parts of a website that students access can give indications of what resources they find valuable. For example, an analysis of student use of a courseware website by Peled and Rashty (1999) found that the most popular online activities were passive and involved getting information rather than contributing. They concluded that students were very goal oriented in their use of the website. Further information can be gained from knowing when students access resources. This can help educators understand students’ preferred learning patterns. A study by McIsaac, Blolcher, Mahes, and Vrasidas (1999) explored interactions of doctoral students with an online environment. Using log file data and qualitative data from interviews, message archives and chat sessions they found that interactions took place when students were attempting a task. They concluded that student interactions were goal focussed.

Information about website design can be gained from exploring students’ paths through a website. This can provide information to both educators and website developers. For example, frequencies of particular paths can show common sequences of activities which can elucidate upon strategies which learners adopt. This was demonstrated in a study of student use of a first year geology website by Hellwege, Gleadow, and McNaught (1996), log file analysis showed that students accessed the most recent lecture notes first, picking out a couple of key slides, before returning to a previous lecture. Over the semester a gradual increase in use of the material was found. This suggested that students were accessing resources according to immediate need. Particular navigation patterns may also indicate the lack of, or prominence of, navigational mechanisms available. They may provide warning of unsuitable links or unnecessary but obligatory navigation steps, indicating a problem with site design (Chavero, Carrasco et al. 1998). An analysis may look at the last pages visited to show where the users leave the website. This may indicate the focus of a website visit or may indicate a point of frustration where the users gave up (Buttenfield and Reitsma 2002).

Analysis of learner interactions may also be used to compare learning behaviours of different groups of students. Some studies have found a relationship between learning outcomes and website usage. Comunale, Sexton, and Voss (2001-2002) found evidence to suggest that higher course grades are related to more frequent website use. A study by Lu, Zhu, and Stokes (2000) which analysed log file interactions with different resources on a courseware website found a relationship between frequency of assess to learning resources and final exam scores. They contend that this provides evidence that the use of relevant Web content improves learning. A more recent study by Gao and Lehman (2003) investigated learning outcomes of students using Web-based learning environments providing different levels of interactivity. Log file analysis showed that the students in the proactive and reactive interactions groups spent more time on task. Interview data revealed that the students in the interactive groups spent more time reviewing and reflecting and learning content and this resulted in greater learning outcomes.

Although there have been various gender based studies of the use of the Web, there are a scarcity of studies which report on differences in courseware usage based on gender. A study by Peled and Rashty (1999) found differences in the type of resources accessed by males and female students. The males used interactive resources significantly more than the females; whereas, females used passive resources more than males.

These various studies illustrate the range of information to be gained from analysis of website usage, and how this information can inform educations about their students. The challenge is finding a means to present this information in interpretable ways.

Representing Learner Behaviour

The different ways that log files may be analysed may be broadly seen as presenting two different views of website interactions. These may be classified as follows:

Benefits of Visualization

Extraction of meaningful and useful information about learner behaviour from log file data requires the use of statistical or data mining techniques. However, without appropriate tools this is difficult and beyond the expertise of most educators to do this (Zaïane and Luo 2001). A variety of tools are available to analyse log files. Many tools produce reports of statistical analysis which provide little insight into user behaviour and traffic patterns. Hochheiser and Shneiderman (2001) claim that output reports are of limited value in that they typically provide summaries with restricted or no facilities to examine individual page requests. Added to this, reports are often presented in terms of page visits without relating these to the site layout. Another consideration is that site usage, content and structure can change greatly over time. Facilities are needed that will enable the data to be interactively interrogated, thus allowing for the dynamic nature of the Web environment.

Also important to educators and instructional website designers are the navigation patterns through online resources, as these give insights into the learning behaviours of students. However, even small websites can have a large variety of paths users may take. When we consider the volume of data and number of possible paths, what is needed is a way of managing and representing this data so that it may be interpreted in a meaningful way. Visualization techniques provide a means of doing this.

Visualization techniques enable a complex set of data to be visualised in a way that may be easily interpreted. Hochheiser and Shneiderman propose that “interactive visualizations of log data can provide a richer and more informative means of understanding site usage” (p.331). They claim that visualization can reveal patterns that are not obvious from statistical analysis and can be used to complement text-based reports. Berendt and Brenstein (2001) state that visualization allows the representation of large and high dimensional data sets in a compact space, allowing comparisons to be made and the discovery of trends that often remain unnoticed in a statistical analysis.

Visualization can provide a way of effectively communicating key information about website usage. It can be used in exploratory data analysis or data mining (HREF5). Visualization tools can incorporate facilities which will provide ways to obscure or view detail. This allows the user to gain a high level view of website usage or focus on specific elements of interest. Various levels of abstraction can allow an overview of navigation through an entire website, a particular area of a site, or activity centered around a single page. It can show summaries of website usage for a group or for a particular user. Visual elements of colour, shading, patterns, shapes, symbols, and text, combined with size and arrangement can be used to provide meaningful views of different aspects of website usage.

Visualization of Website Interactions

Various tools such as DIAGEN (Chavero, Carrasco et al. 1998), Stratdyn (Berendt and Brenstein 2001), and WebViz (Pitkow and Bharat) have been built to visualize navigation paths from log file data. However, these tools generally only use information available in the log file. An example of this is WebViz which uses the log file data to construct a representation of the access between pages in the form of a directed graph, and uses colour and thickness to represent the frequency of access. This is similar to what we do in the ViSION tool.

Of further interest in an educational domain is the incorporation of other details about the students, for example, gender, grade, and their project group. A search of the literature found that very few visualization tools had been designed for educational purposes. Chavero (1998) used DIAGEN to analysis the students’ navigation paths, comparing their paths to the answers they gave on a test. However, the test information is not explicitly incorporated into the visualization. While Berendt and Brenstein (2001) used colour to represent the different tasks that students were set, and perform various qualitative analysis on the navigation paths.

ViSION

In the ViSION tool we combine the log file with the student information into a single database, and incorporate all the information into the visualization. This enables comprehensive views of the navigation behaviour of students using an educational website. ViSION uses graphical representations to provide an effective means to show navigation patterns. This facilitates the identification of access problems and enables the study of learning behaviours of students. This tool has been designed to visualize and compare: A fundamental principle used in designing this tool has been to display the same information in various ways and allow the user the option to customize how the information is visualized. This is a general design principle which will cater for user preferences but will also allow us more readily to investigate and evaluate the effectiveness of different methods of visualization.

Navigation Behaviour

The navigation behaviour is displayed in the form of a graph. The nodes of the graph represent either specific topics, or the pages associated with a specific topic, and the arrows between the nodes represent the traffic between the topics, or pages, corresponding to the nodes. The layout of the graph is not fixed and the user may move nodes to wherever they like. To decrease the amount of congestion in the display, the heads of the two arrows between nodes meet at a point and the position of the point depends upon the relative proportion of traffic between these nodes. This means if the traffic from node A to node B is twice as much as the traffic from B to A, the arrow from A to B will be twice as long as the arrow from B to A. An example of the screen display produced by ViSION is shown in Figure 1.

The tool allows more detailed information of traffic at a particular node. When the user clicks on a node, a window appears and a graph is displayed in the window. This graph shows the nodes that a user visits before and after the node clicked, and the amount of traffic between these nodes is indicated by the thickness and color of the arrows. An example of the screen display is shown in Figure 2.

Usage of resources

By moving a mouse over the node or arrow respectively, the user can display the number of transactions performed through a node or the amount of traffic between two nodes. The relative amount of traffic between two nodes compared to the total traffic is indicated by the width of the arrow and the color of the arrow. By customizing the features of the arrows the user can display specific levels of traffic. An examples of these are shown in Figures 1.

Comparison of Behaviour

To compare various behaviours, a user can simultaneously display multiple navigation behaviours, where each display appears in its own window, and is based on a selection of: Comparisons made between different time periods can show changing patterns of behaviour over time. Displaying navigation information in this way enables a rich picture of learning behaviour.


Figure 1: Screen Display of Interactions with a Selected Group of Pages
 
 
 
 
 

Figure 2  Display of Traffic through a Page on the Website

An Illustrative Example of ViSION

To illustrate the use of ViSION, the tool was used to display student interactions with a courseware website designed to assist students with their capstone project work. The interactions were collected on a log file over the duration of the project. In this course students worked in groups of five on a project over two semesters. During this time the students were required to use the website to record their project times using a time tracking facility. They also needed to access the site to download some essential documents from a repository. In addition, there were other resources provided on the site that students could choose to use. These included a file repository, risk management facility, group and class discussion forums, news, event scheduler and static information pages.

Figures 3 and 4 show summaries of website interactions for two project groups. These high level views show traffic between resources, with each resource representing a group of pages. Thicker lines denote higher traffic volume. The difference in behaviour between these two project groups can be seen by visual inspection of the graphs. The traffic representations in Figure 1 show that the students in Group 1 used the time tracker facility very heavily and sometimes accessed the information pages (in Miscellaneous). They showed a very functional use of the site only accessing the essential resources.  Group 2 also used the time tracker heavily; however, they also showed moderate use of many of the non-essential resources such as the file repository, event scheduler, and communication facilities. The behaviour of this group, demonstrated by their website usage, shows more engagement with project management practices.

The visualization of the interactions with this website highlighted a problem with the navigation structure. The website was designed with a hierarchical structure. Each resource was accessed from the home page and there were very few links between resources. This was a deliberate decision by the designers as it was felt it would be a simple structure for the students to learn and use.  However, the visualisation showed that usually students accessed several resources at each website visit and the hierarchical structure created unnecessary navigation steps. This also created unnecessary hits on the website which had an affect on the response time during peak access times. As a consequence of this discovery the website design was changed to a network structure.

Figure 3: Summary of Website Interactions for Group 1

Figure 4: Summary of Website Interactions for Group 2

Conclusions and Future Work

Understanding how learners use Web-based learning environments is of critical importance for the development of a sound pedagogical basis for the design and use of a Web-based learning environment. This paper describes the design and development of a visualization tool which will enable educators and instructional designers to gain a view of learner behaviour in online environments by graphical representations of their interactions within the environment

A critical aspect of designing technology is establishing its suitability for its intended audience. We also plan to conduct demonstrations and interviews with educators to refine various features of this tool and establish the most effective methods of visualization of learner website interactions.

The current version of the tool has concentrated on static view of the website interactions. In the future we plan to extend the tool to also represent students’ navigation paths, and clusters of students based upon their navigational behaviour. This will provide a valuable tool for educators and educational website developers.

References

Bates, A. W. (1997). Restructuring the university for technological change. 2000.

Berendt, B. and E. Brenstein (2001). "Visualizing individual differences in Web navigation: STRATDYN, a tool for analysing navigation patterns" Behavior Research Methods, Instruments & Computers v.33 n.2 p.243-257.

Borges, J. and M. Levene (2000). "A fine grained heuristic to capture web navigation patterns" ACM SIGKDD Explorations v.2 n.1 p.40-50.

Buttenfield, B. P. and R. F. Reitsma (2002). "Loglinear and multidimensional scaling models of digital library navigation" International Journal of Human-Computer Studies v.57 p.101-119.

Cann, A. J. (1999). "Approaches to the evaluation of online learning materials" Innovations in Education and Training International v.36 n1 p.44-52.

Carr-Chellman, A. and P. Duchasel (2000). "The ideal online course." British Journal of Educational Technology v.31 n.3 p.229-241.

Chavero, J., J. Carrasco, et al. (1998). "A graphical tool for analyzing navigation through educational hypermedia." Journal of Educational Multimedia and Hypermedia v.7 n.1 p.33-49.

Comunale, C. L., T. R. Sexton, et al. (2001-2002). "The effectiveness of course Web sites in higher education: An exploratory study" Journal of Educational Technology Systems v.30 n.2 p.171-190.

Gao, T. and J. D. Lehman (2003). "The effects of different levels of interaction on the achievement and motivational perceptions of college students in a web-based learning environment" Journal of Interactive Learning Research v.14 n.4 p.367-386.

Greening, T. (1998). "WWW support of student learning: A case study." Australian Journal of Educational Technology v.14 n.1 p.49-59.

Hellwege, J., A. Gleadow, et al. (1996). "Paperless lectures on the Web: An evaluation of the educational outcomes of teaching Geology using the Web" in Proceedings of 13th Annual Conference of the Australian Society for Computers in Learning in Tertiary education (ASCILITE '96), Adelaide, Australia, University of South Australia.

Hochheiser, H. and B. Shneiderman (2001). "Using interactive visualisations of WWW log data to charaterize access patterns and inform site design." Journal of the American Society for Information Science v.54 n.4 p.331-343.

Laurillard, D. (2002). Rethinking University Teaching. London, UK, RoutledgeFalmer.

Lu, A. X. Y., J. Zhu, J. H., et al. (2000). "The use and effects of Web-based instruction: Evidence from a single-source study" Journal of Interactive Learning Research v.11 n.2 p.197-218.

MacGregor, S. K. (1999). "Hypermedia navigation profiles: Cognitive characteristics and information processing strategies" Journal of Educational Computing Research v.20 n.2 p.189-206.

McIsaac, M. S., J. M. Blolcher, et al. (1999). "Student and teacher perceptions of interaction in online computer-mediated communication" Educational Media International v.36 n.2 p.121-131.

Mitchell, C., T. Dipetta, et al. (2001). "The frontier of Web-based instruction" Education and Information Technologies v.6 n.2 p.105-121.

Nachmias, R. and L. Segev (2003). "Students' use of content in Web-supported academic courses" The Internet and Higher Education v.6  p.145-157.

Peled, A. and D. Rashty (1999). "Logging for success: Advancing the use of WWW logs to improve computer mediated distance learning" Journal of Educational Computing Research v.21 n.4 p.413-431.

Pitkow, J. E. and K. A. Bharat (1994). WebViz: A tool for World-Wide Web access log analysis in Proceedings of 1st International WWW Conference, Geneva, Switzerland.

Yi, M. Y. and Y. Hwang (2003). "Predicting the use of web-based information systems: efficacy, enjoyment, learning goal orientation, and the technology acceptance model" International Journal of Human-Computer Studies v.59 p.431-449.

Zaïane, O. R. and J. Luo (2001). "Towards evaluation learners' behaviour in a web-based distance learning environment" in Proceedings of the IEEE International Conference on Advanced Learning Technologies (ICALT 2001), Madison, WI, USA, 6-8 August 2001

Hypertext References

HREF1
          http://www.csse.monash.edu.au/~jsheard
HREF2
          http://www.csse.monash.edu.au/
HREF3
          http://www.monash.edu.au/
HREF4
          http://www.csse.monash.edu.au/~dwa/
HREF5
          http://www.math.grinnell.edu/~lindseyd/ResearchState.html

Copyright

Judy Sheard, © 2005. The authors assign to Southern Cross University and other educational and non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The authors also grant a non-exclusive licence to Southern Cross University to publish this document in full on the World Wide Web and on CD-ROM and in printed form with the conference papers and for the document to be published on mirrors on the World Wide Web.