What Learners Want from a Learning Management System

Mr Alan Holzl, Production Manager, Teaching and Educational Development Institute [HREF1] , The University of Queensland [HREF2], Ipswich Campus, Ipswich, QLD, 4305. A.Holzl@uq.edu.au

Abstract

The University of Queensland recently conducted a survey of users of its learning management system (LMS), WebCT, to collect both positive and negative feedback from students and staff.  The majority of responses were from students who provided a wealth of useful comments which were more relevant to the way a LMS is used within the university rather than focusing on WebCT itself.  This paper describes the key issues and themes identified by the students which address questions such as:  Which features of a LMS do learners find most and least useful? What additional features would they like to see? and what aspects of policy and procedures need to be addressed to enhance the implementation of a LMS within an educational institution such as the University of Queensland.

Introduction

The University of Queensland (UQ) adopted WebCT as its learning management system (LMS) of choice back in late 1998 as a prelude to the opening of its new Ipswich campus which was designed to be a test bed for combining flexible learning methodologies with new information and communications technologies (ICT). At the time the original decision to adopt WebCT was taken, the development and use of a LMS within higher education institutions was an emerging field in which WebCT was seen to be the most suitable product for UQ. Since that time there has been an explosion of similar products with WebCT being able to maintain its lead despite major improvements in its main competitor, Blackboard (American Federation of Teachers, 2001. p.10). [HREF3]
During this period of significant changes in the LMS market, there have been even more changes in the higher education sector throughout the world, with the USA and the UK providing the most likely scenarios for the future of Australian universities. This future appears to be firmly rooted in the view that the solutions to all of our problems lie in the increasing use of ICT in the form of online learning, also referred to as eLearning, or, in the case of the USA, as distance learning. Hand in hand with the increasing emphasis on online learning is a surge of interest in the role of a LMS in facilitating online learning within educational institutions such as UQ.
For these reasons, UQ set up a working party to revisit its original choice of a LMS in conjunction with a review of its current policies and procedures for implementing eLearning into the future. As part of this review, the working party commissioned a survey of current staff and student users of WebCT. The survey was conducted via a web page which included the following questions:

• What do you like about WebCT?
• What do you find most useful about WebCT?
• What don’t you like about WebCT?
• What else would you like to see in an eLearn package?
• Would you be interested in discussing this further in a Focus Group?

The web page was made available during the period August to November 2002 with the results of the evaluation to be brought to the working party by the end of 2002. Six staff and 171 students responded giving a total of 177. Most of the students provided their student numbers to allow for any further contact if they answered “yes” to the question on focus groups. At the time of writing this paper the focus groups had not been conducted. For the purpose of this paper, the small numbers of responses from staff have been isolated to ensure that the learners’ perspectives are represented.

Methodology

The answers to all of the questions, linked to the respondent’s student number (where provided), were downloaded from the website into an Excel spread sheet which provided the source materials for further analysis and interpretation. All of the answers for each of the first four questions were then consolidated into separate files and subjected to a manual content analysis to group similar responses into various clusters or “themes”. The number of responses within each theme was recorded at the bottom and each theme was arranged in reducing order from most number of responses to the least with the “general” theme shown at the bottom.

As often happens with this type of survey, the sum of the numbers of responses for each question did not necessarily add up to 177 due the fact that not every respondent answered every question and not all of the responses were usable e.g. responses like, “nothing” and “everything”. In addition some answers addressed multiple points and were split across a range of different themes. There were also some answers that were placed into a “general” theme, as they could not be linked to any specific aspect of WebCT nor an eLearning system in general. No attempt was made to edit or correct the spelling or grammar of the answers from what was originally entered by the respondents

Outcomes of the Survey

The most interesting outcome of the survey, which prompted the decision to write this paper, was that many responses went beyond the original expectations of the survey. Although the questions were addressed specifically at WebCT and desirable features of a future LMS, many of the answers were directed at policies and procedures (or lack of) which are perceived to exist within UQ concerning the way WebCT is used (or not used) at all levels. These responses were seen to be outside the scope and intention of the survey, however, they showed great potential as a source of information for reviewing current policies on e Learning within UQ and similar organizations within the higher education sector. A summary of the responses, divided into various themes, for each of the questions is shown below.

Question 1: What do you like about WebCT?

An analysis of the themes along with their implications for the design and use of a LMS are as follows:

• Computer Mediated Communications (CMC) Tools (email, bulleting boards, chat rooms) - 65 responses. This was by far the most popular theme. It is difficult to separate out the responses in support of these tools within WebCT from support for the concept of two-way communication between lecturers and students and student-to-student. The implications for an LMS is that it should include a capability for two way communication as is found in email, chat rooms and bulletin boards. The most popular tool within this theme appeared to be “discussion groups and bulletin boards”.

• Access to course information and content – 38 responses. The second most popular theme covers the capability to provide access to course information and course content. This can be achieved without WebCT or any other LMS, however, in order to display course content, it needs to be developed easily and quickly by someone so the LMS should include a capability for course developers to place course information and content onto the web and for students to access these materials. This theme links to other issues of access and a central location which were addressed in other themes.

• Flexibility of access (own time, own place. own pace) – 34 responses. This theme is a general statement in support of the flexibility of access provided by online learning and is not specific to any particular capabilities for a LMS.

• Easy to use – 33 responses. The criteria identified here is for a LMS to be easy to use for students.

• Quizzes – 18 responses. Once again this theme shows some confusion over the ability to provide progressive, formative assessment with timely, meaningful feedback with the quiz tools within WebCT. The relevant LMS capability would include a range of online assessment tools which make it easier for students to undertake quizzes and receive timely and useful feedback.

• Means of providing timely information to students – 15 responses. There is no doubt that this theme overlaps with the second group discussed above. The reason it was separated out is that it appears to refer more to the ability to notify of changes to administrative information rather than course related information e.g. changes to dates for submitting assignments or changes in venue etc. This capability can be achieved online without a specific LMS, however, it does link to the need for a feature which alerts students to changes in critical information without them having to specifically log into a central site.
• All information at the one site – 14 responses. This theme supports the concept of having all relevant information at the one site whether it be a general web site or within a LMS. This theme recurs later in a criticism of the existing system of multiple portals and sites within UQ such as WebCT, my.UQ, SiNet, faculty/school sites and the library all of which can contain relevant course information and content.


Question 2: What do you find most useful about WebCT?

As expected, the responses to this question followed similar themes to the first question. They were as follows:
• CMC Tools (email, bulleting boards, chat rooms)- 59 responses.

• Access to course information and content – 38 responses.

• Quizzes – 18 responses.

• Easy to use – 16 responses.



• Flexibility of access (own time, own place. own pace) – 12 responses.

• Improves learning – 6 responses

• Not compulsory – 3 responses

Question 3: What don’t you like about WebCT?

An analysis of the themes along with their implications for the design of a LMS are as follows:

• Listing of old courses - 27 responses. These responses refer to the practice of displaying lists of old courses alongside current courses after the student has completed that course. This is a criticism of policies and procedures for using WebCT rather than WebCT itself. Within UQ, students are given access to completed WebCT courses for the following year in case they want to revise any material or look back on their work. This access is provided in response to requests from students during the early years of use. It appears, however, that the listing of completed courses needs to be separated from those of current courses.

• Discussion Lists/Bulletin Board – Technical aspects – 23 responses. This is one of three categories that refer to the use of CMC tools. The other two relate to perceived student misuse of bulletin boards in terms of “netiquette” and quality of discussion and the use of email. Examples of “technical aspects” include difficulties with changes to the layout of the bulletin board due to an upgrade to WebCT in the latter half of 2002 (5 responses). Other criticisms are directed at a lack of training and/or experience in the use of WebCT rather than WebCT features e.g. the statement that they (the designer) cannot create separate discussion groups is incorrect. Another comment about lack of access to other discussion groups is due to a decision by the designer to restrict access rather than a limitation within WebCT. The need to continually log into WebCT to check whether there is any new material (3 responses) is only partly true as the My WebCT page does indicate whether there are new messages provided the designer does not change the default settings.

• Useability problems – 20 responses. This category could be viewed as the negative of the “Easy to Use” grouping identified in the first two questions. The responses are a combination of general and specific comments linked to certain idiosyncrasies of WebCT such as it is “clunky” and “not intuitive” to use.

• Availability of subjects – 17 responses. In this category, the respondents are critical of the fact that not all courses are available in WebCT. It is a case where two negatives equal a positive although it is a criticism of current UQ policy regarding the use of WebCT. These comments seem to be inconsistent with those that value the fact that WebCT courses are not compulsory.

• Use of WebCT by lecturers – 16 responses. This is another criticism of the way in which WebCT is used, or misused, by lecturers/website designers within UQ rather than a criticism of WebCT itself. One implication of this criticism which may have a bearing on desirable features for a LMS is that it should be simple and easy to use “intuitively” without the need for a lot of additional training and/or experience. This does not address the issue of poor instructional and graphic design skills as opposed to a lack of skill in using the tools. These responses support the need for a comprehensive training program for users and/or a review of the policy which supports a “Lone Ranger” or a DIY approach as opposed to a team approach in which the lecturer works within a team comprising instructional and graphic designers, programmers and WebCT experts.

• Student misuse of Bulletin Boards/Netiquette – 14 responses. These responses are related to the way in which staff and students use bulletin boards and have limited implications for the design of a LMS except maybe for the option that allows a moderator to filter discussions before they are posted. It also supports the need for student users to be educated about issues of “netiquette” and the UQ Internet Code of Practice.

• Technical difficulties – 10 responses. This theme differs from the “Useability Problems” described earlier in that it refers to examples of WebCT failing to function as opposed to functioning in a manner which is unduly complicated or difficult to use. Some examples include the failure to work within some browsers and even some computers as well as slow load speeds and “crashing”. These issues can be addressed by developing a set of technical specifications within the design criteria that include the full range of browsers, computers and other software which must be compatible with the LMS as well as performance benchmarks for speed and resistance to “crashing”. These latter problems are difficult to quantify due to the many variables outside a LMS which can be responsible for performance problems.




• Quizzes – 9 responses. This theme reflects some confusion between technical problems with the quiz tool in WebCT and the ability of lecturers/designers to make best use of that tool or even how to frame good questions that match the content. Technical problems seem to result from confusion about when answers to quizzes are actually saved especially if the answers are changed and then resubmitted. There are also criticisms of the inflexibility of the means of grading answers that does not cope well with decimal places and rounding. It also does not allow for part marks for having correct working but not the exact answer. The implications for evaluation criteria are that the quizzes make it clear when and which answers are saved and submitted when they are changed and the grading of answers to mathematical questions are flexible enough to cope with decimal points and rounding up and down. Some of these issues also relate to poor use of the quiz tool by lecturers and/or designers who may lack formal educational qualifications. This problem further supports the option of a team based approach to web development whether in WebCT or not.

• Downloading and printing documents – 7 responses. These comments are really a subgroup within “Technical Difficulties” but are dealt with separately because they address a specific problem. The implications for the desirable features of a LMS is that they need to be able to download and print out web pages and common files such as Word, RTF, PowerPoint, PDF etc easily, quickly and without corruption. There were some comments relating to students having to meet the extra costs for printing out material which are a matter of policy rather than a technical issue, however, it is an issue which needs to be addressed.

• Late posting of information – 6 responses. This is another issue which relates to the way a LMS is used rather than a criticism of the system itself. These comments are a reflection of the heavy workloads experienced by teaching staff which are not necessarily reduced by the use of a LMS. In fact there is growing evidence that online courses can be, “much more labour intensive for their instructors than comparable face-to-face courses”. (Hara and Kling, 2000, p.575)

• Chat facility - 4 responses. The criticisms related to the Chat facility within WebCT being “primitive”, “difficult to use” and “unstable” while another student preferred face-to- face chatting.

• Difficulty of contacting lecturer – 4 responses. Not related to use of a LMS. This relates to previous comments on the late posting of information. This comment is consistent with research conducted in the US which reported, “The lack of prompt feedback from the instructor was certainly a major source of anxiety and frustration for students”, (Hara and Kling, 2000; Bonk and Cummings, 1998)

• Difficulty with submitting assignments – 3 responses. These comments related to problems with using the WebCT tool for submitting assignments online.

Question 4: What else would you like to see in an eLearn package?

Details of the main themes arising from this question are as follows:

• Integrated with Si Net/MyUQ and Email accounts - 11 responses. These responses were consistent with other themes such as “All information at the one site” in Questions 1&2 and “email” in Question 3 which supported the concept of all relevant material and tools such as calendars, discussion lists and email being located on one site to replace, or be fully integrated with, other portals such as SiNet and MyUQ. These comments also link in with a criticism of having to actually log into WebCT in order to check for new messages. Many students would prefer having only one email account to which all messages were sent or at least having the email within a LMS linked to their general student account. The implications for a LMS is that it provides a single portal which combines all the features currently provided through SiNet, My UQ, WebCT, library, Course/Faculty/School websites, student email account and any other private email account s they may use. In summary, they want an online “one stop shop” which is linked to everything and automatically notifies them via email to their main account when new messages or course updates are posted.

• More courses to be in WebCT – 9 responses. Once again this theme is a repetition of the “Availability of Subjects” theme in earlier questions which supports having all courses online with consistent standards of design, layout and quality. This is a policy issue rather than a feature of a LMS.

• Course content – 9 responses. This theme supports the function of a LMS in providing course materials in a range of formats from lecture notes to full transcripts of lectures and complete learning guides. This is already achievable within WebCT, however, it is also achievable outside of a LMS.

• Guidelines on using eLearning – 8 responses. These comments are directed at issues of policy and procedures for using a LMS rather than matters of design. They include a need to ensure that lecturers and designers adopt consistent approaches to the use of discussion and debate, linking to other sites and reference material, and referencing styles for assignments. They also requested that all course notes be placed online before start of semester and course evaluation surveys be done via WebCT.

• Discussion Board – 8 responses. This theme contains a mix of policy and technical issues which have been raised in earlier questions. They include a request for anonymous postings, a preference for the “old” bulleting board layout before WebCT was upgraded, and a need for “a moderator to remove offensive and stupid discussion board notices”.

• Functionality – 8 responses. These responses are a mixed bag of suggestions for additional functions over those currently found in WebCT. Some have been mentioned before under other themes such as a need for a specific logoff icon and improved ability to download and print without having to use a “compile” feature. Other features include easier access to “bookmarks” and databases and an anonymous feedback page where students can leave comments for course managers.

• Support for audio and video – 7 responses. Includes support for MediaBase (A UQ developed tool for streaming video) for displaying video clips and video recording of lectures. This can be done outside of a LMS.

Discussion

Although this survey was primarily designed to elicit specific information on WebCT and desirable features for a LMS, it also addresses broader issues of policy and procedures for the implementation of eLearning within UQ as well as in other similar types of educational institution. Before addressing the question of, “What do learners want from a LMS?” we need to address the broader question of, “Do learners really want an LMS?” If we examine the responses to these two questions, the majority of students identified generic features of the web rather than those provided specifically by WebCT nor any other LMS for that matter. Features such as, “CMC tools”, “Access to course information” and “Flexibility of access”, could be provided by merely having access to the web and standard internet tools such as email, newsgroups, HTML etc and/or by use of tools within existing portals such as SiNet, MyUQ, and the library.
The theme relating to, “All information on one site” could also be achieved without necessarily resorting to a LMS, however, the use of an LMS would certainly make this easier to achieve, providing the policy issues which result in multiple portals with duplication of functions, in addition to WebCT, are addressed. The two linked themes, “Availability of subjects “ in Question 3 and “More Courses in WebCT” in Question 4 show that a number of current users would like to see all of their courses available online with consistent standards of design, layout and quality. Once again this can be achieved without acquiring a LMS and as demonstrated within UQ, having a LMS does not guarantee that these goals will be achieved.
It is not until we get to features such as interactive quizzes and the ability to post/access student results do we start to identify a need for a LMS which is integrated with student records. It could be argued, however, that the “Easy to use” theme is supportive of a LMS which allows relatively untrained and inexperienced staff and students to design and access online course materials and activities. This need for relatively unskilled staff to be able to develop course materials needs to be consistent with overall policy related to the availability of staff development programs and skilled support staff who can work with lecturers in a team based approach to developing quality materials within a framework of predetermined standards.

Summary and Conclusions

This paper describes the results of a content analysis of the answers to four question on the use of WebCT within UQ, along with a summary of responses and preliminary interpretations. Many of these responses relate more to broader issues of policy and procedures rather than specific comments on WebCT or similar type of LMS. For this reason the results of this survey will be used to inform current and future policies, procedures and guidelines for the implementation of eLearning within UQ in addition to developing criteria for the review of the current and possible selection of a future LMS. As part of this process it has been useful to revisit the question of whether a LMS is required in addition to the current tools and facilities available to UQ staff and students for providing and accessing the major eLearning functions identified in the survey. This needs to be examined in conjunction with the view that UQ already has too many websites, portals and email accounts in use by staff and students, many with overlapping features and functions.
These and other policy issues raised by this survey reinforce the need to provide additional resources and appropriate policies/guidelines to support the effective use of an LMS which is in addition to simply selecting a vendor and paying the licence fees. These resources are required to provide staff development and training for both staff who design materials and students who are the users. Because UQ does provide numerous training opportunities for both staff and students there needs to be further investigation to establish why some users appear to lack the necessary skills to use WebCT effectively. There is also a need to examine whether additional funds should be provided to produce high quality and consistent standards of online course materials through collaboration with professional educational resource developers within UQ rather than depend on a DIY approach with lecturers developing their own materials of varying levels of quality and effectiveness.
Some of the unresolved policy issues arising out of the survey, which need to be addressed within UQ are listed below as they may also be relevant to other organizations that have implemented, or are about to implement, their own LMS. They are:




Acknowledgement

The survey described in this paper was conceived and implemented by the following members of the UQ eLearning Working Party; Elizabeth Coulter, Beth Cavallari, Deb Turnbull and Anthony Marsh. I also wish to thank my colleague Angela Vink of the Teaching and Educational Development Institute for her feedback on early drafts of this paper.

References

American Federation of Teachers, (2001). A Virtual Revolution: Trends in the Expansion of Distance Education. AFT, Washington DC. Available online. [HREF3]

Bonk, C., and Cummings, J. (1998). “A dozen recommendations for placing the student at the center of Web-based learning” in Educational Media International 1998 v. 35 n.2 p. 82-89.

Hara, N. & Kling, R. (2000). “Student Distress in a Web-Based Distance Education Course” in Information, Communication & Society, 2000 v.3 n.4 p. 557-579.

Hypertext References

HREF1
http://www.tedi.uq.edu.au/
HREF2
http://www.uq.edu.au/
HREF3
http://www.aft.org/higher_ed/downloadable/VirtualRevolution.pdf

Copyright

Alan Holzl, © 2003. The authors assign to Southern Cross University and other educational and non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The authors also grant a non-exclusive licence to Southern Cross University to publish this document in full on the World Wide Web and on CD-ROM and in printed form with the conference papers and for the document to be published on mirrors on the World Wide Web.