Figure 1: Primary LMS In Use (n = 40)
Rod Byrnes, School of Social Science, Southern Cross University, Lismore, 2480. rbyrnes@scu.edu.au
Allan Ellis, School of Social Science, Southern Cross University, Lismore, 2480. aellis@scu.edu.au
In less than a decade Australian higher degree granting institutions have embraced Web-based technologies as a mechanism for providing significant components of external and/or internal course delivery. This has resulted in the development of a new software product - the Learning Management System. Currently two commercial products dominate the Australian market - WebCT (50%) and Blackboard (35%) with the third largest user group comprising in-house developed systems (13%).
An evaluation of these two commercial systems in terms of student assessment functionality revealed that neither product performed particularly well. While activities such as supporting the provision of feedback are adequately catered for, there is scope for development in activities such as objective testing and authentication.
This paper consists of two parts. The first part documents a survey into the distribution of Learning Management Systems (LMSs) within Australian universities. It aims to establish a context (or current snapshot) of LMSs in which assessment is embedded. The second part of the paper looks more closely at the issue of assessment. It documents the development of a number of benchmarks for evaluating the assessment capabilities of a Learning Management System, and uses these benchmarks to evaluate the two leading products currently on the market.
At the beginning of 2004 a survey was conducted into the various LMSs in use at Australian universities. For the purposes of this study, 44 Australian universities and other higher degree granting institutions were targeted (including the Australian Film, Television and Radio School; the Australian Maritime College; and the Australian Defence Force Academy). A list of these targeted institutions can be found on the Australian Vice Chancellors' Committee Web site [HREF1] and the Department of Education, Science and Training Web site [HREF2]. Data on the entire population of higher degree granting institutions was gathered.
Relevant personnel in each institution were contacted by telephone and asked the following questions:
Similar research data had previously been collected by the Australasian Council on Open, Distance and E-Learning (ACODE) [HREF3] but for the purposes of this research the ACODE data had several shortcomings. Firstly, it failed to target all higher degree granting institutions in Australia. Secondly, the data was two years old (although it is understood that ACODE is re-conducting their survey at the time of writing). Thirdly, the questions asked had some overlap but were not adequate to fully answer the questions asked by this research.
The desired outcomes of the survey were as follows.
Figure 1 represents the percentage of institutions that use a particular product as their primary, centrally administered, LMS. All institutions, with the exception of those with no online teaching and learning presence, had a primary, centrally administered LMS.
Four institutions (9%) reported that they did not have a primary LMS. These were either private institutions, specialist institutions, very small in size, geographically remote, or institutions possessing a combination of these factors. All traditional, publicly funded universities in Australia had a centrally administered, online LMS.
Within the 40 institutions with a centrally administered LMS, it is clear that WebCT [HREF4] and BlackBoard [HREF5] dominate the landscape. One specialist institution used Fourpoint Learning [HREF6], which they claimed was more appropriate for their requirements. However, this institution also used WebCT to fulfil some of their online teaching and learning needs.
The remainder of the institutions (five institutions) used in-house products. The use of in-house products will be discussed in more detail later.
Not all institutions make use of a single LMS. Figure 2 shows the distribution of other LMSs in use.
The survey data revealed that 16 institutions (40% of institutions possessing a primary LMS) had at least one secondary LMS (three institutions had more than one secondary LMS). In some cases, these secondary LMSs were residual and there were plans to phase them out; in other cases the institution had requirements that could not be met by the primary LMS. The number of instances of these secondary LMSs is shown in Figure 2.
Figure 3 shows the number of years each institution has been using their primary LMS.
This figure indicates that institutions are more likely to have changed their LMS relatively recently than to have been using the same product for a long period of time. This field is less than a decade old, and as products change and develop a degree of fluidity is to be expected. All three of the institutions that have been using the same LMS for eight years or more are using an in-house product. The three institutions that have been using the same LMS for seven years all use WebCT, which is indicative of its early entry into the marketplace. There is a significant difference between BlackBoard and WebCT (p = 0.029) with WebCT having been used longer than BlackBoard.
A significant number of institutions (43%) have changed from the LMS they started out with, indicating a degree of dissatisfaction with the original product. Figure 4 shows which LMSs were in use by institutions that have since changed their primary LMS.
The largest trend displayed here is away from in-house products. Other products that were on the market early but have not survived well are TopClass and Lotus Notes. Of the two main products currently in use, WebCT and BlackBoard, there have been about three times as many institutions moving away from WebCT as there have been from BlackBoard. The platforms in the 'Other' category consist of FirstClass and Course-In-A-Box.
Figure 5 illustrates the distributions of institutions who are planning on changing their existing LMS in some way.
22 institutions (50%) are satisfied with their current product and are not planning on changing it in the immediate future. The other 50% are not satisfied with their current product and are either definitely planning on changing, planning on upgrading to a later version, planning to expand their existing functionality, or are looking at other options (i.e. they have no definite plans at this time but are interested in changing should a more suitable product present itself). There is no significant relationship between the current LMS is use and whether an institution is planning to change.
An analysis was also performed to compare institutions that have previously changed from a different product line with those that have stayed with the product they initially chose, in order to determine whether this difference has any noticeable effect on what an institution is planning to do in the future. There was no significant correlation between whether an institution has previously changed their LMS and whether they are planning to change again in the future, i.e. just because an institution has made the effort of changing, they are not necessarily satisfied with the product they have changed to.
Figure 6 shows the products that institutions are planning on changing or upgrading to.
One of these (Manhatten [HREF7]) is an open-source product. Two institutions were still in the process of finalising legal agreements and were not prepared to reveal their decision at this time. The majority of institutions planning to change had not yet decided on what product they are going to change to. These institutions all fell into the "Looking at options" group. The institutions planning on using WebCT Vista were all current users of WebCT and were planning on upgrading. Of the institutions planning on using BlackBoard, one was a current BlackBoard user planning to upgrade; the other was looking at BlackBoard as a likely option.
Figure 7 shows the existing LMSs used by those institutions that are either definitely planning to change, or looking at options.
Figure 1 showed that there are currently five primary in-house LMSs in use. This figure shows that (of the five current in-house installations) three of these are planning on changing. None of the five changed to an in-house product from another product - they started off using an in-house product. Furthermore, as shown in Figure 4, one-third of the institutions that have changed products moved away from using an in-house product. This demonstrates a significant trend away from the use of in-house products in Australian institutions.
It can also be noted that of the current WebCT installations, almost half are planning to change or looking at other options. Compare this with BlackBoard installations, in which no university is planning to change or looking at other options. This represents a significant difference between BlackBoard and WebCT users (p = 0.031), with WebCT users being more dissatisfied with their product.
Of further interest is that 71% of the BlackBoard installations had been using another product (in 20% of cases that other product was WebCT). In contrast, only 35% of WebCT installations changed to WebCT, the rest starting out with it. Only 14% of these (one instance) was previously using BlackBoard.
Although WebCT has the largest number of university installations, the number of universities who have changed from WebCT is quite large, as are the number planning to change in the future.
These results indicate a recent trend away from WebCT and towards BlackBoard.
The third outcome of this research was to identify how integrated the use of LMSs are at the institution as an indication of the effectiveness of the LMS to satisfy the needs of all the institution's users. A comparison was performed between the primary LMS of institutions that also supported one or more secondary LMSs. The sample size was not large enough to perform a statistical test, though the distribution indicated little difference between WebCT, BlackBoard or the other LMS products.
"For most students, assessment requirements literally define the curriculum" (James, McInnis & Devlin, 2002). However, staff often neglect to treat assessment with the same level of importance. James, McInnis & Devlin go on to state that "assessment [should be] treated by staff and students as an integral and prominent component of the entire teaching and learning process rather than a final adjunct to it ... Assessment is often considered once other curriculum decisions have been made."
In the Web-based arena, there is far greater potential to integrate assessment with coursework. The limitations of executing traditional assessments, such as the need to co-ordinate an entire group of students to participate at the same time in the same location, or to organise an appropriate location, are gone. This opens up opportunities to provide assessment choices that simply aren't feasible using the traditional assessment metaphor.
The requirements of assessment are also changing. McLoughlin & Reid (2002) note that "assessment processes are now in the limelight, with increasing emphasis placed not on testing discrete skills or on measuring what people know, but on fostering learning and transfer of knowledge." Traditional assessment approaches may be okay at testing a students' ability to reproduce learned facts, but it is generally agreed that they are not so good at assessing these new knowledge requirements.
Students are also instrumental in bringing about changes is assessment requirements. A report by the Higher Education Review Process points out:
There is also an increasing expectation by students that they should have choice - in terms of subjects, modes of delivery, methods of assessment and time spent on campus. Many academics believe that the heightened expectation of choice stems from what James (2001) describes as: "... a consumerist pattern of thinking among students, which [academics] believe is a direct result of the expectation that students contribute a greater proportion of the cost of their education." (Higher Education Review Process, 2002)
In order to provide sufficient support for these new and different requirements, an LMS must possess advanced assessment functionality. In the remainder of this paper, a list of "benchmarks" for evaluating the quality of the assessment components of an LMS will be proposed. These will then be used to subjectively rate the two leading LMS products on the market today.
A review of the literature highlights many assessment related factors that are relevant to providing quality Web-based assessment. These factors have been identified and grouped, and from these groups a number of benchmarks have been extracted. The following groupings were identified: Timing, Security, Feedback, Validity, Objective Testing, Authenticity, Technology, Accessibility, and Interoperability.
It should be noted that these benchmarks come from the literature on traditional and Web-based assessment and from current practice. They are by no means exhaustive or definitive, and due to the developing nature of Web-based assessment, will most likely evolve over time.
Each benchmark includes scoring rubrics. In each case, the score given to the benchmark will be from 0 (no support for feature) to 1 (extensive support for feature). In all, 30 benchmarks were determined. An LMS evaluated against these benchmarks would therefore be capable of scoring from 0 (provides very poor support for assessment) to 30 (provides exceptional support for assessment).
No weightings have been applied to any of the benchmarks. As it stands, each benchmark is "worth" as much as any other benchmark. Individuals or organisations wishing to use the benchmarks to perform their own evaluations of LMSs may choose to apply weightings as appropriate to their own requirements.
On the whole, these benchmarks may be used to subjectively rate an LMS. They are not intended to be an objective analysis or to categorically state that one LMS is better than another. In particular, an institution's individual requirements will determine which benchmarks are more important, and they may chose to rate some benchmarks more highly than others.
The final benchmarks determined are described in detail below.
One positive aspect of Web-based assessment for students is that "in terms of scheduling, computer-based systems offer much greater flexibility ... Candidates can be entered for an exam right up until the time of the examination ... The system also offers the potential to move to 'on-demand' testing" (Sealey, Humphries & Reppert, 2003: 372). These aspects of timing are quantified in the following benchmarks.
Benchmark 1: Tests can be scheduled to be taken at any time, or at specific times [Scoring: 0 = no scheduling support, 0.25 = minimal scheduling support, 0.5 = moderate scheduling support, 0.75 = good scheduling support, 1 = exceptional scheduling support]
Benchmark 2: Test schedules can be changed on a student-by-student basis [Scoring: 0 = no support for individual scheduling, 0.5 = some support for individual scheduling, 1 = extensive support for individual scheduling]
Benchmark 3: Test duration can be specified as unlimited, or given a specific time limit [Scoring: 0 = only one option is supported, 1 = both options are supported]
One aspect of security that the literature has largely avoided is the issue of identifying that students are who they say they are (otherwise known as authentication). In a physical examination situation, most (if not all) universities these days require the presentation of photo identification that is used to match the person taking the exam with a student ID number. Most LMS installations use passwords to authenticate students, however, passwords are notoriously insecure. Forms of advanced biometric authentication, such as keystroke analysis [HREF8] are necessary.
Benchmark 4: Support for biometric authentication [Scoring: 0 = no support, 0.5 = some support, 1 = extensive support]
Another aspect of security concerns the detection of plagiarism. Because exam data is stored in electronic format, it can be more easily analysed to detect instances of plagiarism. Sealey, Humphries & Reppert (2003: 373) report that "this analytical potential was illustrated during the project, when an unusually high degree of correlation between two candidate's answers was detected."
Benchmark 5: Plagiarism detection facilities are present, that not only perform statistical checks on results, but can also check for instances of natural language plagiarism in submissions themselves [Scoring: 0 = no support, 0.5 = statistical or natural language plagiarism detection supported, 1 = both statistical and natural language plagiarism detection supported]
Many authors comment on the importance of providing effective feedback (see, for instance, Peat & Franklin, 2002; Alexander, et al., 2002; Reushle, et al., 1999; James, McInnis & Devlin, 2002; Kendle & Northcote, 2000).
Feedback needs to be timely (immediate) and of high quality. Kendle & Northcote state, "Bernard and Naidu (1992) found that providing students merely with right/wrong feedback 'had virtually no effect' (p. 2). However, meaningful, substantive correction met with much greater results" (Kendle & Northcote, 2000).
The following benchmark attempts to quantify this issue.
Benchmark 6: Facility to provide extensive feedback on all assessment tasks, both question by question and at the end of the task, including varied feedback for individual questions based on the students' responses [Scoring: 0 = no feedback options, 0.25 = minimal feedback options, 0.5 = moderate feedback options, 0.75 = good feedback options, 1 = extensive feedback options]
James, McInnis & Devlin (2002) note that "when questions are raised about academic standards they are often associated with assessment practices, in particular student grading." The grading of assessments is an important issue in both Web-based and traditional formats. However, web-based assessment has the potential to surpass traditional assessment in terms of the validity of results, and an LMS should exploit this potential.
Sealey, Humphries & Reppert (2003) describe a system they developed (MarkerBase) to assist with the marking of computer-based exams. They made a number of simple though important implementation decisions, which are reflected, in the following benchmarks.
Benchmark 7: Seamlessly integrate automatically generated marks with those marks requiring human intervention in the marking process [Score: 0 = no, 1 = yes]
Benchmark 8: Provide the ability to mark tests one candidate at a time or to mark all responses to a particular question before moving on to the next question [Score: 0 = supports only one marking option, 0.25 = minimal support for both options, 0.5 = moderate support for both options, 0.75 = good support for both options, 1 = extensive support for both options]
Benchmark 9: Provide a marking scheme for each question that is displayed when the question is marked [Score: 0 = no marking scheme support, 0.5 = some support for marking scheme, 1 = extensive support for marking scheme]
Support for a number of different question types is generally seen as important. Bull, et al. (2002) notes that "there are many types of objective question; the latest IMS QTI specification (v 1.2) lists 21 types." [HREF9]. Thirteen specific question types are mentioned in the IMS QTI FAQ. These include "multiple choice, true/false, multiple response, image hot spot, fill in the blank, text short answer, essay text, numeric entry, slider, drag and drop, order objects, match item and connect the points. Many other types of question are also possible." [HREF10]
Benchmark 10: Provides support for a wide variety of question types [Scoring: 0 = no question types supported, 0.5 = moderate number of question types supported, 1 = 13 or more question types supported].
McLoughlin & Reid describe how they modified multiple choice questions to make them more demanding by introducing the following changes:
From this, the following benchmarks have been determined:
Benchmark 11: Multiple-choice questions can be configured to support multiple responses [Scoring: 0 = no, 1 = yes]
Benchmark 12: Multiple-choice responses can be assigned different weightings, including negative weightings [Scoring: 0 = no, 0.5 = positive weightings only, 1 = both positive and negative weightings]
McLoughlin & Reid also found that "... short answer items were the most demanding ..." (McLoughlin & Reid, 2002). As such, there should be good support for the automated marking of short answer items, more so than simple exact pattern matching.
Benchmark 13: Extended support for marking of short answer items [Scoring: 0 = none, or exact pattern matching only, 0.5 = basic regular expression pattern matching, 1 = advanced pattern matching capabilities]
Whittington, Bull & Danson mention that there are a number of advantages to using Web-based assessment. Some of these advantages include:
These points lead to the development of further benchmarks.
Benchmark 14: Objects (such as images, Java applets, Flash scripts, etc.) can be used in questions and responses [Scoring: 0 = no, 0.5 = used in questions only, 1 = used in questions and responses]
Benchmark 15: Support for randomisation of question banks [Scoring: 0 = no, 1 = yes]
Benchmark 16: Support for adaptive testing is provided, that is, questions are chosen based on their inherent difficulty level in response to how well the student is performing [Scoring: 0 = no, 1 = yes]
Benchmark 17: Questions can be manually assigned a difficulty level, or difficulty level can be calculated automatically based on past students' performance on the question [Scoring: 0 = no, 1/2 = only one method of calculating difficulty level supported, 1 = both methods of calculating difficulty level supported]
Benchmark 18: Question item validity analysis can be performed to determine poorly developed questions or distractors [Scoring: 0 = no analysis supported, 0.25 = minimal analysis supported, 0.5 = moderate analysis supported, 0.75 = good analysis supported, 1 = extensive analysis supported] (See also Bull, et al., 2002)
The number of times students can submit quizzes should also be variable (Whittington, Bull & Danson, 2000; McLoughlin & Reid, 2002), resulting in the following benchmarks:
Benchmark 19: Quizzes can be set-up to be taken a variable number of times, from once to infinite [Scoring: 0 = no options, 0.5 = some options, 1 = extensive options]
Benchmark 20: If used summatively, final scores should be able to be determined in a number of ways, such as final attempt, best attempt, average, etc. [Scoring: 0 = no options, 0.5 = moderate number of options, 1 = comprehensive number of options]
James, McInnis & Devlin (2002) provide evidence of the need for an additional benchmark for online exams. The subsequently developed benchmark is:
Benchmark 21: Can all answers be changed before final submission of a test [Scoring: 0 = no, 1 = yes]
Many authors highlight the importance of providing authentic assessments such as role-plays, simulations, portfolios, and so on (Reushle, et al., 1999; James, McInnis & Devlin, 2002; Kendle & Northcote, 2000; Bennett, 1998: 8). Authentic, or situated, learning is when "learning occurs ... in the specific context in which it is intended to be used" (Kendle & Northcote, 2000) and gives "students a chance to learn and be assessed under conditions similar to those encountered by practitioners" (Bennett, 1998:13).
Due to the inherent flexibility in designing authentic assessments, the following benchmark is quite general.
Benchmark 22: The LMS can be easily extended to support third-party authentic assessments such as simulations, while allowing results to be seamlessly integrated with those of built-in assessments. [Scoring: 0 = no, 1 = yes]
"Technology failures were the most negative factor for candidates ... The vast majority of technology failures were due to PC system problems" (Sealey, Humphries & Reppert, 2003: 368). Sealey, Humphries & Reppert reported that the product they developed for conducting computer based exams (ExamBase) "... has built-in 'failsafe' features: these enable a candidate whose PC has crashed during an exam to move to a spare PC and carry on from the point at which the interruption occurred" thereby attempting to minimise the impact of this type of failure. This leads to the following benchmark:
Benchmark 23: Support for restarting a test from the point of interruption should be provided in the event of a PC or browser crash [Scoring: 0 = no support, 0.5 = some support, 1 = full support]
Sealey, Humphries & Reppert (2003: 375) also made the finding that "another concern, unique to computer-based exams, is the impact of the performance of individual computers. Potentially, a candidate with a 'faster' computer [or network connection] would have more time to answer the questions than a candidate with a slower computer [or network connection]."
Benchmark 24: The speed of individual computers and network connections is taken into account when timed tests are undertaken [Scoring: 0 = no, 1 = yes]
Sealey, Humphries & Reppert (2003: 366) found that "the single most critical factor [in implementing Web-based exams] is that the user interface must not be a barrier to assessment ... and should not require anything other than the most basic IT skills." To reflect this, the following benchmark has been developed:
Benchmark 25: User interface is simple and easy to use, even by novice computer users [Scoring: 0 = difficult to use, 0.5 = okay to use, 1 = simple to use]
The following benchmarks for blind and partially sighted users (as mentioned by Harris, et. al., 2002) have also been developed.
Benchmark 26: Assessments comply with W3C accessibility guidelines [Scoring: 0 = no support, 0.5 = moderate support, 1 = extensive support]
Benchmark 27: Support for blind and partially-sighted users, who may need to change colours, fonts, or use text-to-audio translation, is provided. [Scoring: 0 = no support, 0.5 = moderate support, 1 = extensive support]
The ability to exchange question data and other assessment resources between different systems, and access to online pools of questions, is seen as a critical success factor by a number of authors (for example, Bull & McKenna, 2000; White & Davis, 2000; Bull, et al., 2002).
Benchmark 28: Support for common standards, such as IMS QTI and SCORM [HREF11], for exchanging assessment data between different systems, is provided [Scoring: 0 = no support, 0.5 = partial support, 1 = comprehensive support]
Benchmark 29: Access to online question banks is provided [Scoring: 0 = no, 1 = yes]
Benchmark 30: Ability to exchange pools of questions between colleagues [Scoring: 0 = no support, 0.5 = moderate support, 1 = extensive support]
The most widely used LMSs in use at Australian Universities are WebCT and BlackBoard. Version numbers of these products in use not withstanding, they account for 77% of all LMS installations at Australian Universities. This section will evaluate the latest versions of these two products using the benchmarks for Web-based assessment as previously determined.
Figure 8 shows the ratings for BlackBoard 6.1 and WebCT Vista against the 30 previously defined benchmarks.
|
This represents a significant difference between BlackBoard and WebCT (p = 0.02), with WebCT being the better performer.
The survey results in the first section of this paper showed that the current environment surrounding LMSs is quite fluid, with much changing and tweaking of systems. This indicates a certain level of dissatisfaction with current LMS implementations.
Overall, WebCT and BlackBoard dominate in Australian higher education institutions. There is evidence to suggest that in the early days of LMS implementations WebCT had a commanding lead, but that over the last few years BlackBoard has been eating into that lead and is set to do so even further in the near future.
In terms of an LMSs ability to provide a high quality assessment experience, 30 benchmarks were developed, from a review of the literature and current practice, in order to subjectively evaluate an LMS. Neither of the two main platforms in use in Australian higher education institutions, WebCT or BlackBoard, rated highly. However, WebCT had a significant advantage over BlackBoard with a score of 66% compared with BlackBoard's score of 51%. This advantage is only subjective, and must be seen in context, as individual institutions' ratings may vary depending on their needs. It is interesting to note, however, that BlackBoard outperformed WebCT on only two benchmarks (benchmark 5 - plagiarism detection, and benchmark 14 - the ability to use objects such as images in multiple choice responses), so unless these options were rated very highly by an institution, WebCT would be likely to emerge on top.
Bull, et al. (2002) mentions that "a clear lesson is not to assume that because a system is widely accepted it has been developed with any clear strategy, nor that it is necessarily an ideal solution." The conclusion of this research, in terms of student assessment, supports that statement.
Alexander, S., Kandlbinder, P., Howson, E., Lukito, L., Francois, A. & Housego, S. (2002). "Simassessment: Enhancing Academics' Understanding of Assessment Through Computer Simulation" in Proceedings of ASCILITE 2002, Auckland. Available online [HREF12]
Bennett, R. E. (1998). Reinventing assessment: Speculations on the future of large-scale educational testing. Princeton, NJ, Policy Information Center, Educational Testing Service. Available online [HREF13]
Bull, J. & McKenna, C. (2000) "Computer-assisted Assessment Centre (TLTP3) Update" in Proceedings of the Fourth International Computer Assisted Assessment Conference, Loughborough. Available online [HREF14]
Bull, J., Conole, G., Davis, H.C., White, S. Danson, M. & Sclater, N. (2002). "Rethinking Assessment Through Learning Technologies" in Proceedings of ASCILITE 2002, Auckland. Available online [HREF15]
Harris, R. A., Beevers, C., Ewins, R., Haywood, J. & McAteer, E. (2002). "Encouraging Research into On-line Learning and Assessment" in Proceedings of ASCILITE 2002, Auckland. Available online [HREF16]
Higher Education Review Process (2002). Striving for Quality: Learning, Teaching and Scholarship. Department of Education, Science and Training, Canberra. Available online [HREF17]
James, R., McInnis, C. & Devlin, M. (2002). Assessing Learning in Australian Universities, Centre for the Study of Higher Education, University of Melbourne. Available online [HREF18]
Kendle, A. & Northcote, M. (2000). "The Struggle for Balance in the Use of Quantitative and Qualitative Online Assessment Tasks" in Proceedings of ASCILITE 2K, Coffs Harbour. Available online [HREF19]
McLoughlin, C. & Reid, N (2002). "Seachange: Design of Online Quiz Questions to Foster Deep Learning" in Proceedings of ASCILITE 2002, Auckland. Available online [HREF20]
Peat, M. & Franklin, S. (2002). "Use of Online and Offline Formative and Summative Assessment Opportunities: Have They Had Any Impact on Student Learning" in Proceedings of ASCILITE 2002, Auckland. Available online [HREF21]
Reushle, S., Dorman, M., Evans, P., Kirkwood, P., McDonald, J. & Worden, J. (1999). "Critical elements: designing for online teaching" in Proceedings of ASCILITE 99, Brisbane. Available online [HREF22]
Sealey, C., Humphries, P. & Reppert, D. (2003). "'At the Coal Face' Experiences of Computer-Based Exams" in Proceedings of CAA Conference, Loughborough. Available online [HREF23]
White S. & Davis, H. (2000). "Creating large-scale test basks: a briefing for participative discussion of issues and agendas" in Proceedings of the Fourth International CAA Conference, Loughborough. Available online [HREF24]
Whittington, D., Bull, J. & Danson, M. (2000). "Web-Based Assessment: Two UK Initiatives" in Proceedings of AusWeb 2K, Cairns. Available online [HREF25]