Bronte Moran, Computer Aided Learning Development Officer, South Australia Police Academy, Strathfield Terrace, Taperoo, South Australia, 5017. Email: moran.bronte@police.sa.edu.au
Quality assurance for the development of online learning is a critical issue for the South Australia Police (SAPOL). Standards are identified and monitored at two levels - the organization or macro level and the product development or micro level. At the macro level, tools and procedures are used for quality assurance for development of all training, regardless of the mode of delivery. At the micro level, specific standards for the development of online learning have been written and tools created to measure online learning against these standards. The standards are based on cognitive and adult learning principles, instructional and web design principles and an understanding of the learner and leaning environment. Tools such as guidelines, templates and defined procedures provide for quality assurance in managing the development process of online learning and the quality of the completed online learning product.
Online learning is a recent addition to the training delivery options used
by the South Australia Police (SAPOL). The first online learning product, Firearms
Identification and Safety, was made available to the members of SAPOL in October
2001. This was followed by Minor Shop Theft - respond and proceed in November
and Searching of Persons in Custody in February 2002.
These three products are information sessions that are delivered via the Intranet
and completed entirely online. The latter two provide information and training
for police officers throughout the State to learn about changes to legislation
and police procedures. SAPOL has 4,753* active members,
of which 3,868 are police officers. When new legislation has been proclaimed,
the training must be delivered in a timely manner so that all police officers
are able to work according to the new law by its implementation date.
The development and delivery of all online learning in SAPOL is managed by
the Computer Aided Learning Development Officer (CALDO). A key responsibility
of the CALDO is to ensure that the online learning products developed meet the
highest standards SAPOL can deliver. Quality assurance has a significant impact
on the development process. Products are evaluated on a regular basis to provide
the assurance that they will satisfy identified quality standards.
Prior to the development of the first online product, therefore, the focus of
online development was the identification of standards appropriate for SAPOL
and tools to ensure that those standards are consistently achieved and reviewed.
SAPOL provides ongoing and wide-ranging training opportunities to its members. Police officers begin their career by completing an intensive cadet training program at the Police Academy. Throughout their employment with SAPOL, they frequently receive training to keep their skills and knowledge up-to-date. This training is provided by over 160 corporately approved training courses that are offered on a regular and ongoing basis. In addition, many information sessions are presented to meet one-off information and training needs, such as changes to legislation or corporate procedures.
The training can be divided into three categories:
Establishing methods to achieve consistent quality across the organization is an important issue. SAPOL has recognized that "ensuring quality requires specific preparation and support systems" (National Education Association, 1999). For the development of online learning, this preparation and support occurs at two levels:
At the macro level, quality assurance is broad-based but provides a way of thinking that supports the more precise tools employed at the micro level. The next section of this paper, Quality assurance - the macro level, describes the tools used organization wide to establish a quality framework for training. The section Quality assurance - the micro level will examine those tools used to ensure that each online learning product meets established quality standards.
SAPOL has adopted nationally recognized standards for the development and delivery of training by becoming a Registered Training Organization (RTO) within the Australian Quality Training Framework (AQTF). The key objective of this Framework is to provide a basis for a nationally consistent high quality vocational education and training system (Australian Quality Training Framework, 2001). As an RTO, SAPOL is obliged to comply with the set of 12 standards listed in Appendix 1. Most of these relate to administrative responsibilities. Three of the standards, however, translate into an organizational structure, processes and procedures that provide SAPOL with its own operational framework for quality assurance. They are:
All training in SAPOL that is linked to promotion, an individual's ability to perform in a particular role or to their personal development must be approved. The Quality and Curriculum Management Section centrally manages these approvals and then conducts audits of its continued development and delivery, as illustrated in Figure 1. Approved training is reviewed for content and instructional needs at least every two years, and to ensure that the training continues to be aligned with corporate objectives.
To further support the quality of training in SAPOL, the Section
SAPOL has 209 members who have obtained a Certificate IV in Assessment and Workplace Training. This is the currently recognized qualification for vocational workplace training. SAPOL has customized the Certificate to include additional competencies to meet the organization's training needs, for example, extra requirements regarding occupational health, safety and welfare and the ability to demonstrate a skill in the workplace.
SAPOL has to rely on its job experts to deliver the specialized training. These subject experts, therefore, are also qualified as training experts, with the ability to plan, develop, deliver and assess training. An additional 258 members, who have been awarded a Statement of Attainment to train small groups, can assist to deliver the training.
The awarding of training certificates does not of itself ensure that quality training is developed, but it does provide SAPOL with a knowledge base in learning principles and instructional methodologies.
A key element of the development and approval process is the preparation of curriculum documents. These documents are prepared for all approved training, regardless of the mode of delivery. This process keeps the development of training focused on the required learning outcomes rather than the method of delivery - the technology does not drive the training. Once training needs and learning outcomes have been defined, the training is developed for the most appropriate delivery options available. Templates, examples, guidelines, consultation and training are used to manage the process and to provide a consistent level of quality between the documents.
The result is training that has been developed by qualified instructors who have incorporated learning theories and instructional methodologies to meet the training needs of the organization and its members. In some cases, the desired learning outcomes and instructional needs will suggest that the training be delivered completely or partially online. The development of the online learning product is supported and monitored by a selection of more precise tools and methodologies to ensure that the final product meets identified quality standards.
Alley and Jansak (Alley& Jansak, 2001) stated that:
"The instructor's best strategy is to arm himself with a few sturdy principles, to be guided by a translation of these principles into practices effective in a web-based environment, and to think creatively while using specific application methods as a point of reference." (p.4)
SAPOL uses a similar approach by establishing practical guidelines for assuring quality in the development of online learning products. Quality is based on core principles of learning and design but is assessed from the viewpoint of the students. To translate the principles into practical guidelines, we have attempted to find a balance between the core principles and the environment in which the learning will occur by:
The Institute of Higher Education Policy in the United States published a list of benchmarks to ensure quality in the overall development and delivery of Internet-based distance education (Institute for Higher Education Policy, 2000. HREF1). This document includes benchmarks that focus on the institutional or organizational support of Internet-based learning, including training and support resources for instructors and students. Three of the benchmarks - Course Development, Teaching/Learning and Course Structure - relate specifically to the development of online learning products. These benchmarks are intentionally broad and encapsulate sound principles. While they provide little help as a practical guide to the online learning developer, they identify the key areas in which guidelines are needed.
The principles of cognitive learning and what is known about how adults learn are the driving forces in the design of online learning in the workplace. From the literature and experience, ten key characteristics of an effective learning situation that apply in the context of the development of online learning were identified: (Fardouly, 1998.HREF2)
In addition, consideration was given to other characteristics that are specific to adult learners. Adult learners are internally motivated to learn and are motivated by the usefulness of material. They prefer: (Zemcke. & Zemcke, 1995)
The presentation of online learning must also be based on principles of web design. These are largely based on issues of readability, with additional concerns based on technological issues, such as accessibility and the ability to download or view the material. Many guidelines on the do's and don'ts of web design are available and several are included in the references (HREF3). These principles, however, build on those that have always supported good presentation design:
The principles discussed so far relate to the development of online learning in any workplace. Principles that must also be considered are those that the organization supports in relation to human resources: occupational health safety and welfare, equity and diversity:
An understanding of the principles of learning, instruction and web design support the development of online learning that has some intrinsic quality. "Quality expresses itself, however, through the viewpoints, values, and needs of the course consumer." (Alley. & Jansak, 2001, p.3) We need, therefore, to understand the conditions of the learning environment to create effective, and therefore quality, learning products.
At SAPOL we have used several techniques to learn about the potential students and instructors who will use online learning and the environment in which they work and learn. The Computer Aided Learning Advisory Group provided support and advice for the first year of planning for online learning. This group consisted of representatives of training coordinators from both metropolitan and rural areas, instructors from the Police Academy, Information Systems and Technology (IS&T) and from Quality and Curriculum Management. The members of the group included both police officers and non-police officers. The purpose of the group was to keep the focus of online learning on the specific needs of the students and instructors. The group assisted with the development of online learning criteria that reflect SAPOL needs. The IS&T representatives also provided vital information regarding the information technology infrastructure.
Further information about the potential students and the playback or learning environment was obtained from a staff survey and visits to selected police stations. The results of the survey in April 2001 indicated that:
In many ways, the survey results illustrated a homogeneous culture within the organization. When survey questions were cross-tabulated with respondent characteristics, such as gender, rank, work status or location, no significant correlations were identified. It should also be mentioned that entrants to SAPOL must pass a computer literacy test and police officers generally have basic computer skills because all incident reports are prepared and filed electronically.
Visits to selected police stations in both rural and metropolitan areas confirmed many comments provided on the surveys. We were able to develop some clear images of the workplace learning environment, including busy work areas and large work demands over which the student has little control, and a desire for easily accessible training and resources that are available around the clock.
The next step was to use all the information gathered to develop tools and processes that would enable us to develop online learning products that are consistently of the highest quality that is effective for SAPOL.
When a course, information session or training program has been approved at the organization level, a request for the development of an online learning product can be submitted. This request includes a description of the purpose of the product and why the trainer expects that online delivery of the training will achieve the learning outcomes. The request for online development is then reviewed and prioritized by an online learning selection committee. The first phase of product development - the development process - begins as illustrated in Figure 2.
Processes that focus on quality assurance immediately come into play. In the first instance, the product is assigned to a learning development pathway based on characteristics that are common to similar types of learning, particularly with regard to assessment needs. A work breakdown structure template (PMI Standards, 1996. HREF5) is used for each pathway to ensure that all aspects of the development are included and to streamline the development process.
The work breakdown structure template for the development of an online information session includes 18 key activities that fall into the categories of:
The more complex stages of the development, such as content, instructional design and online preparation, are detailed with multiple steps to ensure that the product meets specified standards and supports teaching and learning strategies.
The use of a work breakdown structure template fulfills the multiple roles of providing quality assurance in managing the process of product development and the quality of the product itself. The tasks identified in the work breakdown are allocated to members of the product development team. In addition to the CALDO, this team includes a subject expert and qualified trainer.
The content, instructional design and web presentation are continually measured against several sets of criteria. The first is the set of learning criteria presented in Appendix 2. These criteria were identified as suitable for SAPOL by blending the principles of cognitive and adult learning, identified earlier, with the characteristics of the learners and learning environment highlighted by the survey and workplace visits. In the section of the criteria called For preparing content, for example, #7, "enable learning to be done in clearly defined increments, maximum of 30 minutes each" is critical for our learners because of the frequent demands on their time and the sharing of computers. In this way, we support the adult learning principle of enabling students to learn at their own pace and rate of comprehension. Similarly, the design of navigation, and having the content organised in meaningful sequences, is important because we expect the learners to frequently exit and re-enter the learning.
These criteria are used to highlight those features of content development that, in the final product, have a greater impact on the learning outcomes than might be expected in a face-to-face learning situation. The division of the criteria into the categories of 'preparing content' and 'web design' is an artificial one. All the criteria are used simultaneously when measuring the product against them. The division is helpful, however, in the early stages of development and reflects the division of tasks between the subject expert/trainer and the CALDO.
Features of instructional design are also included in the template provided to instructors for the development of the content, such as identifying goals and learning outcomes, welcome and introduction, formative evaluation and feedback. Web development guidelines and style sheets further support the elements of web design included in the criteria.
Whenever practical, templates are used to streamline the development process and to encourage consistent quality. As already mentioned, a Microsoft Word template is given to instructors or subject experts to guide the development of content. Templates are also used in web design and for recording changes identified during product previews and testing.
When the product is nearing completion, selected individuals, some of whom are familiar and some unfamiliar with the content, are asked to preview the product. Any necessary editing is done and group-based usability testing is completed. Further editing is carried out until the trainer who initiated the request for the product's development accepts the product.
Two steps complete the quality management cycle - product development reports and feedback about the product. Both these steps feed information back into the development process for improvements in the quality standards and the tools we use to determine how well we meet those standards.
Product development reports are prepared by the CALDO to address quality in
the development process. They summarize the difficulties, challenges and successes
in the development, the amount of time and cost for each stage of development
and make recommendations for changes in the development process.
Feedback on the final product is primarily received from feedback forms completed
by the learners. Reports on usage of the product are generated by IS&T using
WebTrends software. These reports provide a context for the feedback forms by
enabling an estimate of the percentage of members using the product who have
completed a feedback form and to speculate on the learning path followed by the
majority of those using the product. The combination of both types of feedback
were available for the launch of the product, Searching of persons in custody.
The online information session, Searching of persons in custody, was made available to all members of SAPOL via the Intranet without requiring any additional log on or password. The session provided training in new procedures and safeguards for searching persons held in police custody. The entire training could be completed in 45 to 60 minutes.
The session was organised into six main sections:
WebTrends software was used to measure the usage of the product. The WebTrends report does not provide an accurate record of the usage of Searching of persons in custody but does give some indication of trends.
During the first four weeks that the online information session was available, the Welcome (or home) page was the main point of entry and was viewed 1,292 times. Visits to the same page were recorded as 1,130 with only 78 of these being single access visits (exiting the site without viewing another page.) During these visits, the Declaration of Completion form, to be completed and submitted by members when they had completed the online training, was viewed 1,525 times. The numbers suggest, therefore, that approximately one third of police officers in SAPOL viewed at least part of the online information session.
At the time of writing, feedback forms had been received from 56 police officers of all ranks, from Probationary Constable to Chief Inspector. Most of the respondents (53) completed all sections of the information session and all respondents tried some of the self-assessment interactions that were included in the Learn it Now section. The respondents reacted positively to the online training as indicated by the results in Table 1.
Table 1: Selected results from online training feedback form| Feedback statement (N=56) |
Number of Responses |
% |
| They would recommend the online information session to their colleagues | 53 | 94.6 |
| It was easy to navigate: strongly agree / agree | 54 | 96.4 |
| All questions regarding the learning were answered | 48 | 85.7 |
| No additional training needed | 48 | 85.7 |
| Felt confident to correctly respond and proceed according to the new law and procedures | 53 | 94.6 |
The feedback forms also indicated that learning outcomes had been achieved.
When asked if they had any questions, regarding searching persons in custody,
that were not answered by the online session, the majority of the respondents
stated they did not. The same number of respondents also stated they did not
need any additional training to correctly implement the new searching procedures.
The success of the learning was further measured by the large percentage of respondents
who stated that, having participated in the online session, they felt confident
to correctly respond and proceed according to the new law and procedures.
The data on Searching of persons in custody, together with comments on the feedback forms and those received by email and in person, suggest that the product was well received by the members of SAPOL and, for those completing the feedback forms, has met their learning requirements. Acceptance by the learners and the ability to deliver training that has met their learning needs is a measure of the quality of the product and an indication that the criteria used in developing the product have functioned effectively as a quality assurance tool.
The criteria presented in this paper have been evaluated and are currently being expanded. We plan to expand the set of criteria to better assist instructors who are new to the development of content and instruction for online delivery. A more precise tool will also make it easier to measure the product against quality standards.
We are continuing to develop online learning products in response to legislative changes. The next phase is to expand our online learning scenarios to include other types of training and to pilot test their delivery by using a learning content management system.
The use of online learning in SAPOL is expanding and the number of trainers
who participate in its development will increase. The need to monitor quality
standards and to use tools to assure us that those standards are reached will
be critical.
* Numbers are correct as of 13 February 2002 and exclude
those employed on a casual basis or as trainees.
Alley, L.R. & Jansak, K.E. (2001) "The ten keys to quality assurance and assessment in online learning", Journal of Interactive Instruction Development, v.14, n.3, pp 3-18
Australian Quality Training Framework. (2001) Standards for Registered Training Organisations, Australian National Training Authority, Melbourne.
Fardouly, N. (1998) Principles of instructional design and adult learning: how students learn, UNSW, [Online, accessed 15 Jun 1999]. Available at: <http://www.fbe.unsw.edu.au/learning/instructionaldesign/studentslearn.htm>[HREF2] This has a summary of cognitive learning theories.
Guevin, C. (2001) "Visual architecture: the rule of three", Digital Web Magazine, [Online, accessed 16 May 2001]. Available at: <http://www.digital-web.com/features/feature_2001-4.shtml>[HREF4]
Institute for Higher Education Policy. (2000) Quality on the line : benchmarks for success in internet-based distance education, [Online, accessed 14 Dec 2000]. Available at: <http://www.ihep.com/Pubs/PDF/Quality.pdf>[HREF1]
National Education Association. (1999) "Distance education quality checklist", Technology Briefs, [Online, accessed 8 Jan 2002]. Available at: http://www.nes.org/technology/briefs/16.html (Not available online 14 May 2002)
PMI Standards. (1996) A guide to the project management body of knowledge, Project Management Institute, Upper Darby, PA. Available at: http://www.pmi.org/publictn/pmboktoc.htm [HREF5]
Zemcke, R. & Zemcke, S. (1995) "Adult learning: what do we know for sure", Training, vol. 32, pp 31-40
The following were retrieved from a simple search using Google:
Web Page for Designers, http://www.wpdfd.com
Page Resource.com, http://www.pageresource.com
Web Design Group, http://www.htmlhelp.com