Karen Woo, Project Officer, Centre for Flexible Learning [HREF1], Macquarie University, North Ryde, NSW, 2109 kwoo@laurel.ocs.mq.edu.au
Maree Gosper, Senior Lecturer, Centre for Flexible Learning [HREF1], Macquarie University, North Ryde, NSW, 2109 maree.gosper@mq.edu.au
Donna Gibbs, Associate Professor, School of Education [HREF2], Macquarie University, North Ryde, NSW, 2109 donna.gibbs@mq.edu.au Tim Hand, Principle Education Officer, Open Training and Education Network, TAFE NSW [HREF3] 51 Wentworth Rd, Strathfield, NSW 2135 tim.hand@tafensw.edu.au Sharon Kerr, Manager Online Education Services, Centre for Flexible Learning [HREF1], Macquarie University, North Ryde, NSW, 2109 sharon.kerr@mq.edu.au David Rich, Professor, Centre for Flexible Learning [HREF1], Macquarie University, North Ryde, NSW, 2109 david.rich@mq.edu.auResearch on learning object systems has traditionally centered on the development of standards for metadata, interoperability, and digital rights. The assumption seems to be that once the standards are accomplished, these systems will naturally be embraced by the intended users. The problem is that often these users have not been properly defined and have rarely been given a chance to speak about their needs and concerns. This paper reports on a study which invited teachers, educational developers and curriculum consultants from three educational sectors (school, vocational education and training, and higher education) in Australia to comment on learning objects and learning object systems. The findings of this study and the documented motivators and inhibitors to using a learning object system can guide designers towards developing user-friendly systems that demonstrate an understanding of the existing culture and the work practices of Australian teachers and other user groups.
The notion of learning objects as high quality, up-to-date learning resources that can be reused in a range of educational contexts and technical environments has immediate appeal to institutions as they engage with the realities of delivering quality education in an increasingly competitive market. Consequently, enormous effort has been poured into developing standards in the areas of metadata, interoperability and digital rights. Varieties of learning object systems have flourished to support different aspects of the trading and sharing of learning objects. However, it is becoming obvious that simply because these systems exist does not mean they would be embraced by all. In fact, lack of uptake is commonly experienced and it can be partly attributed to the neglect of user concerns. The end users of these systems have rarely been identified, and seldom given a chance to speak about their needs and concerns. The reason is that many of the learning object projects are driven top-down with market economies as prime motivators for system development. However, for these systems to be successful, they need to balance and integrate the goals of the institutions with those of the intended users. While much is known about the institutional drivers for their interests in these systems, this paper reports on a study which invited teachers, educational developers and curriculum consultants in Australia to comment on learning objects and learning object systems.
The study, entitled The Selection and Use of Learning Objects for Teaching: Users Perspectives [HREF4], was funded by the Interaction of IT Systems and Repositories project (IIS&R) and sits within the broader research and development being undertaken by the COLIS Consortium, an alliance of five Australian universities. COLIS is currently funded by the Department of Education, Science and Training. One of the aims in establishing COLIS was to develop a scalable standards-based model for institutional interoperability which enables the seamless sharing of resources for face-to-face and online learning, as well as for scholarly research purposes.
A key feature of the COLIS project was to demonstrate standards-based interoperability among online learning and information systems. The COLIS Demonstrator, the learning object system used in the study, was developed to achieve systems interoperability in order to provide users with access to multiple learning systems using a single sign-on. Users can acquire suitable learning objects from a Learning Object Exchange (LOX) or through a federated search gateway (which accesses content from distributed repositories) and import them into their Learning Object Management System (LOMS). From LOMS, teachers can export their learning objects to a Learning Management System (LMS) for their students to use. As such, the COLIS Demonstrator has pushed the notion of interoperability beyond the traditional boundaries, and achieved interoperability between learning object systems, not just between LMS. In addition, the COLIS Demonstrator has overcome technical hurdles to support users to input metadata and manage their digital rights.
While the COLIS Demonstrator embodies the qualities desired in classic learning object system literatures - accessibility, interoperability and digital rights management - it has become apparent that these qualities have cultural and educational implications that may impact on the potential for such tools to support teaching, and may ultimately impair the uptake of these systems. Each of these themes and their relations to the users will be explored in the following sections.
In the context of learning object systems, accessibility refers to the degree of ease for a learning object to be found in the repository by a teacher, student or possibly a computer agent. The most common method of facilitating accessibility is by acquiring metadata.
Standards bodies like IEEE, Cancore, and IMS have produced metadata standards that are specific to the education sector (Cancore 2002). The number of standards available reflects the wide range of opinions on the optimal number of metadata fields and the relative importance of different fields. One of the most comprehensive standards available, IEEE Learning Object Metadata (LOM), provides 65 fields in total (IEEE 2002). An issue facing institutions is that even with the proliferation of metadata authoring tools, it is still a time-consuming and costly endeavour to ensure all 65 fields of the IEEE metadata fields are inserted. At the same time, and somewhat paradoxically, users expressed concerns that the terminology was totally inadequate for the task of defining the complexities of learning resources and activities (Currier & Campbell 2002, cited in Philip & Dalziel 2003). The complaint led Philip & Dalziel (2003) to suggest the inclusion of secondary usage metadata , which includes information such as previous uses of the learning objects and users comments.
Before metadata implementers rush to add or delete fields from their schema, they need to know what pieces of information would be useful to Australian teachers when they are searching for and choosing learning objects, and what search criteria they would be using. An understanding of users' perspective on what constitutes useful information for them will enhance efficiency on the users' end and saves necessary effort for the implementers.
Traditionally, interoperability has been a topic in military training and workplace training, where developers were interested in developing content that did not need to be tied to a single LMS. Technical standards like SCORM have been developed precisely to actualise this goal (ADL 2002, AICC 2002, Cisco 2002).
Interoperable content not only gives developers the freedom to move between LMS when a more competitive product is available, it also allows content to be traded between companies who do not use the same LMS. As the standards become popular and interoperability becomes common, teachers and students would be expected to begin using learning objects developed by other institutions which have different administrative structure, division of disciplines, vocabulary, hardware and software, as well as assumed work practices around these objects.
Despite the potential economic advantages for institutions of sharing resources, it is unclear whether teachers perceive any advantages for themselves in adopting learning objects, and whether there would be additional workload and technical barriers they need to overcome before they could enjoy the perceived benefits. Once institutions resolve these issues for the teachers, system designers will need to support the teachers work by providing the right types of information to help them make decisions about which learning objects to use in their teaching contexts.
Five types of learning object exchanges can be identified in the current commercial and educational institutions, namely proprietary exchanges, commercial exchanges, free exchanges, shared exchanges and peer-to-peer exchanges (Johnson 2003). In general, current exchange systems for university and school audiences are implemented on a free or shared exchange model. Peer-to-peer exchange opens the option for practicing teachers to contribute their learning objects for others to use. In a study involving school teachers on the EduNet project, Littlejohn, Jung & Broumley (2003) found that both standardised resources and resources in the EduNet repository were used. Interestingly, the most popular resources in the EduNet repository were standardised resources, but it was a common experience for teachers to find the resources created by other practicing teachers to be more useful.
While it is desirable for teachers to contribute resources for the benefit of the others, it is unclear what would motivate them to share their learning objects using learning object systems. The assumption that sharing will happen automatically because of technological advances assumes teachers are intrinsically motivated to share, but a brief survey of the literature shows that this is often not the case. From Wetterling and Collis's (2003) report on their experience with working on the CANDLE project, where an academic community from twelve universities and colleges from seven countries shared learning objects, the major problems faced was were related to instructors generally being unwilling to reuse other people's resources, let alone share their own to others.
The reluctance of staff to share is a common problem in learning object projects. Koppi & Lavitt (2003) reported a similar experience at the University of New South Wales. Their Learning Resource Catalogue (LRC) provided a web application designed to enable teaching staff to describe learning objects in standard metadata terms which can be submitted onto an online searchable database. Staff demonstrated a reluctance to engage with the catalogue as they did not want to invest the time to understand the rationale and benefits of using learning objects - and this was before they even considered the time needed to upload and tag their learning objects. Some faculties deemed learning objects to be inappropriate to their learning and teaching programs, and others were reluctant to give away their intellectual property without any rewards.
It is clear that learning object systems that do not take heed of the concerns of practicing teachers will not be used. In the words of Norm Friesen (2003): "For the full potential of e-learning standardisation and infrastructure efforts to be realised, it is important that these efforts place significantly greater attention on existing educational practice, on issues of innovation adoption, and on the heterogeneity of educational activities and contexts in general."
One of the aims of the study, The Selection and Use of Learning Objects for Teaching: Users Perspectives was to provide a user perspective to the functionality and design of learning object systems.
The participants in the study were 53 teachers, curriculum consultant and educational developers across three education sectors: the school, vocational education and training (VET) and higher education. The school sector was represented by teachers from metropolitan NSW government schools and curriculum consultants; the VET sector by distance teachers, instructional designers and metadata librarians at the Open Training and Education Network (OTEN) of TAFE NSW; and the higher education sector by academics and educational developers from Macquarie University, Sydney. All participants were experienced in the use of information and communication technologies in their teaching.
A two-stage methodology was adopted. The first stage was one of orientation and discussion to identify issues, and the second stage was to consolidate opinion. In the first stage, participants attended a workshop in which they were introduced to the concept of learning objects and were invited to discuss the applicability of these to current teaching contexts. For the purpose of this research, learning objects were defined as digital resources for teaching and learning with user rights and descriptions (metadata) attached to them. The concept of the COLIS Demonstrator as a system for storing, managing, sharing and acquiring learning objects was also introduced and participants were invited to comment on their likelihood of using the system and to raise any questions, issues or concerns they had. Throughout the workshop, participants were asked to give their opinions and discuss issues relating to the content presented. Their comments were recorded on butcher's paper, which were later transcribed and collated by the first author.
In the second stage, issues arising from the workshop were explored further through an online questionnaire distributed to all participants, of whom 40 responded (representing a 75.5% return rate). Both quantitative and qualitative questions were used. Where appropriate, responses were categorised and basic statistical procedures were applied to obtain the count or the mean score for the questionnaire items.
Full details of the study are available in the full report located at http://www.melcoe.mq.edu.au/projects/Education.htm. Only the key themes relating to learning object systems that emerged from the findings are reported here. For ease of reporting, the term teachers refers collective to the participants in the study and include classroom and distance teachers, curriculum consultants and educational developers.
A major theme arising from the study was that the criteria teachers used to select learning objects are no different to those they apply when selecting traditional resources for their teaching. Teachers are primarily concerned with the object's potential to enhance educational efficacy and efficiency in their work. Therefore, it was seen to be important for learning object systems to provide high quality, relevant objects that could be accessed and acquired in a time efficient manner.
As users acquiring learning objects from a system such as the COLIS Demonstrator, teachers found the most useful features to be:
The concerns they had in relation to using such a system to locate and acquire learning objects reflected their concerns for educational efficacy and the need for efficiency. Their top four concerns were:
Another important theme that emerged was the presence of a general culture of sharing in education. However, the motivation to share learning objects with colleagues is dampened by a number of factors including:
Most of the inhibitors listed above could be addressed at the institutional level through changes to policies in relation to resource allocation and work practices. The most interesting issue for the present discussion is the issue of time constraint. By developing and implementing more user-friendly and efficient learning object systems, teachers would be able to contribute, as well as acquire learning objects more easily. The findings from the study revealed five main areas where more user-friendly systems designs could improve efficiency:
If teaching staff are required to contribute their learning objects to a learning object system, it is a necessary (but not sufficient) condition for the process to be short, intuitive and user-friendly. The version of the COLIS Demonstrator in the study made use of the Intralibrary System which has a very simple interface for uploading objects.
Figures 1 and 2 show the screenshots shown in the workshop to demonstrate the steps teachers need to go through when describing their learning objects and adding user rights.
| |
| Figure 1: IntraLibrary System - adding metadata for a learning object (general) |
| |
| Figure 2: IntraLibrary System - adding metadata for a learning object (digital rights) |
The process only required the teacher to enter a title and a description for the object, and a copyright statement they would like to accompany that object. By minimising the process of uploading a learning object, teachers' chances of participating in sharing is increased. The participants found the steps easy to follow, and many commented that the interface was intuitive. These are important qualities to be considered in the design of a system.
IntraLibrary was chosen for its simplicity and was deemed appropriate for demonstrating the concept of a streamlined process in digital rights management. However, the system assumes that teachers have the full rights to decide whether or not they would like to charge for the resource, and that they are capable of protecting themselves by writing or including appropriate copyright statements.
Unfortunately, the legal landscape surrounding the sharing of learning objects is far from being simple. The authors were aware of the hidden economical problems in the demonstrated process. For example, in minimising teachers' data-entry, it would be necessary to involve librarians and IT support personnel to complete the metadata record. A question as to who is going to pay for the librarian and IT support emerges. If it is the institution, then who will own the learning object - the creator or the institution? Despite these recognised problems, the authors persisted with the demonstration because it was deemed sufficient for the purpose. However, when the issues surrounding intellectual property rights were probed in the follow-up questionnaire, it was clear that teachers were uncertain about how intellectual property laws would affect them, and that uncertainty would deter a large proportion of them (40% of questionnaire respondents) from sharing their learning objects. When they were made aware that all components of a learning object needed to be copyright cleared before they could share them, 67.5% of respondents indicated they would not be inclined to share their learning objects if they had to undertake the clearance process themselves.
The results suggest that if institutions are interested in encouraging peer-to-peer sharing of learning objects, the relationships between the different roles of the players (teachers, technical support personnel, librarians and institutions) and the rights each has to the object will need to be clarified for all involved. Unless the interests of all involved are balanced and the outcomes and operational procedures are clarified, it is likely that the future use of learning object exchange systems will be severely impaired.
Teachers find the rights and ability to customise learning objects essential if they are to use these objects in their teaching. Thirty-four (85%) of the questionnaire respondents rated customisation as being either important or very important. The rationale for customisation was largely driven by educational requirements - for example, tailoring to teaching and learning objectives or students' needs. The list of likely customisations nominated by participants included adding specific, local examples, and changing the tone of the text to make it more personal and engaging. Very few were related to achieving technical interoperability, formatting or graphic design consistency.
In some current learning object systems, learning objects can only be customised by removing or rearranging predefined components. While this type of customisation simplifies the digital rights arrangements, this broad level of customisation is not likely to be sufficient to achieving teaching and learning goals. The absence of support tools compounded with the fear of breaching intellectual property laws may mean that some teachers will prefer traditional resources over learning objects. Both teachers' clear need to customise the details of the content for their localised, specific teaching objectives and students' needs must be considered in designing a usable learning object system.
Many participants nominated searching to be the most useful feature of a system (30% of questionnaire respondents), while others found it to be the cause of major concerns (10%). Both instances reflect the importance of a good search mechanism that allows users to quickly locate relevant learning objects in a learning object system.
The criteria used for browsing and searching for objects in a learning object system differed across sectors. In the school and VET sectors, participants reported that it would be very helpful if the objects were matched to school syllabuses or the VET competencies. Littlejohn et. al. (2003) reported similar findings. They found that after directly linking online resources and activities (learning objects) to specific learning outcomes, reuse was improved. In light of the educational efficacy and efficiency concerns of the teachers, it should not be surprising that the linkage is important because it allows teachers to quickly locate a learning object for a specific lesson they are teaching and evaluate its potential to help them achieve their teaching objectives.
In some of the workshops, participants suggested they would expect to search for learning objects in the same way that they search for an item in a library catalogue. Further investigation revealed that the desired criteria for searching included Subject, Keyword, Outcomes/ Competencies, Syllabus requirements, Media type, Sector, State/ Country, Institutions of creator(s) and Creators. Table 1 shows the search criteria in descending order of preference, and the IEEE LOM (Learning Object Metadata) fields associated with them. For each criteria the mean ranking is given which is calculated on a five-point likert scale, where 5 denotes strongly agree and 1 denotes strongly disagree.
| Table 1: Search criteria and their associated IEEE LOM fields | ||
| Search Criteria | Mean | IEEE LOM Field |
| Subject | 4.73 | (none) |
| Keyword | 4.13 | Field 1.5 General. Keywords Field 9.4 Classification. Keyword |
| Outcomes/Competencies | 3.98 | |
| Syllabus requirements | 3.85 | |
| Media type | 3.70 | Field 4.1 Technical. Format |
| Sector (School, VET, Higher Education) | 3.30 | Field 5.6 Education. Context |
| State/Country | 2.60 | |
| Institution of creator(s) | 2.43 | |
| Creator(s) | 2.40 | Fields 2.3.1 Lifecycle. Contribute Fields 2.3.2 Lifecycle. Role/ Entity |
When comparing the desired criteria with the available IEEE LOM fields, it is questionable whether existing metadata criteria meet the educational requirements of the end-users. The desired criteria are more than a wish list. Their availability translates into efficiencies for the potential users of learning objects in the search process. Given that teachers from all three sectors are time-poor, the provision of metadata which can contribute to efficiency is essential. If the metadata provided is not relevant to user needs, the uptake of learning objects will be severely restricted.
Metadata are often used not only for searching purposes, but also to help teachers make choices between learning objects. The workshop participants suggested that when making decisions about the suitability of a learning object, they would want information relating to the technical requirements and instructions for using the object, its intended student audience, the teaching/ learning role it plays, the associated aims and outcomes, review by peer/ curriculum/ syllabus authorities, and a record of previous use.
A mean ranking of the need and desirability for this information is given in Table 2 along with a comparison between the information teachers requested and the associated fields in the IEEE LOM.
| Table 2: Useful information on learning objects and their associated IEEE LOM fields | ||
| Useful information on learning objects | Mean | IEEE LOM Field |
| 1. Technical information such as hardware and software requirements, file size, download times | 4.28 | Field 4.2 Technical. Size Field 4.4 Technical. Requirements Field 4.6 Technical. Other Platform Requirements |
| 2. The student group for which it was developed (e.g. age, expertise, special needs) | 4.20 | None (Field 5.10 Education. Description?) |
| 3.The teaching and learning context associated with its use e.g., its role, pre-requisite knowledge required, supporting materials | 4.13 | None (Field 5.10 Education. Description?) |
| 4.The specific teaching aims and learning outcomes for which it was developed | 4.00 | Field 9.3 Classification. Description |
| 5. Technical instructions for students on how to use the learning object (if applicable) | 3.83 | Field 4.5 Technical. Installation Remarks |
| 6. A review provided by my teaching peers | 3.40 | Field 2.3.3 Lifecycle. Contribute. Date |
| 7. The life history of the object - details of creation, publication and last modification | 3.40 | Field 2.3.3 Lifecycle. Contribute. Date |
| 8. A review provided by a curriculum / syllabus authority | 3.23 | Field 8 Annotations |
| 9. A record of previous uses of the learning object | 3.13 | Field 8 Annotations |
The first four types of information listed in Table 2 all scored a mean of 4.00 or above, indicating a strong need for this type of information. It is highly recommended that learning object systems collect and display this information for its users in the selection process.
It should be noted that there may be difficulties for IEEE LOM implementers to display categories 2 and 3 (the associated teaching and learning contexts and its aims and outcomes) because the IEEE LOM specification has no designated fields for them. Nonethless, designers of metadata software can ask for this information and utilise a generic field, such as 5.10 (Education. Description) to store that information for user reference.
In addition to this ranking exercise, participants were invited to add any requests they had for additional information. Notable additions were information on copyright (which maps to IEEE LOM Field 6.2), costs (maps to IEEE LOM Field 6.1), feedback from students (missing from IEEE LOM), customisability (missing from IEEE LOM), contact information (missing from IEEE LOM), and other related activities or supplementary materials (maps to IEEE LOM Field 7). Lastly, it should also be noted that none of the participants has mentioned any of the fields under the IEEE Education category as being useful to them.
Like the desired search criteria, the availability of this additional information in a learning object system is essential in helping teachers to maximise their productivity in the analysis process required to make an effective match between objects and the teaching/ learning context.
In order for a learning object system to be adopted, it must be compatible with the workflow of its users. Actualising technical possibility is exhilarating, but without understanding the end-user's technical ability and their needs the system may be left unused. Indeed, despite generous funding for technical development around the world, the uptake of learning objects is still nowhere near being pervasive (Johnson, 2003).
This paper has explicated the user requirements for a learning object system. Seeing that efficiency is a critical condition for use, system designers are challenged to create simple and intuitive interfaces and processes that maximise teachers' productivity. Furthermore, understanding the teachers' concern for educational efficacy led to an appreciation of the importance of customisation. To be adequate in its support for teachers, learning object systems will need to take into account such practices and offer intuitive tools to help teachers in their endeavour to give the best educational experience to their students. Lastly, the brief comparison of the present metadata standards and user requirements reveal a general neglect of the users' perspective in the area. It is hoped that more consideration be paid to teachers' practices, and more research be carried out to understand the different needs of users from different education sectors and different cultures.
ADL (2002) "About SCORM". Available online: [HREF5]
AICC (2002) "AICC FAQ (Frequently asked questions)". Available online: [HREF6]
Cancore (2002) "Survey of Learning Object Metadata Implementations" Available online: [HREF7]
Cisco (2001) "Reusable learning object strategy: Definition, creation process, and guidelines for building Version 4.0". Available online: [HREF8]
Friesen, N. (2003) "Three objections to learning objects". Available online: [HREF9]
IEEE (2002, Jul 15) "Draft Standard for Learning Object Metadata". Available online: [HREF10]
Johnson, L.F. (2003, Jun) "Elusive vision: Challenges impeding the learning object economy". Available online: [HREF11]
Koppi, T. and Lavitt, N. (2003) "Institutional use of learning objects three years on: Lessons learned and future directions". Paper presented at the Learning Objects 2003 Symposium: Lessons Learned, Questions Asked, 24 June 2003, Honolulu, Hawaii.
Littlejohn, A., Jung, I. and Broumley, L. (2003) "A comparison of issues in the reuse of resources in schools and colleges". In A. Littlejohn (ed.), Reusing online resources (pp. 212-220). London: Kogan Page.
Philip, R. and Dalziel, J. (2003, Sep) "Implications of COLIS for course development: The need for secondary usage meta-data". Paper presented at the ODLAA Conference, 1-3 October, Canberra.
Wetterling, J. and Collis, B. (2003) "Sharing and reuse of learning resources across a transnational network". In A. Littlejohn (ed.), Reusing online resources (pp. 182-194). London: Kogan Page.