Figure 1. Processes involved at different stages
Kar-Tin Lee , Head, Department of Information & Applied Technology, Hong Kong Institute of Education, 10 Lo Ping Road, Taipo, New Territories, Hong Kong SAR. Email: ktlee@ied.edu.hk
To encourage teachers of secondary two science to link assessment and teaching strategies to promote learning through use of a web-based environment, two online assessment and diagnostic tools were trialed in five Hong Kong schools. The Diagnostic Tutorial Assessment System (DTAS) and the Intelligent Content Assessment Marking (ICAM) System were used in a collaborative project which was initiated by faculty of the Hong Kong Institute of Education. The project involved teachers, students, course designers, curriculum officers and a technology company. Two tools, integrated within an e-learning environment allowed teachers to diagnose student's learning difficulties and to focus on students' preparedness for learning. By identifying more explicitly the type of questions or topics that a student can or cannot solve, teachers could further determine whether a student was ready to learn and what format of instruction was most appropriate. The outcome of the project was a set of customised teaching and learning content materials based on the Hong Kong secondary two science curriculum. Both teachers and students could also access an elaborate online tool that provided accurate profiles of individual students learning progress. These profiles served as a basis for teachers to design remedial or enhancement work for students. The findings to date indicate that both teachers and students recognised the potential of such tools and were happy to explore further.
Online assessment, diagnostic tools, learning management systems, teaching strategies, individual differences, multimedia content design
In Hong Kong secondary schools, teachers are constantly commenting on the fact that there is a great need for online content development that is linked to an efficient assessment mechanism in secondary science. The sections below provide an account of how a curriculum innovation project was conducted in secondary schools to establish a valid and reliable measuring tool for knowledge components and problem-solving skills when learning science. Information is also provided on how the use of engaging and cognitively demanding computer-based curricula could be used to promote academic achievement among many students who are currently not well served by the more traditional methods used currently. The collaboration among faculty from the Hong Kong Institute of Education, Litespeed, a technology company, teachers and students proved to be very challenging. However, the ensuing outcomes far surpassed the difficulties encountered along the way.
DTAS and ICAM are two web-based assessment tools that can be used to assess student learning and provide detailed individual student learning profiles. The use of these tools coupled with solid instructional design input can facilitate new ways of learning in schools. This would allow teachers to use the eventual constructed learning space as the locus of rich and satisfying experiences in collaborative learning in which the learners can actively construct knowledge by learning together and be able to closely map their progress according to an extensive science skills framework. This framework maps each science skill that has to be learnt onto the system so that it can be closely monitored. In addition, the convenience and effectiveness of this new mode of learning will assist the formation of student lifelong learning and independent learning skills.
The three partners in the project were (i) faculty from the Hong Kong Institute of Education; (ii) Litespeed Pty Ltd, a technology company; and, (iii) teachers and students from five secondary schools:
What is DTAS?
The Diagnostic Tutorial Assessment System (DTAS) is an innovative tool that intelligently identifies the specific strengths and weaknesses of individual students throughout each subject's relevant lesson, and automatically prompts component lessons for remediation in weak areas. It is web-based and aims to add value to the teaching and learning process of students and make learning more effective and efficient. It does so by empowering the learner to engage in self-learning through the web on any subject, for example science as is the case for this project. As no system is complete if lessons and test questions are mutually exclusive, DTAS instigates a complete learning cycle by replicating the teaching and learning process while automating the performance feedback mechanism.
DTAS can be used in class to supplement face-to-face teaching. It can also be used by students for individual self-learning through repetition without the presence of the teacher as the system can be accessed from home. On the one hand, it provides opportunities for students to be independent learners. On the other hand, it offers an opportunity for schools to embark on and create an e-learning culture through learning and teaching via the Internet. Through this approach, students are exposed not only to quality content but are also given the chance to experience knowledge acquisition in a different mode; an important process in the overall objective of the current education reform agenda.
What is ICAM?
The Intelligent Content Assessment Marking (ICAM) system is a technology that is able to mark open-ended content like short and paragraph answers for subjects like science and any other content that does not require writing style as a criterion for performance. This system is also able to classify and group various students' answers especially non-standard (creative answers) to facilitate immediate classroom discussions. With this engine, teachers can facilitate online learning and is no longer constrained to conventional inputs such as Multiple-Choice (MC) and Fill-in-the-Blanks (FIB) questions. This enables any the online environment to mark thousands of open-ended answers instantaneously and provide feedback to students as they learn.
ICAM is the technology that enables the marking of open-ended questions for teachers. It uses two different modes to set model answers (i) the Answer Training Mode and (ii) Students Training Mode.
Answer Training Mode
After the open-ended questions are developed and set by teachers within the system, the first step in training the system to mark is the Answer Training Mode. This mode allows the teachers to key in the model answers for each question. The system is then able to provide a solution to this by using a technology called Artificial Neural Cortex Algorithm. With ICAM, the system is able to generate all possible variants keyed in by the teacher.
Subsequently, teachers can train the system to allocate marks by identifying keywords in the students' answers. Teachers can also choose to put words in an exclusion list as well. As Artificial Intelligence can never be perfect as it lacks the human touch, ICAM is therefore only a tool for the teacher to use to save time and effort when marking open-ended answer scripts. After the teacher has set the marking scheme, it may cover up to 80% of the entire marking scheme. However, it is inevitable that there are still other creative ways of answering the question which teachers may not have thought of. Students will always have a different way of responding to questions, although their message is the same. The teacher's input is still vital as without it, the computer cannot eradicate the problem.
Student Training Mode
Other answers that are not identified by the system will require some manual marking on the part of the teachers. Teachers can reset the marking schemes simply by clicking on the selected student answer and dragging it to another mark allocation column and subsequently this answer will be included for future marking.
The system can automatically tabulate all the marks of each student after the new marking scheme has been set and saved. In this manner, the teacher does not have to mark every script. All he/she has to do is to sieve out all the creative answers and allocate the marks that need to be rewarded for these answers.
Each student's answer will be profiled to allow the teachers to monitor the performance of each student. The profiling system for ICAM allows heads of coordinators of subject departments and teachers to gather real-time statistics of all students' performance.
For instance, each developed question in science was structured in such a way that it will have a specific skill and sub-skills attached to it. ICAM is then able to assess the students on their literal, interpretive or applied skills and provide a profile with clear diagnostics of areas of weakness or strength.The project was conducted from July 2003 till December 2004. The two assessment tools, DTAS and ICAM were integral to the e-learning system which is web-based, and offers a range of interactive lessons, tests and quizzes on aspects of the secondary two science syllabus. Most students accessed the system while at school but some groups were also encouraged to access the system from home.
The resources made available through the system were devised by a specially created group of ex-science teachers who wrote the lessons and developed the questions for the students. The entire process is illustrated in Figure 1.
Figure 1. Processes involved at different stages
The project's aims were to establish and implement the development and trial of a suite of teaching and learning content based on the two tools Diagnostic Tutorial Assessment System (DTAS) and the Intelligent Content Assessment Marking (ICAM) System described in the sections above. Both these web-based e-learning and diagnostic systems provided teachers with the facility to identify the strengths and weaknesses of students based on a comprehensive Science Skills Network which is developed in conjunction with officers of the Curriculum Development Institute and science experts.
The main objectives of the project were to:
It was intended that the project would help and motivate weaker students in grasping the concepts of science and stretch the better students beyond the basics in a most efficient manner. The creation of original Secondary School content will also help students better understand the skills through interactive multimedia lessons. DTAS and ICAM integrated with new content will enable students to be able to learn independently while being assessed diagnostically on their exact strengths and weaknesses with the entire spectrum of assessment tools. Teachers would also be able to utilise the proposed software and technology to complement their teaching and facilitate the learning process of students. Students can learn in and out of class from anywhere, anytime.
So what were the teachers and students able to do with the web-based tools?
For the students:
Students used DTAS and ICAM systems online. science content was delivered to allow teachers to teach, assess, diagnose and profile students' differing abilities. Students using DTAS and ICAM could engage in self-learning on the web from home or from school, using their individual accounts. This made learning efficient as the degree of learning for each individual was monitored and assessed through the Diagnostic Profiling System (DPS).
The entire system (see Figure 2) provided students with the following items:
Figure 2. The DTAS System
For the teachers:
Teachers using the system were provided with versatile teaching and learning tools that could be used to introduce lessons or reinforce teaching points. Aside from using the system as a complimentary resource tool to teach concepts, teachers benefited in the following ways:
Figure 3. ICAM: An AI tool for teachers to train the answers
With this additional manual training of the system, individual profiles of students could then be generated more accurately as shown in Figure 4.
Figure 4. A Individual Student Profile
The entire e-learning environment comprised the following elements:
For each topic of study a concept map (Figure 5) was provided online in addition to the more linear progression of topics and task depicted by menus.
Figure 5. Concept Map for the Topic on Electricity
An enormous amount of time was spent on the instructional design aspects. The elements within each learning topic were integrated as depicted in Figure 6 below. The entire learning flow and each component of the topic had to be carefully structured. Each aspect of the learning had to be agreed to by the teachers and the question setters.
Figure 6. Structure of the E-learning environment
A key issue when introducing new technological tools into the classroom is teacher time which is often overlooked and underestimated. Substantial resources need to be made available for staff. The potential move to the use of web-based tools as cognitive tools in classrooms clearly signifies a pervasive transformation of the means by which teaching and learning occur in schools. It was very important for teachers to be competent in cognitive development, particularly as it relates to helping students gain mastery in specific content areas. A critical skill for teachers to develop was the capacity for intellectual empathy. By this we mean that teachers needed to be able to detect the extent and nature of each individual studentˇ¦s knowledge and competence in a subject so that they may help that learner move ahead in developing independent mastery and lifelong learning skills.
For this project, the crucial element for success was to provide as much of the content as possible in the form of readily usable science content developed from the Hong Kong curriculum. This content when integrated into the e-learning system can be used to train and facilitate teachers to reflect on a clear vision of why technology is being used and how it can strengthen teaching and learning activities. At present, although the hardware infrastructure is mainly in place, most Hong Kong schools cannot afford to spend large amounts on software. They tend to focus on applications rather than content software thereby leading to a distressing lack of good and appropriate content software that can be readily integrated into curricula and concurrently be applied to gain monitoring and assessment data that will inform teachersˇ¦ practices. As Bransford, Brown and Cocking (2000) asserted, good educational software and teacher-support tools, developed with a full understanding of principles of learning have not yet become the norm. They cautioned that the use of technology to improve learning is never solely a technical matter. Teachers needed to be more collegial and openly share their experiences and avoid perpetuating their own private practice (Riel & Becker, 2002).
Through collaborative efforts with an industry partner, teachers, principals and science experts, a model for integration can be developed and provided to teachers as a tool to help them move beyond computer operation to computer integration in a meaningful way. With careful attention to instructional design and facilitation of new ways of learning, this constructed learning space can become the locus of rich and satisfying experiences in collaborative learning in which the learners can actively construct knowledge by learning together and be able to closely map their progress according to an extensive science skills framework. The learning environment can provide scaffolding support to augment what learners can do and reason about on their path to understanding through use of scientific visualisation and model-based learning that is made possible with the technology tools (Bransford et al., 2000). In addition, the convenience and effectiveness of this new mode of learning will reinforce the formation of student lifelong learning and independent learning skills.
International and local policy papers (DEETYA, 2000; EMB, 2004; Teacher Training Agency, 1999) examined indicated that the effective application of technology in teaching will result in new organisational forms and a strengthening of the principle of differentiation of teaching. It was therefore the aim of the project to develop a totally new learning approach, which incorporates interactive online learning elements for the teaching of carefully targeted curriculum areas within the secondary schools. Teachers will be given the opportunity to work with industry partners and other Hong Kong schools within a collaborative and supportive framework to define the instructional strategies, curriculum objectives, student needs, and assessment strategies which they will need to implement in order to bring about changes in their daily teaching practice.
The construction of this new kind of learning environment ultimately depends on the beliefs and actions of those responsible for setting up the environment, particularly the underlying pedagogical philosophy of the teacher (New house, et al., 2002). Through organized teacher group meetings with technical specialist from the industry, teachers will gain extensive experience in teamwork and in preparation of materials in a format suitable for technology supported delivery. Teachers will not only be using the technology to teach the standard curriculum (Law et al., 2000) but will have plenty of opportunity to actively reflect on their own actions to rethink their existing classroom practice in order to integrate use of IT and to be more cognizant of individual student learning.
The successful completion of the project would allow other curricula content to be easily adapted for delivery based on the current model which have been field tested by the participating schools, teachers and students. In the wider context, this model aimed to increase the quality and efficiency of several school programs and to establish the model for future delivery of curricular content to new and diverse communities of students. Expertise and experience gained from this project meant that dissemination of its success would be one step further into the acceptance of the fundamental need for the use of technologically-based learning to facilitate better access, improved quality of content and closer monitoring of student learning for formal and informal assessment purposes.
This section is devoted to the challenges, issues and problems encountered during the trials that were conducted in the five schools, and how steps were taken to resolve them.
Challenge #1 - Extent of the work to be done
The entire secondary two science curriculum comprising five topics had to be developed with careful instructional design to turn them into interactive multimedia lessons accessible online for trialing in the schools. The intention was to map the science content on to discrete skills outcomes which were identified by codes recognised by the DTAS and ICAM systems so that individual student profiling can occur. Altogether for the five topics there were over 175 skill identification tags (Table 1) that had to be incorporated within the software.
Table 1. Mapping of Skill Identification Tags
Challenge #2 - Unrealistic expectations of stakeholders
The development of content proved to be a great strain on the resources available. It was the expectation of the science curriculum officers that the content and assessment items had to be designed so that they would be able to enhance the learning of abstract and more advanced topics. In discussions about these issues, there was not always agreement among the content writers, curriculum officers and the project team. Each stakeholder had a different expectation and it did not necessarily match those of the project team. At times, stakeholders did not fully understand the costs involved in designing the so called 'highly interactive content that was capable of extending the learning of students'. Some principals anticipated the learning system to be just as good if not better than the teaching that the teacher would do in a face-to-face situation.
To meet the above challenges the project team had continual meetings with the stakeholders to arrive at an agreement on the design model that would be adopted. It was finally agreed that since teachers did not have the time to write these materials, suitable experts would be found to develop the content and would work with the project team and the technology partner to develop the interactive multimedia lessons to meet the deadline. Topics would be staggered and lessons would be uploaded at different times for schools to do the trial.
Firstly, in each of the schools the timetable was different and the scheduling for teaching the science topics were not the same from one school to the other. A lot of careful organisation and co-ordination had to be carried out so that the trials would go smoothly and data could be collected for analysis.
Secondly, teachers and principals were not willing to trial the online tools with classes that were of lower academic standards as they did not want the data collected to reflect the weaker students' profiles.
To solve the above two issues, trials had to be staggered and topics had to be uploaded at different times for different schools according to their schedule for teaching the relevant topics. As for the lower ability groups, principals and teachers were eventually persuaded to accept the view that the tools were in fact there to help at both ends of the spectrum i.e. the extension of the more academically include students as well as the weaker students. They were also informed that whatever data collected would be used judiciously and they would be consulted before any release of the results to government authorities.
Developing interactive science content is not an easy task. Teachers and curriculum officers had very high expectations of what is to be produced. In view of the limited resources the project team had to work under certain constraints and often measures had to be taken to restrict the amount of content that was requested for a topic or to limit the level of interactivity that teachers expected. An example of activities within an interactive lesson is shown in Figures 7a & 7b below. Students could explore and predict the outcomes based on different scenarios presented.
 |
 |
Figures 7a & 7b. Interactive Lesson on Electricity
The timeline was also a key problem. Teachers were always busy and we could not always get their feedback on time. After the initial struggle the team decided to outsource and found a panel of experienced science teachers who were commissioned to review all the content and provide feedback on schedule. A number of these teachers eventually became our content designers and writers which took a whole lot of stress off the actual teachers participating in the trials.
Extensive efforts were been made to solve the above issues and problems. A strict timeline had to be adhered to and by the fourth month after the commencement of the project, content had been developed and was ready for trialing in the schools. The first topic 'electricity' was ready after three months preparations.
Although the partner schools were always very busy, they were very cooperative and the project team was able to successfully conduct the trials. Over the months, other topics were scheduled, these being 'Space Travel' and 'Common Acids and Alkalis'. In each trial, the students completed a pre-test, studied the multimedia lessons and finally finished the post-test. For the less able students who had not achieved the learning targets, the system directed them to the component lessons and subsequently retested them on the weak areas identified.
The teachers were also very dedicated to the project and invested an enormous amount of time to work with the project team and their students. Teachers gave up a lot of their spare time, usually in the evening or on weekends to participate in focus groups across the schools to train the answers using the AI tool within the ICAM system.
Throughout the project both teachers and students were asked for feedback to allow progressive improvements to be made as each topic was developed and uploaded for trial. Generally they were positive and always provided useful pointers as to how certain aspects might be improved.
In the final few months of the project a questionnaire was administered to all the participating students. The response rates to the questionnaire were representative of the schools in the trial, with the exception of one school where students had difficulty with using English as the medium for instruction. The general feeling was that the system was of value for the students and they provided both ratings of components and some comments about the most useful aspects.
In response to whether the e-learning platform enhanced their interest in learning science, 80% of the students agreed with this statement. 82% of students also indicated that the system helped them to identify their weaknesses accurately. In the area of the component lessons, over 50% of students found them to be helpful and did help them to revise what they had learnt in class and to prepare for tests. Students liked instantaneous feedback that the system provided although they felt that it was not always the case that all answers were marked correctly as they had anticipated.
For the schools where use of English was more of a challenge, they found the questions to be difficult. Students often tried to avoid the open-ended questions which required higher-level problem-solving skills in composing more comprehensive paragraph answers.
From the teachers' perspective, they were very pleased that all content and component lessons were developed for them thus saving them a lot of time. However, because the system was used for the first time in their respective schools they had to assist with the AI training of the answers and this they felt was rather time consuming and demanding. But on the whole, they felt it was constructive and had potential.
Only some findings are reported here, but, overall in response to most questions students judged it positively. Of course, there are still areas of concern that need to be addressed, for example, the way the system matches the open-ended questions. More work is required here. Students were largely happy with the system but there remain challenges which over time the improvement of question asking, the regular use of the system to reduce the problems with language and the novelty of the system. Students liked the multimodality (especially when it reduced the language requirements), the revision possibilities of the system, the visual illustrations and the experimentation, and the games within the learning strategy.
In future with further in-depth analysis of students' answers to open-ended questions especially the noted wrong answers collected by the system, it should be possible to see where they seem to be misinterpreting the science concepts and not just having language difficulties with the questions.
The experience from this project correlates with research findings which indicate that technology supported innovative classroom practices can help to change classroom teaching and learning (Kosma, 2003 ; Lee, 2002; Means , Penual & Padilla, 2001; Schofield & Davidson, 2001). When teachers are stretched and required to collaborate with outside actors they begin to see the potential use of technology well beyond their normal expectations. They begin to use technology differently and accept the fact that their students will also use technology differently. Both these changes in classroom practice demonstrate the possibilities of how curriculum can be delivered and how technology can be used in a more integrated ways to achieve desired learning outcomes.
Teachers also acquired collaborative skills as a result of involvement in this project. We hope that this project can inform and help educators to recognize the need to carefully and painfully plan, design and implement any form of online learning because the change of mindset is a gradual one. Teachers need a lot of hand-holding and scaffolding to facilitate them to adjust and accept cultural changes at the school level. We hope that over time schools using these tools will be able to:The question remains 'how can these changes be transferred to other settings and sustained, and would the government continue to provide a similar level of resources and policy levels to actively encourage these changes?'
Acknowledgement
The project was supported by a grant from the Quality Education Fund, Education and Manpower Bureau, Hong Kong SAR. The author thanks team members, principals, teachers and students who contributed to the success of this project.
Bransford, J., Brown, A., & Cocking, R. (2000). How people learn: Brain, mind, experience, and school. Washington , DC : National Academic Press.
Department of Education, Training and Youth Affairs (DEETYA) (2000). Learning for the knowledge society: An education and training action plan for the information economy. Canberra : DEETYA
Education and Manpower Bureau (EMB) (2004). Information Technology in Education: Way forward . Hong Kong : Hong Kong Special Administrative Region.
Kozma, R. (Ed.) (2003). Technology, innovation, and educational change: A global perspective. Eugene , OR : International Society for Technology in Education.
Law, N., Yuen, H., Ki, W., Li, S., Lee, Y. & Chow, Y. (2000). Changing classrooms: A study of good practices in using ICT in Hong Kong Schools. Hong Kong : Centre for Information Technology in School and Teacher Education, University of Hong Kong .
Lee, K. T. (2002) Effective Teaching in the Information Era: Fostering an ICT-based Integrated Learning Environment in Schools, Asia-Pacific Journal for Teacher Education & Development, 5, 1, June. 21-45.
Means, B., Penuel, W., Padilla, C. (2001). The Connected school: Technology and learning in high school. San Francisco : Jossey-Bass.
Newhouse, C. P., Trinidad , S. & Clarkson, B. (2002). Quality Pedagogy and Effective Learning with Information Communications Technologies ( I CT): a review of the literature , Perth : Western Australia Department of Education.
Riel, M. & Becker, H. (2000). The beliefs, practices and computer use of teacher leaders. Irvine : University of California
Schofield, J. & Davidson, A. (2002). Bringing the Internet to School: Lessons from an urban district . San Francisco : Jossey-Bass.
Teachers Training Agency (1999). Ways Forward with ICT, School of Education , Communication & Language Sciences, formally Department of Education, United Kingdom : University of Newcastle upon Tyne .