John Vargo, Senior Lecturer in Information Systems, University of Canterbury, Christchurch New Zealand. Phone: +643-362-2627, Fax: +643-362-2727, j.vargo@afis.canterbury.ac.nz
Education effectiveness, higher education, education technology, media research, interactive learning, Internet educational technology
There is much excitement about the application of interactive
instructional methods using Internet based technologies. But do
these technologies really provide a pedagogically sound
foundation on which to build more effective (as well as
efficient) educational programmes. This question is the key focus
of the authors stream of research of which this paper is a report
on work-in-progress, including a summary of literature and a
synthesis of aspects of effective education which may be
supported by Internet based technologies. Also included is early
work on constructing a course module to test the hypothesis that
this technology will increase instructional effectiveness.
The author, together with a number of international colleagues collaborated on the development of a Web based, multi-media, multi-national, interactive case from September 1995 to May1996. The results of this collaboration, THE Harlequin Multi-media Case [HREF1] were presented as a plenary session to the World Association for Case Research and Application (WACRA) at their international conference in Warsaw Poland in June 1996. The presentation was enthusiastically received with many questions coming during the closing question-and-answer session. However one question was raised which left the author of this current paper disturbed. That question was "what is the evidence that this method of instructional delivery (referring to Web based technology) is any more effective than traditional classroom teaching?" It was this question that has led to the current work.
With the explosive growth of the Internet and related application of the World Wide Web (WWW) there is much discussion on the application of this technology to electronic commerce and the information society. One of the largest information based segments of our economy is the education sector. Primary, secondary and tertiary education together with corporate training and continuing education represent one of the largest and most knowledge intensive areas for application of this new technology. But does Web based technology really provide a pedogogically sound foundation on which to build more effective (as well as efficient) educational programmes, or is it just another "flavour" of the year (or decade) technology which will in the end have very little impact on long term educational issues? This question is the focus of the current paper and ongoing work of the author.
The application of Internet based technologies to tertiary
education may be viewed in its context by comparing the
traditional Residential University setting to the Distance
Learning University. This comparison is shown in Figure 1, with
the Virtual University shown as a synthesis of these two
traditions. (Hutchison, 1995a). Issues of the effectiveness of
the Virtual University, and particularly the efficacy of Internet
technologies, and their support for effective learning methods is
the focus of the current research.
Figure 1: The Virtual University, (Hutchison 1995-a)
Issues affecting the conjunction of effective pedagogy and the place of technology in effective learning are included in this work-in-progress.
Ives and Jarvenpaa (1996) believe that the Internet "will transform business education, although not necessarily its traditional supplier, the business school". Such a dramatic statement from such august Information Systems academics point up the need for substantive research on this emerging educational medium.
The issues to be addressed in this research spring from three primary sources:
This current work aims at a synthesis of these three streams of research. The longer term research project, of which this paper is a report on work-in-progress, intends to use in-depth case study work to discover the causal nature of educational systems and the effectiveness of Internet based technologies to support higher education.
This work intends to address the following research question:
"Does the Internet provide a pedagogically sound foundation on which to build more effective higher education programmes, and why or why not?"
The remainder of this paper covers an initial survey of the literature, a synthesis of this literature and comparison to the delivery characteristics of Internet technologies compared to effective learning methods, and a description of early efforts to create a course module to test the research hypothesis.
In reviewing the Information Systems, Educational Technology, and general Higher Education Literature, focusing on the conjunction of effective pedagogy and the place of technology in effective learning, the primary themes that come through deal with:
This literature survey covers all but the last area, and includes: definitions of effective learning; models covering the range of effective learning and learning methods; discussion of the media versus methods debate on the cause of effective learning; and finally a discussion of Leidner & Jarvenpaa's important work synthesising this field in the light of the needs of Information Systems Education.
The topic of effective learning is one that is naturally core to the whole process of higher education. Various definitions of effective learning have been posited including:
"…learning in educational institutions should be about changing the ways in which learners understand, or experience, or conceptualise the world around them…learning as a qualitative change in a person's view of reality…" (Ramsden, 1992, p.4)
"Education with inert ideas is not only useless, it is harmful…We enunciate two educational commandments, "Do not teach too many subjects," and again, "What you teach, teach thoroughly…Education is the acquisition of the art of the utilisation of knowledge…"(Whitehead, 1929 pp.2-6)
In discussing deep versus surface learning, Marton (in Ramsden, 1992, pp.43-54) concludes that "It was overwhelmingly clear as well, however, that outcome and process were empirically linked", those who used deep learning processes passed the courses far more frequently than those using surface learning processes."
"The cognitive domain… includes those objectives which deal with the recall or recognition of knowledge and the development of intellectual abilities and skills… the affective domain…includes objectives which describe changes in interest, attitudes and values and the development of appreciations and adequate adjustment." ( Bloom, B.S., ed. 1956, p.7)
"Effective teaching is best estimated in relation to your own goals of teaching…is sometimes equated with successful teaching - that is, the students learn what is intended. While this argument has some appeal, it is not the whole of the matter. Effective teaching is concerned not only with success but also with appropriate values." (Brown & Atkins, 1988, p.4-5)
"…true education for all is a major part of the answer. But we're not talking here about academic education. We're talking about personal growth (which includes self-esteem), life-skills and learning-to-learn. Once you know how to learn, you can accelerate learning." (Dryden & Vos, 1993, pp.19-21)
"one six year old boy in the so-called LEGO/Logo class built a clump of blocks and placed a motor on top… wrote a more sophisticated program...result was a moving pile of blocks that followed a black squiggly line…child became a hero…This small moment of glory gave him something very important: the joy of learning. We may be a society with far fewer learning-disabled children and far more teaching-disabled environments than currently perceived. The computer changes this by making us more able to reach children with different learning and cognitive styles." (Negroponte, 1995, pp.197-8)
So effective learning is not just about the efficient transfer of certain quantities of knowledge, but it is also about developing skills and attitudes for life-long-learning, it is about experiencing the joy of learning, it is about both factual knowledge and developing good judgement.
The span of effective learning may be viewed as seen in three
models: Bloom's and Marton's covered in this section, and Leidner
& Jarvenpaa's covered under the heading "Synthesis of
technology and effective learning in higher education" later
in this paper. The first is a taxonomy of educational objectives
in the cognitive domain seen in Figure 2 based on Bloom's work.
This model shows a hierarchy moving from the basic learning of
information and facts (knowledge) through to the ability to
evaluated and make judgements based on prior levels of
increasingly deep understanding and insight (evaluation)
describing the depth and applicability of knowledge (Bloom, 1954,
p.18, 201-207):
| Knowledge: learning and recall of facts and information |
| Comprehension: The lowest level of understanding with a basic ability to use the facts and information. |
| Application: the use of abstractions to apply knowledge to other areas or fields and predict probable outcomes of introduced changes. |
| Analysis: decomposition into constituent elements so that the hierarchy of ideas is made clear and the relations betweeen ideas is made explicit. |
| Synthesis: The putting together of elements to forma whole, arranging and combining so as to create a pattern or structure not evident before |
| Evaluation: Judgements about the value of material and methods for given purposes. Qualitative and quantitative judgements about the extent to which material and methods satisfy criteria. |
Figure 2: Bloom's taxonomy of educational objectives in the cognitive domain
The second model based on Marton's work (Figure 3) is concerned with the approach to learning. Is the learning in context, retaining the structure of knowledge, or is it focusing on the facts and parts separate from the whole? What is learned, is it a deep understanding of the purpose and intention of the learning situation apprehended, or is attention simply given to the surface facts and symbols of the knowledge to be learned.
Figure 3: A model of learning approaches
These two models cover the spectrum of effective learning. Bloom's taxonomy is primarily concerned with the content aspects of effective learning, while Marton's model is concerned with the process aspects of effective learning. The next section introduces the "media versus methods" debate over how effective learning is caused.
An ongoing theme in the Educational Technology literature has been the debate over what impact the delivery medium for an educational programme has on the learning outcomes for that programme, versus what impact the educational methods have on those learning outcomes. This debate is an important one in shedding light on the programme of research covered by this paper. Internet technologies are a medium for delivering an educational programme. This section of the literature survey looks at this debate.
The debate in the Educational Technology literature involves two opposing viewpoints. On the one hand are the researchers who hold that the media used to deliver an educational programme can only affect the efficiency of delivery, not the effectiveness of the outcomes. On the other side of the debate are those who support the view that the media such as video, text, computers, audio tapes and broadcast TV can and do impact the effectiveness of the delivery outcomes and should be pursued vigorously for the benefits they deliver.
The primary concern of the "media as efficiency only" proponents is that valuable research resources are being expended on media research that is insufficiently rigorous to differentiate between the learning outcomes caused by the instructional methods used (interactivity, learner centeredness, group learning, etc.) and the media used for the delivery. Because much of the past media research has not controlled sufficiently for the instructional methods variable, therefore the reliability of the results are highly suspect. Clarke, a primary supporter of this viewpoint, holds that because there is no way to separate the impact of instructional methods used from the influence of the media used, all such research is confounded. (Clarke, 1994). Proponents of this view assert that the media can have no affect on the effectiveness of the outcomes, no more than a delivery van will improve the nutritional value of the food it delivers. Since they believe in the "media as efficiency only" viewpoint, it then follows that any resources used to research the effectiveness of media represents wasted resources that could be far better spent on instructional methods research instead.
On the other side of the debate, the "media effectiveness" proponents have a concern. They foresee in the near future the convergence of communication technologies and digital computing power. This convergence offers great hope for educational effectiveness (and efficiency). There is the serious danger however that if the "media as efficiency only" viewpoint wins the day, this powerful new media will be relegated to interactive soap operas and home shopping. Proponents of this viewpoint hold that the delivery vehicle can and does impact the effectiveness of the learning outcomes. It is much like the refrigeration truck delivering frozen foods or moving other perishable foods. (Reiser, 1994) Without the specialised delivery vehicle, the nutritional value of the food will become seriously degraded. Kozma, a major supporter of the "media effectiveness" view, holds that theories of media effectiveness must "reflect both the capabilities of media and the complexities of the social situations within which they are used." (Kozma, 1994, pg.14) He goes on to assert that media theories must identify the causal mechanisms through which cognitive and social processes work in order to more securely establish the connections between media and learning outcomes.
Jonessen identifies the cause of the conflict between the "media as efficiency only" and the "media effectiveness" supporters as the traditional scientific theories used to view the debate. (Jonessen et al., 1994) The traditional scientific theories subdivide the situation into components, attempt to control some variables so as to measure the results of the other variable(s). This approach ignores the large body of "systems" literature which demonstrates that a system is more than the sum of its parts. Learning systems are not about media or methods, but about learners learning. Those learners will use various methods and various media in complex social settings. It is all of these components working together synergistically that in the end produce learning outcomes.
With valuable insight, Shrock in (Shrock, 1994) recognises that in this debate much is available to be learned from the opposition. Clarke in raises valid questions about research validity and Kozma identifies excellent research methods. The debate is a timely one given the convergence of technologies that are taking place, and a deeper understanding of the potential synergisms is badly needed.
The work by Leidner and Jarvenpaa (1995) provides a more comprehensive taxonomy, that largely subsumes much of the earlier learning effectiveness literature as well as adding the technology dimension. Aspects covered in Figure 4 under the Process Dimensions include learner control over pace and content and objectivist (also called instructionist) versus constructionist approaches to learning. These two models, called Objective and Constructive in Figure 4 represent the most common poles of the larger framework in learning theory, with the Objective model representing the left side of Figure 4 and Constructive model representing the right side of Figure 4. Leidner includes a number of other learning models as well including collaborativism, cognitive information processing and socioculturism. These later three models are in many ways children of the constructionist model.
The Objective model, is based on stimulus-response theory and the concept that the objective world can be communicated accurately as factual and procedural knowledge disseminated by instructors expert in the field of study. The objective model is typically implemented using the lecture method of teaching and tends to focus on the lower order cognitive objectives from Bloom's taxonomy.
The Constructive model on the other hand assumes that learning consists of individuals constructing knowledge based on the learner's experience and personal viewpoints. This construction of knowledge involves the higher levels of Bloom's taxonomy of cognitive objectives and is typically implemented using "learner centered" approaches involving discovery, control of pace and content, investigation, case analysis and other similar learning experiences.
Although Leidner's work, as summarised in Figure 4 presents a polarity of these two models, the work of Ramsden (1992), Hammersley (1986) and Brown and Atkins (1988) support the view that both models may work depending on the instructional goals and a combination of the models may prove suitable in other learning situations. However the consensus amongst most researchers in the higher education field appears to be that "learner centered" approaches arising from a moderate constructivist viewpoint are proving to produce the best results for both factual learning and higher order thinking (Ramsden, 1992).
Leidner's work also summarises the use of Information Technology (IT) in the classroom ranging from:
Figure 4: Taxonomy of the impact of IT on learning
(from Leidner and Jarvenpaa 1995)
Ives and Jarvenpaa (1996, p.33) assert that the Internet "will transform business education, although not necessarily its traditional supplier, the business school". They go on to suggest that this transformation will occur for both effectiveness and efficiency reasons in the marketplace and the slow response of Universities to this transformation may well endanger their control of business education. But does the Internet provide a good delivery vehicle for effective education, and if so, why?
Bloom (1956) presents a taxonomy of educational objectives ranging from the learning of surface facts to deeper objectives such as synthesis and evaluation. Ramsden (1992) and others suggest that effective education is based on deep learning which covers the full range of cognitive objectives, and this type of learning must be learner centred, active and in context. Are the Internet based technologies capable of providing a platform for such learner centred education, and what are its strengths and weaknesses in filling this role?
Finally Leidner and Jarvenpaa (1995) place the Constructivist based, learner centred educational systems at the "deep" learning end of a Taxonomy of impact of IT on learning. They further suggest that virtual learning spaces are the technology element to support these deep learning activities in a Transforming vision of business education. How well suited are the Internet technologies to providing "virtual learning spaces"? They in particular state:
"Perhaps the most widely known hypermedia tool in academic circles is the World Wide Web,…although we are unaware of research that examines the potential of the WWW in the context of classroom analysis…research examining uses of the Internet and WWW is much needed." (pg. 277)
Further, the many studies done on the effectiveness of technology in support of quality learning, the vast majority have been of the quantitative experimental nature or survey based (Morrison, 1994, Kozma, 1994). Very few have been done in depth and in context so as to explain the causal links in real life situations (Yin, 1994, Jonassen et al, 1994). The complex nature of real learning environments demands a more holistic approach to research on the interaction of technology, learning methods, students perceptions and their environment. A good example of this need is the question over the application of the hypertext paradigm to a structured argument or instructional concept. If students are allowed to hyperlink all over the place, will this not destroy to ability to follow a reasoned train of thought? What is the impact on this conjunction of technology and learning?
The next section will describe the proposed research questions to be addressed and the methods to be used in carrying out the research.
The rise of the Internet to prominence on the technology horizon during the 1990s has raised the question of its potential as an educational medium, and the influence that it might have on the future of higher education. There is currently much work being done to experiment with the use of this technology in delivering educational programmes. (Aoun 1996, Bearman 1996, Bytheway 1996, Eden et al 1996, Galegher and Kraut 1994) However there is very little work being done to date on the effectiveness of this media. This is one of the factors giving rise to the current research of this paper.
To what extent do Internet based technologies support the effective learning methods summarised in the earlier section of this paper? The following table is a first draft look at a comparison of the demands of effective learning methods and the potential of Web based technologies to support this pedagogy. The author would welcome additional views and insights on this material. Figure 5 shows a matrix associating effective learning methods with various Web based technologies. An X in a cell indicates a hypothesis that the technology does significantly support the effective learning method. It is proposed that this matrix be tested using both experimental and more in-depth research methods to validate its contents.
Web based technologies
| Effective learning methods | Hyper text | Inter active & multiuser games | Links to current infor mation | Database & CGI, Java, ActiveX support | Email & Chat | Audio & video, confer, encing | Bulletin boards and discussion groups |
| Delivery of accurate facts | X | ||||||
| Interactive learning | X | X | X | X | X | X | X |
| Deep learning | X | X | X | X | |||
| Ability to apply knowledge | X | X | X | X | |||
| Development of judgement | X | X | X | ||||
| Personal responsibility | X | X | |||||
| Life long learning attitudes | X | X | X | X | X | X | |
| Joy of learning | X | X | |||||
| Group learning | X | X | X | X | X | ||
| Two way communication, feedback | X | X | X | X | X | ||
| Learning in context | X | X | X | ||||
| Problem & case based learning | X | X | X | X | X | X | X |
| Control over content & pace | X | X | X | X |
Figure 5: Matrix associating effective teaching methods with Web based technologies
Based on the proposed relationships shown in Figure 5, the author has embarked on a stream of research aimed at measuring the causal effects in higher education and the extent to which they are supported and enhanced by Web based technologies. One of the early exploratory studies in this research is to create a Web based course module and use it with a group of students to discover their reactions to this delivery vehicle and measure its effectiveness in an instructional programme. This module is still under construction and the next section describes the module design features, and how these design features are planned to discover issues of higher education effectiveness.
The course module chosen for development was the double-entry accounting concept. This was chosen because it presented a historical problem with the Introductory Accounting course. Some students arrived at University having done a couple years of accounting in high school, others arrived with no experience. This significant disparity between students meant that during the early part of the course the double-entry material went to fast for the inexperienced and too slow for the experienced, thus not really meeting the needs of either group. Workshops and special sessions helped with this, but it was an ongoing problem. It was also felt by the course lecturers and supervisor that students who did pass the course and go on to further accounting studies often did not have a deep enough grasp of the concepts and impacts on the financial statements.
In the light of this the author has undertaken, with a research assistant, the task of devising an instructional module (which is still under construction), a double-entry accounting module [HREF 2] incorporating the following learning method characteristics:
These learning method characteristics are being incorporated
into the Web based module using various Web based technologies
for delivery initially in an on-campus (Intranet) setting. The
association matrix in Figure 6 shows the support given by each
technology to the given learning methods:
| Effective learning methods | Hyper text | Interac tive & multi user games | Helper apps: spread sheet | Database & CGI, Java, ActiveX support | Email & Chat | Audio clips | Bulletin boards and discussion groups |
| Interactive learning | X | X | X | X | X | X | |
| Deep learning | X | X | X | ||||
| Ability to apply knowledge | X | X | |||||
| Joy of learning | X | X | X | ||||
| Group learning | X | X | X | ||||
| Two way communication, feedback | X | X | X | X | X | ||
| Problem & case based learning | X | X | X | ||||
| Learning in context | X | X | |||||
| Control over content & pace | X | X | X |
A further description of the implementation of the above design is given in the sub-sections below, including components or concepts that incorporate both the effective learning methods and the technology delivery in order to demonstrate some of the synergy of these elements. Although the following is not an exhaustive list of all the module features, it is indicative of the direction of development:
Not all of the above elements are fully functional in the course module as it is still under construction.
This paper has described work-in-progress on a research project involved with effective learning in higher education using Web based delivery technologies. There is currently a void in the literature in this area and this research is aimed at redressing this void. The literature says much about what effective learning is, and how various media or technology can be used. Experimentation and delivery of coursework using Web based tools is exploding, and the promise of radical improvements in both efficiency and effectiveness are widely hyped. But the question remains: will this technology make more of a difference than past technologies, or will it prove to be another "flavour" of the year (or decade) technology which will in the end have very little impact on long term educational issues?
It is intended that future aspects of this research will measure the effectiveness of Web based technologies in supporting effective higher education programmes. The proposed methodology will include a combination of quasi-experiments and in-depth case studies, with the aim of determining the causal aspects of the learning environment, both learning methods and delivery technology, that best support effective learning.
The author would welcome the opportunity to discuss this material with all interested parties, and would be particularly interested in assisting in measuring the effectiveness of Web delivered coursework already in place at your institution.
B. Aoun, (1996) "Agent technology in electronic commerce and information retrieval on the Internet", Proceedings of the Second Australian WorldWideWeb Conference, AusWeb, 1996, p.139-149
M. Bearman, et al (1996) "Lessons learned in publishing an electronic journal", Proceedings of the Second Australian WorldWideWeb Conference, AusWeb, 1996, p253-258
B.S. Bloom, ed. (1956) Taxonomy of educational objectives, Longmans, Green and Co. New York, 1956, pp.207
G. Brown, and M. Atkins (1988) "Effective teaching in higher education", Methuen, 1988, London.
A. Bytheway, (1996) "Grasping the Internettle", Management Focus, April 1996 (from the Cranfield School of Management)
R.E. Clarke, (1994) "Media will never influence learning" Educational Technology, Research and Development, Vol. 42, No. 2, 1994, pp.21-29.
G. Dryden, and J. Vos, (1993) The Learning Revolution, Profile books, Auckland, 1993, pp.512
H. Eden, M. Eisenberg, G. Fischer, and R. Repenning, (1996) "Making learning a part of life", Communications of the ACM, Vol. 39, No. 4, April 1996, p. 40-42.
J. Galegher, and R.E. Kraut, (1994) "Computer mediated communication for intellectual teamwork: an experiment in group writing", Information Systems Research, V.5, no. 2, June 1994, p.110-139
M. Hammersley, ed., (1986) Case studies in Classroom research, Open University Press, 1986.
C. Hutchison, (1995-a) "The 'ICP OnLine': jeux sams frontieres on the cyber campus", Journal of Computer Mediated Communication, Vol. 1, No. 1. 1995-a
B. Ives, and S.L. Jarvenpaa, (1996) "Will the Internet Revolutionize business education and research", Sloan Management Review, Spring 1996, p.33-41
D.H. Jonassen, J.O. Campbelll, and M.E. Davidson (1994) "Learning with media: restructuring the debate" Educational Technology, Research and Development, Vol. 42, No. 2, 1994, pp. 31-39.
R.B. Kozma, (1994) "Will media influence learning? Reframing the debate" Educational Technology, Research and Development, Vol. 42, No. 1, 1994, pp. 7-19.
D.E. Leidner, and S.L. Jarvenpaa, (1995) "The use of information technology to enhance management school education: a Theoretical view", MIS Quarterly, September 1995, p.266-291.
G.R. Morrison, (1994) "The media effects question: 'unresolvable' or asking the right question" Educational Technology, Research and Development, Vol. 42, No. 2, 1994, pp. 41-44.
N. Negroponte, (1995) Being Digital, Hodder & Stoughton, Rydalmere NSW, 1995
P. Ramsden, (1992) Learning To Teach In Higher Education, Routledge Publishing,, London, pp269, 1992
S.A. Shrock, (1994) "The media influence debate: read the fine print, but don't lose sight of the big picture" Educational Technology, Research and Development, Vol. 42, No. 2, 1994, pp. 49-53
A.N. Whitehead, (1929) The aims of education and other essays, Williams and Northgate Ltd., London, 1929, pp. 247
R.K. Yin, (1994) Case study research, design and methods, Sage
Publishing, pp. 171, 1994
HREF 1
http://www.afis.canterbury.ac.nz/mediacase/wwwcase.htm - Harlequin Multi-media Case
HREF 2
http://www.afis.canterbury.ac.nz/afiswww/home.htm - In progress WWW Accounting Course Module
John Vargo ©, 1997. The author assigns to Southern Cross University and other educational and non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The author also grants a non-exclusive licence to Southern Cross University to publish this document in full on the World Wide Web and on CD-ROM and in printed form with the conference papers, and for the document to be published on mirrors on the World Wide Web. Any other usage is prohibited without the express permission of the author.