Joan Robson, Course Coordinator, Faculty of Education, Australian Catholic University, PO Box 256, Dickson, ACT 2602. Email email@example.com
With the burgeoning use of the Internet to facilitate on-line learning, it is timely to examine past and current theory and practice in evaluation of on-line and distance education (Calvert, et al., 1991; Robson, 1997). In doing so, the role of the significant number of students who discontinue emerges as a factor (Peters, 1992). Discontinuation has long been noted in face-to-face education and is even more noticeable in distance education - between 31% and 76% in some cases (Malley, 1976, 171: Castles, 1992, 39: Commonwealth of Australia, 2001). The factors identified as related to this (Long, 1994) give little reason to suspect that on-line learning is different.
In the past there has been much theoretical and practical work done on the classification of such students and the reasons for their discontinuation (Calvert, 1995; Long, 1994). Robson's study (1996) of students learning mathematics through teleconferencing, raised the question of the effect of those students who left on the results which are used to judge the effectiveness of the course, and the effect of discontinuing students on course evaluations in general, highlighting the need to extend the current evaluation theories to embrace this aspect. Discontinuation is an action which results from evaluation of the situation by a central participant, but most evaluation models for distance education do not specifically involve these students. This may be because of difficulties in data collection or because of a philosophical stance which discounts the relevance of their perceptions.
In considering students who discontinue, Durkheim's suicide theory (1952) and Spady's model of influences on discontinuation (1971), determine the factors influencing such student evaluation and subsequent action. Durkheim's strong emphasis on the importance of integration of the individual with society confirms the need to examine the educational environment provided by an institution, the types of students in the institution and the integration between these, together with the integration between the educational environment and the course goals. The timing of the student input also has relevance here so that all students can make a contribution to the discourse (Owen & Rogers, 1999, 96; Cheung, 1998, 26).
The problem then is to use a framework for evaluation of a contemporary on-line course which acknowledges the past and present practices and theories, but which addresses the current issue of discontinuing students. Immanent critique reveals that discontinuing students are not considered in most evaluations purporting to include and represent the student body, while ideology critique reveals the distortions caused by ignoring these students.
Theorists, and the schools of thought they represent, have much to offer to the problem of evaluating a distance education course and considering these discontinuing students. Their impact must be considered in relation to the integration of students in a course, with the length and timing included as factors. An evaluation needs to consider these issues in relation to the dominant paradigms to be relevant in the twenty-first century on-line education scenario.
In developing a framework for resource-based learning, it must articulate a shared point of view to support and inform planning and action (NBEET, 1997), 'to provide the necessary structure to cope with the inherent complexity of human behavior' (Garrison, 1988, 209). A theoretical framework cannot tell practitioners what to do in a particular circumstance, but can be modified to suit the circumstances with the criteria for deciding relevance being a matter of deliberation, interpretation and interaction between the parties (Posavac, 1997, 125). House argues that the evaluator should act like a human being, 'as a critical intellect, less on technique' and 'the evaluator must get close to the course, experience it and convey that experience in its totality' (1973, 23). This is leaning towards the Interpretive or Social Critical paradigms with just a touch of Post-Structuralism. Generally educators have moved from the rigidity and sometimes inappropriateness of a Scientific methodology towards the Interpretive approach to evaluation, then to a Social Critical approach but all of these approaches are used still. Evidence of evaluators' philosophies are plentiful in the literature. For example, Rossi, Freeman and Wright's definition of course evaluation (Rossi, Freeman & Wright, 1979, 2) reveals a non-Scientific approach with the charter to improve social conditions, so leaning towards the Social Critical Theory. An evaluation framework must accommodate this flexibility.
The framework for evaluation of an on-line or distance education project developed by Robson (1999) takes into consideration the Spady (1971) and Tinto models (1975) of factors influencing discontinuation and is therefore pertinent in this case. Considering the importance of this group in an evaluation, the framework uses and extends their models to cater for distance education. Rather than being prescriptive, the framework highlights areas that should be explored in order to better understand a situation. It shows evaluation generally, so that it has the flexibility to be individualised to a specific situation, yet illustrate the permanent inter-relationships between the various factors. It is in harmony with the dominant evaluation methodologies and styles currently adopted in on-line and distance education. It is designed to facilitate input from discontinuing students and to be undertaken in a time frame which is realistic when rapidly changing technologies form part of the program being investigated.
Consideration of the works of Durkheim (1952) on suicide, Habermas (1971) on evaluation paradigm choice, Foucault (1980) and Lyotard (1994) on individuality of contexts, and to a lesser extent Spady (1971) on discontinuation, then leads to the sub-sections of a framework for evaluation. In analysing a situation using Post-Structuralism, the course is dissolved into parts in order to understand it. Thus, a framework gives an indication of important sub-sections that could aid de-construction, so the extent of individualisation of an evaluation in the framework becomes of interest in a Post-Structuralist approach, even though this approach would normally reject any prescribed structure.
It is argued that the students who discontinue are such a vital element in any course that the areas of concern in their case, are causes for concern to the course as a whole. Therefore the Scientific Durkheim and Spady theories become an important part of an evaluation in that the degree of integration of the student goals and the educational environment of an institution is an indication of the nature of a course. So, too, is the harmony between an institution's goals for a course and the environment in which it operates. Technology and the human-computer interface, which are essential to contemporary distance education (Inglis, 1999), are factors in both social and academic integration. These aspects of Durkheim's and Spady's theories provide a theoretical platform for the inclusion of both continuing and discontinuing students in determining the extent of the integration. Consequently, an evaluation incorporating consideration of the integration of student and course goals with the educational environment, better reveals the situation.
Although a framework, as an heuristic device, cannot capture all the complexities of practice, it can act as an organiser of options. The theoretical framework for evaluation of on-line education within higher education used here, incorporates many of the aspects of evaluations previously developed and implemented. It encompasses fundamental theory on evaluation, individualisation and change. But, rather than being prescriptive, it highlights areas to be considered or explored in order to better understand and reveal a situation. It shows evaluation generally, so that it will have the flexibility to be individualised to a specific situation, yet illustrate the permanent interrelationships between the various components of educational evaluation where, 'the field of distance education encourages multiple perspectives, different epistemologies and thus different ways of knowing' (Jakupec & Kirkpatrick, 1997, 199). It is in harmony with the dominant evaluation methodologies currently used, and incorporates most of the attributes of the models and styles currently employed. Importantly, it caters for consideration of the discontinuing students.
Figure 1: A framework for evaluation of a distance education unit or project.
Figure 1 illustrates how the evaluation rationale will determine the methodology used in an evaluation and that this methodology, together with the course or project goals, the educational environment, the student goals and the matches between these will affect the judgement or understandings resulting from the evaluation. The resultant action will inform the course goals, educational environment and student goals making it an on-going, cyclical phenomenon.
The question of the discontinuing students affects how much is to be evaluated. If their input is seen as an essential part of the evaluation, then as much information as possible must be gleaned from them before they 'disappear'. This is especially true when it can be seen from the literature that predicting whether a student persists or not, is a complex task. This implies that the evaluation must be thought through before students enrol, so that all students have the opportunity for appropriate input before discontinuation starts. Obviously, there will still be more opportunity for input from the students who continue. This means that the timing of the evaluation should include the time interval from the enrolment of students to the withdrawal of the first student, or even the completion of the first assessment task. It is important to decide on the evaluation methods early on in the evaluation, so that the evaluators can maximise the use of their time during that interval. Therefore, in deciding how much to evaluate, the evaluator sets the boundaries for the exercise. Habermas (1975, 57), Herman and Mandell (1999, 31) hold that everyone with an interest to participate in a communicative discourse should do so, and consideration is normally given to this when developing an evaluation rationale.
There is value in an evaluation in examining the rationale for developing and implementing the course by the institution or others in the first place, and its guiding educational philosophy. Whether a course aims for massification, individualisation, or globalisation is of interest here. So too, are any policies on incorporating open learning or flexible learning into the course. The target audience should be identified and the acceptable discontinuation rate determined at this point. Even if the course goals cannot be determined precisely (Posavac, 1997, 131), there is value in considering them, because a course does not exist in a vacuum, but is a response to a variety of influences, including that of need, in the given context. Evaluation may already be part of the course plan and an instrument in achieving the goals (Owen & Rogers, 1999, 29). Generally, evaluators would assume that improving student learning would be part of the course goals (Lewis, 1989, 9; Morgan, 1993, 86), and these espoused aims can be compared with reality. For instance, the evaluators can investigate the course rationale to determine whether, as part of a university, its role is to transform students to be instruments of social change, an extension of knowledge, or tools of economic growth (Paul, 1999, 31) because curriculum assumptions are usually based on some educational philosophy, sociology and psychology (Jakupec, 1998, 111). For example, '[w]hen students become 'learners' changes follow in both the control and context of curricula' (Posavac & Carey, 1997, 26).
The learning theory, on which a course is based, and its enactment, will influence these course goals. As well, the nature of any technology used can be relevant. Also, when the course goals are considered in the light of the institutional goals, the relationship between the institution and the outside world becomes an issue. However, as technology in the workplace where the students aim to go is changing, the skills they need to learn at university are changing too, so the new 'entrepreneurial' form of higher education impacts on the course goals and community context (Garrick, 2000, 244). Failure of an institution to adjust to this, is like Durkheim's anomic suicide.
Most institutions are set physically within a community, which may or may not include the on-line education students. The hopes, goals and expectations of the community will influence the academic system of the institution and the educational environment that students will encounter. The community itself lies in contemporary society with its changing interests in globalisation, justice and technology, but cost is a dominant factor too, in distance education today. Power is closely aligned with this and so resources available in the development and presentation of a course exposes underlying power structures (Foucault, 1980, 89) especially where technology is involved. Financial links between the community and the institution in the form of government support or control, also influences the strength of the community commitment, so cost becomes an important factor and a realistic control on the implementation of theory in a community. It addresses human, technological and institutional constraints and possibilities. Therefore, the community context within which the evaluation takes place must be acknowledged if the findings are to be valid and generalisable (Jakupec, 1998).
Institutions within contemporary higher education have profiles that emphasise the changing academic, administrative and technological cultures. This background affects the relationship between the learning theories espoused by an institution, and the teaching strategies used to educate the student, within the academic system. The current merging of distance education, open learning and flexible learning (Tait & Mills, 1999) and the accent on life-long learning (Candy, 1991) can also influence the model used to construct and guide the on-line education in an institution. Therefore, content, materials, support and design, become important elements of the academic system of an institution, influencing the administrative structures and presentation standards (Cheung, 1998, 28). Gibson and Hatherell (1997, 133) talk of this as the 'tone' of a university. But, technology changes rapidly. Therefore, as on-line education is strongly dependent on the technologies used, and as student satisfaction is also dependent on the technologies, then the maximum time for an evaluation may be limited by the lifetime of the particular technologies being used. Consideration of the large amount of work on human-computer interaction (HCI) and the emerging work on teaching and learning with technology, is ensured by incorporating these issues into the examination of the academic system and educational environment supplied by the institution.
These institutional policies and procedures, together with student and teacher attitudes towards on-line courses, play a part in supporting the learning environment. Therefore, policies on aspects, such as interaction, assessment procedures and individualisation of teaching have a bearing on the learning process. The strength of negotiation is another facet of the system. This can be seen where students and the teacher explain and justify their ideas to others, reflecting critically on other's ideas, together with some flexibility of time, place and pace and choice of curriculum. Identification of the power structures within the academic system is of interest to evaluators here.
The generalised framework in Figure 1 aids in the consideration of HCI and technology which incorporates the psychosocial characteristics of learning environments together with the effect of the computer interface on the students' learning, influences the academic system and consequently the educational environment. Because technology changes rapidly, hasty turnover can affect student satisfaction. This will in turn influence the student's goals and how the student reacts to the course and the technology. Therefore, there can be value in an evaluator identifying both changes in technology and the forces behind these changes to reveal the basis of power in the situation. In summary the academic system or 'learning milieu' (Morgan, 1993, 95) influences the overall educational environment of a course. Inclusion of the academic system in the framework ensures consideration of this factor during an evaluation.
Durkheim demonstrated the influences of the society within which the students and course operate (Owen & Rogers, 1999, 23). The social system (social context, roles and relationships, human performance and well-being, equipment, and the physical or virtual social environment), which an institution facilitates, influences the students' persistence as well as their learning, and in turn generally effects the resultant educational environment (Spady, 1970; Freitag & Sullivan, 1995; Thorpe, 1995, 175). As shown by the investigations of discontinuing students (Robson, 1997), informal and formal peer group interactions (Lyall & McNamara, 2000, 118), informal and formal staff-student interactions and external interactions all have a place in developing the social system of a course.
Higher education social systems are shaped by the changing times (Rossi, et al., 1999, 15) and there will be social systems which are independent of the institution which have a bearing on the students. These various social areas can differ in nature and goals from that being evaluated (Rossi, et al., 1999, 5) but generally the institutional social system will have the dominant effect on the resultant educational environment, but this is not necessarily always so.
Students, especially younger students, need socialisation (Wilson, 1997, 281) and technology may or may not be used to facilitate this social interaction between students and between the teacher and students (Nunan, 1990, 3). Papert noted that the 'context for human development is always a culture'. The social system a vital part of an educational environment and consequently of its evaluation.
The academic and social systems together form the educational environment (Chambers, 1995, 349) or 'organisational milieu' set within a community context in which the teaching, learning and interaction take place (Morgan, 1993, 95). The teaching strategies used to educate the student via various technologies, arise out of the academic system to form the educational environment. Results from the examination of this influences any judgements and understanding resulting from an evaluation.
Following Durkheim, the degree of integration between the course goals, the academic system, the social system, community context and the educational environment affects the institutional and the community commitment to an on-line course. Habermas argues that social integration is possible through understanding life-worlds but that as system integration does not involve life-worlds, integration becomes impossible (Stevens, 1995, 38). Durkheim however, has argued strongly that it is not only possible, but necessary and in following Durkheim's arguments, the importance of integration is shown by the serrated edges of the areas in Figure 1. The resultant commitment to a course influences the judgement and understanding of the course during an evaluation. An arrow connecting the serrated edges to the judgement and understanding illustrates this.
Durkheim and Spady have demonstrated that a student's background will affect that student's attitudes, goals or needs. A basic pedagogic principle is to find out about the students' social and educational backgrounds, that is, socio-cultural information (Chambers, 1995, 347) in order to move forward. Prior mental contents and external inputs such as teaching, form this background (Young & Marks-Maran, 1999, 178). So, investigation of previous experiences informs an evaluation (Powell, et al., 1999, 98). Assessment procedures also play a part, as a student's perceptions of assessment are influenced by that student's previous experiences. In fact, most evaluation approaches see students influenced by their societal context. So, the attitudes towards study and technology are formed in the student's background and in turn influence how the student reacts to the course and the technology and how the student forms goals. An arrow in Figure 1 connecting the student background and goals illustrates this. So students' experiences, fears, prejudices and perceptions of using technologies in the course, form an insightful part of the evaluation of attitudes towards study and technology (Chambers, 1995, 349). For instance, technophobia detection is important, both in the conduct of the course and in the evaluation. It is useful to ascertain such student background variables early as they become important data if a student discontinues. Because this act flags student evaluation of the course, factors leading to that decision and subsequent action, have a central place in the analysis.
Evans (1994, 20; 1999, 222) notes the importance of researching the diversity of student backgrounds in order to accommodate them in appropriate educational courses as knowledge of a student's intellectual background is the usual starting point for helping students learn. Certain variables have been shown to affect the discontinuation rate and as part of the participant cohort, then a study of these variables would normally be of interest in an evaluation. However, all the variables influencing students to discontinue cannot be addressed when considering discontinuation rates in the evaluation process, but an appropriate selection can illuminate the situation.
Generally students want to be informed about the field of study they have chosen and to reason about the area, as part of the course they are undertaking. Nevertheless, there has been a shift in values and self-interest of both students and community members, over the past few decades, influencing students, courses and their evaluations (Rossi et al., 1999, 19). However, improving their learning would normally be among the student goals (Morgan, 1993, 86). A learner-centred approach sees determination of the student goals and needs as pivotal. But, if a course or evaluation is individualised, then there will be multiple needs and goals to be answered. The task is simpler when a common set of student goals is determined by the institution, the students or the evaluator. Technology may well be the tool which enables these goals to be achieved, but if an educational environment is not tailored to these individual goals, then any mismatch with the culture, say through globalisation, will be detected when compared to the student goals. For example, Gooler (1979, 191) uses retention of students as an indicator of how well personal goals are met. Therefore, when a student discontinues, having pre-determined the student's goals sheds light on the degree of success of the course in meeting those needs.
This is normative congruence identified by Spady (1970), academic integration as seen by Tinto (1975) or criteria of appropriateness of Chambers (1995, 348). Student persistence is an indication of commitment and depends on the integration between the student goals or needs and the educational environment provided by the institution. The strength of this commitment is evidence of the integration and informs the judgement or understanding of the course (Chambers, 1995, 344). Consequently, input should come from students with low commitment who discontinue, as well as students who have strong commitment and persist. Among that input should be evidence concerning goals that have changed and goals achieved along the way. Financial aspects can also be considered here as student financial costs and support schemes have significant influence on commitment.
Wadsworth (1991) argues that where users can go elsewhere for education then attendance is an indicator of commitment. But, where education is a monopoly then indicators may range from discontinuation, conflict, resistance, non-attendance, disinterest and passive compliance to more favourable comment. But, all the discontinuing students have one factor in common - the fact that the disharmony that they felt between the educational environment and their needs, resulted in discontinuation. Because of this, any evaluation of a course must take their input into account or it risks producing spurious results. Furthermore, it should not just take the discontinuation or their special characteristics into account, but the factors that made them discontinue.
Student development is taken up in an evaluation by considering both student and course goals (Cheung, 1998, 28). As well, the new emphasis on flexibility in education impacts on both the 'student goals - academic environment' match and the 'course goals - academic environment' match. The importance of these is shown by the serrated edges of the areas in Figure 1. An arrow connecting the serrated edges to the judgement and understanding illustrates this commitment.
If a technology changes during an evaluation then it can be considered as evaluation of different situations, or as separate evaluations with different uses of technology creating different scenarios. Alternatively, this could be viewed as an evaluation of a changing scenario as in formative evaluation. Either way the point must be addressed. In a contemporary on-line course the timescale becomes wedged between maximising the opportunity for students to express their ideas about the course, and minimising the effects of changing technologies. The Post-Structuralist and Post-Modernist influences reach into the other methodologies when they hold that what is viewed can only be considered valid in a specific context at a specific point in time. They show that timing the evaluation to include the discontinuing students and also conducting it over a generally shortened time frame within the lifetime of a technology, should form part of an evaluation framework. This means that positioning the evaluation to include discontinuing students, is in accordance with the Post-Structuralist view that evaluation considers a 'frozen instance' (Evans, 1994, 17). This is reflected in the inclusion of a timescale as a component as shown in Figure 1. This introduction of the elements of time and timing, provides a theoretical account of 'our present moment of time in history' (Jameson, 1984, 85). The thicker arrows on the timescale pointing inwards indicate the need to consider a shortened timescale for an evaluation.
However, a number of contradictions arise in evaluation involving discontinuing students. One is the espoused ideology of access and equity for students seeking to undertake on-line education. Most distance education policies base their raison d'être on this foundation. However, in the majority of cases in the higher education sector, the discontinuing students are not included among the evaluation participants. The curtailing of their access to education and the strength of the equity of their education within the course is not being questioned. Their voices are not heard. The framework used here addresses this inconsistency by ensuring that they are at least considered.
A second contradiction concerns the generalisations arising from an evaluation of a changing situation. Most evaluations result in judgements or better understandings of situations. These and recommendations for action, are usually written up in a report for wider dissemination to support any responsive action in this situation and others. If an evaluation is tailored to a specific situation then great care must be taken in forming generalisations, which purportedly apply to the specific scenario even after it has been changed, and even more care must be taken in applying these generalisations outside this situation. The virtues of an evaluation of a stable environment should guide evaluators to consider this issue.
In addressing these contradictions, evaluation is grounded in the fundamental philosophy that education is designed for self-fulfillment of the student and for the betterment of society. Even though the third emerging aspect in education, of cost-effectiveness and efficiency, is put to one side in this case, discontinuation of studies by students, is at some cost to the student, the educational system and society. If an evaluator, by listening to these voices, can enable courses to better cater for these students, then resources, previously wasted through discontinuation, can be used more fruitfully.
This framework used to cater for students who discontinue has been formed by merging different theoretical perspectives. Of special interest is the Durkheim/Spady theory demonstrating the influence of the aspirations of both an institution and its students and the educational environment that the institution provides, on a judgement and/or understanding of a situation and consequent action, or inaction. Durkheim's basic paradigm is used to justify a structure within the framework to facilitate input from all students - both continuing and discontinuing.
The perspective of time incorporates the work of Foucault and Lyotard and the idea of evaluation being 'a Post-Modern moment' (Usher & Edwards, 1994, 174). This leads to the stance that the 'moment' generally contains both a stable technology and the perceptions of the students who subsequently discontinue in order to be truly discerning. This is done by including consideration of the timing and timescale in the framework.
Ideology critique has been used to establish a central place in the framework for the educational environment of the course. Opportunities also exist within the framework for consideration of the aims of the course and actual achievements. As well, immanent critique reveals the virtue of considering both the course and student goals and course implementation to expose the espoused aims. The changed, and changing, nature of the students in an on-line course is catered for by consideration of the student background. The emphasis on determining the student goals and needs is helpful if a student discontinues; but is helpful too, in matching student needs with the educational environment provided by the institution.
Similarly, consideration of the social system ensures that the students, as part of the critical reference group, remain central in an evaluation. This generally espoused, but sometimes neglected, aspect is highlighted. As well, the early timing of the evaluation caters for the discontinuing students, as does the emphasis on integration of the students' goals with the educational environment. By moving the timing of the evaluation to an earlier position, evaluation caters for the sometimes very large student discontinuation rate and by planning an evaluation before the students start the course, valuable data can be gleaned from these students before they leave. Also, the short length of time for the evaluation to accommodate a stable technological context has been incorporated. This way, data can be collected before the technology changes and if necessary after it changes.
Thus, evaluation is set within a social context and the considerable changes in higher education currently affect this context in which the on-line course is conducted. But, by siting the elements of the evaluation within the community context, consideration of this changing scenario is ensured. Associated with this is the blurring of distinctions between distance education, open learning and flexible learning, so although the focus is on on-line education, it has the potential to inform practice in distance education, open learning and flexible learning as well. Again these influences can be considered in the community context and the educational environment. An evaluation then enables evaluators to look to the future and includes a cyclical process of action and betterment.
The main attribute of this form of evaluation is flexibility to cater for changes
in society, the higher education environment, distance education and on-line
practices. Yet, at the same time, as it encourages flexibility, it ensures reflection
on the fundamental components of evaluation and relationships between them within
an educational evaluation. This reflects the diversity of scenarios to be examined,
so a balance must be found between flexibility to cater for different situations
and consistency to enable generalisations to be made. The use of a framework
accommodating this ensures that an evaluation is comprehensive and efficient.
It is especially appropriate in this era of expensive technologies and burgeoning
on-line courses and students. Its use should point the way to better quality
evaluation and thus better quality teaching and learning.
Calvert, J. (1995). A study of the academic results on on-campus and off-campus students. Toowoomba: NCODE.
Calvert, J., Castro, A., Finnane, P., Holt, D., Juler, P., King, B., Mitchell, I., Robertson, B. & Jakupec, V. (1991). Distance teaching and learning 2 Study guide. Geelong: Deakin University and University of South Australia.
Candy, P. (1991). Self-direction in lifelong learning: a comprehensive guide to theory and practice. San Francisco: Jossey-Bass.
Castles, I. (1992). Education and training in Australia. Canberra: Australian Bureau of Statistics.
Chambers, E. (1995). Course evaluation and academic quality. In F. Lockwood (Ed.), Open and distance learning today (pp. 343-353). London: Routledge.
Cheung, D. (1998). Developing a student evaluation instrument for distance teaching. Distance education, 19(1), 23-42.
Commonwealth of Australia (2001) Students Preliminary 2001: Higher Education Statistics, http://www.dest.gov.au/highered/statistics/prelim/shes2001prelim.pdf Accessed 2 May 2002.
Durkheim, E. (1952). Suicide (Trans. Spaulding, J. & Simpson, G.). London: Routledge and Kegan Paul.
Evans, T. (1994). Understanding learners in open and distance education. London: Kogan Page.
Evans, T. (2000). Flexible delivery and flexible learning: developing flexible learners? In V.Jakupec & J. Garrick (Eds.), Flexible learning, human resource and organisational development: putting theory to work (pp. 211-224). London: Routledge.
Foucault, M. (1980). Power/knowledge (Trans. Gordon, A.). Brighton: Harvester
Freitag, E. & Sullivan, H. (1995). Matching learner preference to amount of instruction: an alternative form of learner control. Educational Technology Research and Development, 43(2), 5-14.
Garrick, J. (2000). Flexible learning, work and the management of 'intellectual' capital. In V. Jakupec & J. Garrick (Eds.), Flexible learning, human resource and organisational development: putting theory to work (pp. 239-256). London: Routledge.
Gooler, D. (1979). Evaluating distance education programs. Canadian Journal
of university continuing education, 6(1), 43-55.
Habermas, J. (1971). Knowledge and human interests (Trans. Shapiro, J.). Boston: Beacon Press.
Habermas, J. (1975). Legitimationsprobleme im Spatkapitalismus (Trans. McCarthy, T.). Boston: Beacon Press.
Herman, L. & Mandell, A. (1999). On access: towards opening the lifeworld within adult higher education systems. In A. Tait & R. Mills (Eds.), The convergence of distance and conventional education (pp. 17-38). London: Routledge.
Jakupec, V. (1998). Learning in flexible, open and distance education. Sydney: University of Technology, Sydney.
Jakupec, V. & Kirkpatrick, D. (1997). The problem of generalisation in higher education research in distance education. In T. Evans, V. Jakupec & D. Thompson (Eds.), Research in distance education 4 (pp. 197-210). Geelong: Deakin University.
Jameson, F. (1984). Postmodernism, or, the cultural logic of late capitalism. New Left Review, 146(July-August), 52-92.
Lewis, R. (1989). What is "quality" in corporate open learning and how do we measure it? Open Learning, 4(3), 9-13.
Long, M. (1994). A study of the academic results of on-campus and off-campus students; comparative performance within four Australian tertiary institutions (Commissioned Report 34). Canberra: National Board of Employment, Education and Training.
Lyall, R. & McNamara, S. (2000). Influences on the orientations to learning of distance education students in Australia. Open Learning, 15(2), 107-122.
Lyotard, J. (1994). The Postmodern condition: a report on knowledge (Trans. Bennington, G. & Massumi, B.). (Vol. 10). Manchester: Manchester University Press.
Malley, J., Brown, A. & Williams, J. (1976). Drop-outs from external studies: a case study of the investigation process. Epistolodidaktika, 2, 170-179.
Morgan, A. (1993). Improving your students' learning. London: Kogan Page.
Nunan, T. (1990, September 17-19). Evaluation of distance teaching - a review of techniques and standards. Paper presented at the Improved Distance Education on Agriculture, (pp. 1-9). Glenormiston: Victorian College of Agriculture and Horticulture.
Owen, J. & Rogers, P. (1999). Program evaluation. (2nd Ed.). St. Leonards:
Allen & Unwin.
Papert, S. (1985). Computer criticism vs. Technocratic thinking. Paper presented at the Logo85 Theoretical Papers, Cambridge, Mass.
Paul, J. (1999). Digital technology and university sovereignty: compatibility or collision course?, In V. Jakupec & J. Garrick, (Eds.), Flexible learning, human resource and organisational development: putting theory to work (pp. 30-46). London: Routledge.
Peters, O. (1992). Some observations on dropping out in distance education. Distance Education, 13(2), 234-269.
Posavac, E. & Carey, R. (1997). Program evaluation: methods and case studies. (5th Ed.). Upper Saddle River: Prentice-Hall.
Powell, R., McGuire, S. & Crawford, G. (1999). Convergence of student types: issues for distance education. In A. Tait & R. Mills (Eds.), The convergence of distance and conventional education (pp. 86-99). London: Routledge.
Robson, J. (1996). Some outcomes of learning through teleconferencing. Journal of Instructional Science and Technology, 3(June). http://www.usq.edu.au/electpub/e-jist/vol1no3/article1.htm Accessed 7/9/2001
Robson, J. (1997). The Significance of Students Who Discontinue in the Evaluation of Distance Education Courses. Paper presented at the International Council on Distance Education, Pennsylvania.
Robson, J. (1999, June). A framework for evaluation in distance education. Paper presented at the International Council on Distance Education, Vienna.
Rossi, P., Freeman, H. & Lipsey, M. (1999). Evaluation. (6th Ed.). Thousand Oaks: Sage.
Spady, W. (1970). Dropouts from higher education: an interdisciplinary review and synthesis. Interchange, 1(1), 64-85.
Spady, W. (1971). Dropouts from higher education: towards an empirical model. Interchange, 2(3), 38-62.
Stevens, K. (1995). Towards a philosophy of distance education. Unpublished Doctoral thesis, University of Melbourne.
Tait, A. & Mills, R. (Eds.). (1999). The convergence of distance and conventional education. London: Routledge.
Thorpe, M. (1995). The challenge facing course design. In F. Lockwood (Ed.), Open and distance learning today (pp. 175-184). London: Routledge.
Tinto, V. (1975). Drop out from higher education: a theoretical synthesis of recent research. Review of Educational Research, 45(3), 89-125.
Usher, R. & Edwards, R. (1994). Postmodernism and education. London: Routledge.
Wadsworth, Y. (1991). Everyday evaluation on the run. Melbourne: Action Research Issues Association (Incorporated).
Wilson, G. (1997). Multi-campus universities. In J. Sharpham & G. Harman (Eds.), Australia's future universities (pp. 277-288). Armidale: University of New England Press.
Young, G. & Marks-Maran, D. (1999). A case study of convergence between conventional and distance education: using constructivism and postmodernism as a framework to unconverge the mind. In A. Tait & R. Mills (Eds.), The convergence of distance and conventional education (pp. 175-187). London: Routledge.