Steve Rowe [HREF1], Lecturer, School of Commerce and Management [HREF2], Southern Cross University [HREF3], PO Box 157 Lismore, NSW, 2480. srowe@scu.edu.au
Peter Vitartas [HREF4], Lecturer, School of Commerce and Management [HREF2], Southern Cross University [HREF3], PO Box 157 Lismore, NSW, 2480. Email: pvitarta@scu.edu.au
This paper presents results of an analysis of secondary data sources that examine the participation and effect of changes to assessment tasks over two subsequent offerings of a unit in accounting theory being studied as part of an undergraduate business degree at Southern Cross University. The results are compared to performance on the assessment tasks of assignments and end of semester examination results as well as gender. The paper also presents the results of the effect of relaxing assessment deadlines on when contributions are made throughout the semester. The findings contradict a number of other studies of on-line discussion forums. Findings and avenues for further research are discussed.
A host of developments since the introduction of the Internet have seen on-line education gain greater status, acceptance and integration into tertiary education courses. Educators considering the adoption of on-line tools are drawn to the many advantages of incorporating on-line into their courses. An on-line course provides the ability for students to communicate with each other, resulting in additional interaction, particularly for those studying at a distance (O'Reilly & Morgan, 1999). Having an on-line course also provides the opportunity for students studying in different modes the ability to interact with each other, for example distance students can discuss topics with students studying internally.
With changes in the way students are studying, courses are being designed with greater flexibility in mind (Williams, 2002 [HREF5]). The use of on-line courses can facilitate such flexibility through a wider range of choices being available to students. In addition students have greater choice in the manner in which they can download resources and study materials, communicate with others in the course, and undertake activities at times that suit them. However, in developing on-line courses educators are warned of the need to consider the overall design of their course, and in particular to take into account the assessment (Kibby, 2002 [HREF6]; Hayes, 1999 [HREF7]).
While there has been a proliferation of articles and publications discussing
the issues of interaction, on-line participation, flexibility and assessment
there have been very few quantitative evaluations of courses that have adopted
on-line tools. This paper provides results of an analysis of secondary data
sources that examine the participation and effect of changes to assessment tasks
over two subsequent offerings of a unit in accounting theory being studied as
part of an undergraduate business degree at Southern Cross University [HREF3].
Text based on-line discussion boards have intuitive appeal to teaching staff who see them as a natural substitute for face-to-face discussions that can either augment or replace a tutorial, workshop or informal meeting situation. Such discussion boards are often incorporated into a course to ensure student participation (O'Reilly & Newton, 2001) and interaction.
Students have reported a number of advantages of online education including: 1) increased interaction for both quantity and intensity, 2) access to group knowledge and support, 3) democratic environment, 4) convenience of access: the `24 hour' class, 5) user control over the learning interaction, 6) motivational aspects and 7) text-based communication (Harasim, 1987 p. 124).
Interaction can be viewed in two ways, as interaction with content where critical thinking and reasoning skills are developed and as interaction with others to bring about a shared learning environment (Sringam & Greer, 2000). This can be with either the lecturer or student (Moore, 1989). In terms of the latter, interaction has been found to be particularly useful for forming friendships, offering advice, empathy and encouragement to continue studying in the learning environment, overcoming isolation and engaging in mutual support with peers (O'Reilly & Newton, 2001).
One approach to developing on-line interaction and discussion is through the use of discussion forums, or discussion boards (Bunker and Ellis, 2001 [HREF8]). On-line discussion, reflection and interaction encourages learning referred to as "social learning" where understanding and learning is acquired through modelling the behaviours, attitudes and reactions of others. In this way the process of observing, formulating an understanding and using that understanding to guide one's own behaviour occurs (Bandura, 1971).
Graham et al (2001) [HREF9] drawing on seven principles for good practice in undergraduate education developed by Chickering and Gamson (1987) evaluated four on-line courses to develop a list of "lessons learned" for on-line instruction. A summary of the findings are presented in Table 1. The evaluations were based on an analysis of on-line course materials, student and instructor discussion-forum postings, and faculty interviews. The principles and lessons developed by Graham et. al. (2001) [HREF9] highlight the importance of interaction and assessment in the development of on-line courses. Bunker and Ellis (2001) [HREF8] have also used the seven principles to identify potential enhancements that can be made by bulletin board use. These include providing the lecturer more opportunity to interact and encourage students to reflect and provide thoughtful responses, opportunities for peer feedback, keeping students on task for a greater amount of time and enabling less vocal students more opportunities to respond compared to a face-to-face situation.
|
Table 1: Principles for on-line instruction |
|
|
Principle (1) |
Lesson for on-line instruction (2) |
| 1: Good Practice Encourages Student Faculty Contact | Instructors should provide clear guidelines for interaction with students. |
| 2: Good Practice Encourages Cooperation Among Students | Well-designed discussion assignments facilitate meaningful cooperation among students. |
| 3: Good Practice Encourages Active Learning | Students should present course projects. |
| 4: Good Practice Gives Prompt Feedback | Instructors need to provide two types of feedback: information feedback and acknowledgment feedback. |
| 5: Good Practice Emphasises Time on Task | On-line courses need deadlines. |
| 6: Good Practice Communicates High Expectations | Challenging tasks, sample cases, and praise for quality work communicate high expectations. |
| 7: Good Practice Respects Diverse Talents and Ways of Learning | Allowing students to choose project topics incorporates diverse views into on-line courses. |
1: Chickering, A., & Gamson, Z. (1987)
2: Graham et al (2001) [HREF9]
There are two categories of choice in assessment tasks: choice within an assessment task and choice between assessment tasks. Choice within an assessment task is where the student can choose a topic or question that is of interest to them from a range of topics. This provides students the opportunity to study a topic that has greater interest rather than a topic that is imposed upon them. Choice between assessment tasks is where the student can make decisions about when they can submit their particular piece of assessment. Flexibility as to when an assessment piece is to be submitted allows the student to structure their own study plans and to cater for individual circumstances, such as due dates for assignments in other units.
External or distance students, and students who are working benefit from greater
choice in assessment. External students often have their patterns of study dictated
by forces outside their direct control by comparison to full-time internal students.
For example work and family issues directly impacting on their time. However
increasingly internal students are also coming under external pressures such
as holding down part-time jobs in order to pay their way through university.
Options to provide greater flexibility to students are welcomed by students.
In the traditional university setting assessment is generally accepted as the key to learning (Ramsden, 1992) and as the driver of students' approaches to study (Morgan, 1993). Gosper ( 2001) [HREF10] has identified two reasons for assessing participation - the first recognises students' workload and time commitments associated with on-line discussions and secondly to encourage students to participate and complete learning activities associated with the discussion. It is argued that those who choose to participate are often more engaged in the learning experience than for those whose participation is compelled. Voluntary participation it is suggested indicates a commitment to the task and signals a higher motivation to do well (Stecher et. al. 1996).
However, a number of educators have suggested that assessing participation does not lead to the facilitation of good learning outcomes because it is too subjective, it may discourage free and open discussion, students may be shy or students may talk to gain credit (Davis, 1993; Lacoss & Chylack, 1998 [HREF11]). It is suggested that a reward system is a more appropriate way of recognising participation (Davis 1993). Others see on-line participation's support benefits sufficient to not warrant assessment (O'Reilly & Newton, 2001).
However there are arguments put forward to support assessment of participation.
Heath (1998) [HREF12]acknowledged
that "even if the students in an on-line course possess strong motivation
and good writing skills, there is still the matter of insuring that enough students
are participating, thoughtfully, in the on-line discussions" (p. 13). Hallett
and Cummings (1997) found that students did not post additional on-line comments
beyond the required assignments because the work was not graded. While Muirhead
(2000) [HREF13]
when conducting focus groups found that students suggested there be student
accountability for keeping up to date with weekly discussions.
While there does not appear to be a clear answer to the question of whether on-line participation should be assessed, the importance of student participation has been found to be beneficial. For example Levengurg and Major (1999) found students who spend more time reading their classmates postings and posting their own messages achieved course objectives at a significantly higher level than those who spend less time engaged in those activities. Barrett and Lally (1999) reported that men and women took distinctively different roles in an on-line learning environment with men sending more messages than women. Others have called for further research into the effects on participation levels of graded participation in discussion groups (Ho, 2002 [HREF14]).
In this paper the results of a study into the effect of relaxing assessment
requirements on when students submit their assessment items, the relationship
between the number of contributions and gender and performance in other assessment
items are reported.
One of the decisions the educator faces when adopting an on-line mode of teaching is whether to replace or supplement existing teaching methods. In teaching accounting theory the traditional approach to learning involved having students read articles and cases that are then discussed in a tutorial. The volume of theoretical and technical reading is high, with attendance and participation rates traditionally low. The teaching challenge in the unit is that students are expected to take a different approach to their learning. In the past students often commented that Accounting Theory is different from the other accounting subjects they study. The structured approach used in accounting subjects prior to taking accounting theory does not fit the material in the unit. Students are challenged to present alternative ways of using accounting information rather than simply preparing that information within specified constraints. A useful way to express this is that they are encouraged to learn how to determine what the numbers are, rather than what to do with the numbers with which they are provided.
Accounting subjects generally involve the preparation and analysis of numerical data for decision making with relatively clear cut answers to problems. Accounting Theory by contrast examines a number of philosophical approaches to the study of accounting. One of the major objectives of the unit is to provide students with an alternative perspective of what has been synthesised into generally accepted accounting principles (GAAP). Students are challenged to take the perspective of different users and preparers of (financial) information, and to appreciate how accounting information may be utilised in different ways, often with conflicting interpretations. The different views and interpretations that can be applied to problems have been found to unsettle students and as a consequence they become confused and unsure of their positions on a topic.
The subject has been primarily taught through lectures and tutorials where student have an opportunity to put forward their interpretations and views on topics and argue for the positions they take. Tutorials can be lively and engaging for many students, however it has also been noted that some students may retire to the background and not contribute strongly to the discussions. Anecdotal evidence suggested that, for at least some students, putting forward their view in the tutorial in a spontaneous manner was threatening and although they had been thoroughly prepared and willing, they did not contribute in a face-to-face situation. This reflects the theoretical and technical nature of the readings that pose language and definitional problems with their expression.
In 2001 a review of the unit identified the potential of utilising an on-line discussion forum as one way to facilitate greater interaction from students. A steady decline in the number of students undertaking the unit in internal mode meant that opportunities for highlighting differences in opinion and interpretation had become limited. Prior offerings in the unit had evolved to assessment requiring two essays and a final exam. Some way of harnessing the views of the growing cohort of external participants was considered with the use of an asynchronous medium seen as providing an opportunity for renewing student interaction and discussion. In addition an on-line discussion forum was seen as an opportunity to overcome the difficulties of dealing with the theoretical and technical nature of the reading material. The shy or inexperienced student would also have the opportunity to prepare and contemplate their answers prior to posting. While the flexible nature of a discussion forum was seen as providing students greater opportunity to prepare, as well as being encouraged by the contributions and activity of others in the class. Hence the assessment title "reflective forum" was chosen. A discussion forum was developed for use in the unit to meet the following unit objectives:
The on-line component of the unit was first made available to students in 2001. In designing the delivery an effort was made to utilise the technology in a manner that reflected the activities undertaken in the traditional face to face running of the unit. This involved the provision of study and lecture notes, and the use of a discussion forum to facilitate the reflection about, and discussion of, topic readings. There was concern that students would not become involved with the on-line system. As a result, the grading for the discussion forum was limited to an "attendance" mark as an incentive for students to make contributions. The activity required students making a reflective contribution to each of 12 topics based on their readings for that topic. A reflective activity was selected as it was seen as an appropriate means of developing critical learning skills and can easily be utilised in an on-line asynchronous learning environment (Dysthe, 2002). In addition students were required to comment on at least one other student's contribution in a constructive manner. Each week a forum was opened relating to the topic of study for that week and after two weeks the discussion forum was archived. This effectively meant that the forum was closed as the archive is read only - no new comments could be added to that forum. This protocol was relaxed after Topic 4 as feedback from students, and the number of comments submitted to an "after-archiving" forum, made it clear that study patterns were not as regular as traditional face-to-face scheduled classes allow.
Students were briefed on how to use the discussion forums with emphasis on the objective of the contributions as being to share their reflections with others. They were also reminded to be considerate of others and of appropriate netiquette. They were also provided with sample contributions and appropriate responses as a guide.
The discussion forum was judged to be a success by the teaching staff with the lecturer receiving a number of very positive comments from students and ideas for improving the forums. In 2002 some modifications were made to the assessment requirements to recognise the time involved in contributing to the discussion forum activity. The requirement that students make a posting each week was removed so students could post their contributions at any time during the semester, however they were still required to post contributions for each topic. Students were also advised that it was preferable if they made contributions each week, however there was no requirement that they had to do this every week so that if they fell behind they could "catch-up" in a later week. In addition the assessment weighting of the activity was increased from 10% to 20% and the requirement of an assessment item reduced from two to one. Not only did this recognise the amount of reading required of students, it was also intended to encourage greater contribution and involvement rather than simply "attendance". The discussion forum mark was allocated based on both number of contributions and the extent of involvement over the duration of the semester.
The following section reports on the results of data collected on the number
and timing of contributions to the discussion forums.
The students used the discussion boards in Course Info made available through
the university student portal. The results reported here are based on records
collated from the discussion board contributions at the end of semester. As
2001 was a trial period less detailed information was collected compared to
the following year. Additional information was collected in 2002 allowing a
more comprehensive data set for analysis and to examine the effect of changes
in the assessment task. For 2001 & 2002 data was collated on the number
of contributions per topic for each student while the 2002 data also included
the number of contributions per topic by week of the semester. In addition the
students' on-line participation mark, assignment marks and final grade were
also available. Results presented here are based on 78 students enrolled in
2001 and 59 students from 2002. Students were mainly in the latter stages of
completing a business degree majoring in accounting with the majority enrolled
in the unit externally (77%). There were roughly even numbers of males and females
(47% and 53% respectively).
Results for the two student cohorts are presented in Table 2. The data covers the number of students contributing to each topic, the total number of contributions made by students, the average number of comments for each topic and the maximum number of comments made by one student for each topic.
The results show that not all students contributed to each topic with the greatest
number of students contributing to the early topics and the least number of
students being at the end. This effect is also apparent in 2002. In 2001 there
were a total of 1,317 comments made by students, an average of 110 for each
topic and an average of 1.79 per student. This is in comparison to 1,453 contributions
made by the 2002 cohort with an average of 121 per topic and 3.04 per student.
While the number of students in 2002 contributing to the forum is lower they
have contributed a larger number of comments. This is attributed to two factors,
the first was the higher assessment weighting but also from the less restrictive
requirement that comments be made within a restrictive time frame. It is also
noticeable that the maximum number of contributions by any one student was also
higher among the 2002 cohort.
The pattern of contributions for the two groups is similar, with the greatest
number of contributions being in the first part of the semester and the least
number in the later part of the semester. The relationship was examined further
by way of a correlation analysis. The results show a strong correlation between
the number of comments made by each cohort (r=.754) which was significant at
the 0.01 level. This result indicates there is little difference in the number
of comments made for each of the topics by the two groups.
Contribution level, assignment and examination marks
Of interest is whether the level of contribution was also reflected in other
marks. To examine this respondents were divided into three groups based on the
number of contributions they made to the discussion forum. Marks for the assignment
and final examination were compared between the three groups, low, medium and
high contributors using an analysis of variance (ANOVA). Marks for participation
and final grade were not analysed due to a multicolinearity effect, as the participation
marks and final grades were based on the number of contributions.
When comparing the marks achieved in the assignment and final examination for
the three groups of contributors no significant differences were found (F(2,
131) = 2.103 and F (2, 131)=.894 for assignment and examination mark respectively)
indicating that there was no difference in the marks for the assignments or
the exams for student regardless of the number of contributions they had made
to the discussion forums.
Barrett and Lally (1999) reported that men and women took distinctively different roles in an on-line learning environment with men sending more messages than women. An examination of differences between gender was made with comparisons between the number of contributions for each topic (range from F (1, 107) = 0.001 to 3.432), the number of overall contributions (F (1, 132) = .311), and final examination mark (F (1, 135) = .464). All of the results were not significant indicating no differences existed between males and females.
In 2002 the requirement to submit contributions each week was relaxed. Of interest was whether relaxing the requirement would result in students reallocating their contributions throughout the semester and avoid peak times such as when other assignments were due and the end of semester. The results of the number of contributions each week are presented in Graph 1 and Table 3. Based on the results it would appear that students did not reallocate their workload, with almost half of the contributions (44%) being made in the final two weeks of the semester. This would suggest that students have not taken advantage of the assessment freedom nor shown good time management skills. The results also raise the question as to whether the students would learn effectively from participating in the activity if they completed a large proportion of their assessment items at the end of the semester.
The on-line participation and assessment reported in this study has utilised information drawn on the number and timing of contributions to an assessment task utilising a discussion forum.
The findings revealed that increasing the weight of the assessment did increase the number of contributions made by the students between the two years. The greater expectation of the activity, as reflected in the mark and less pressure from an additional item of assessment resulted in more contributions by students. It is noted however that it did not increase the proportion of the students engaging in the activity although on checking the records 2002 had a larger number of students who did not contribute to the discussions due to extenuating circumstances.
In relation to the number of contributions made to each topic it would appear that students make most comments early in the semester and then the effort decreases as the semester progresses. This drop-off may occur due to other workload pressures, although for the 2002 cohort the greater freedom in when students could contribute should not have had an effect. The finding is more likely highlighting a drop in interest in the latter part of the course and consideration of these latter topics may be useful to provide some stimulus that will rejuvenate the level of interest in the latter topics. In effect the students may have "burned-out" and needed another activity to maintain their interest in the latter topics. This finding raises questions about what is the appropriate length of an on-line activity, a topic that would appear worthy of future research.
The finding that there were no differences in final examination and assessment marks between groups with different levels of contributions raises questions about the motivation of students in contributing to on-line discussion forums. This finding is contrary to those of Levengurg and Major (1999) who found that students who spent more time with on-line achieved course objectives at higher levels than other students. While the nature of assessment in this study was different from those of Levengurg and Major it is clear more work is needed to understanding how on-line discussions contribute to learning objectives and their assessment.
Another surprising finding was that there were no gender differences apparent in the number of contributions or the assessment marks for the two cohorts of students. This is contrary to Barrett and Lally (1999) who found that males and females took different roles in an on-line learning environment. Again the nature of the subject and type of assessment is different in this study however it is noted that the Barrett and Lally study used a much smaller sample size and evaluated qualitative differences rather than quantitative results.
The findings also revealed that when students are given more flexibility in when an on-line assessment task is due, they will not necessarily demonstrate better management of their time. For some subjects the use of assessment deadlines may be required to provide impetus for students to complete tasks within certain times.
While other researchers have recounted the many positive responses from participants in on-line discussion, and while it is acknowledged there are advantages to utilising a discussion forum, the evidence presented here suggests that further work and refinements are required in the use of the discussion forum for this course. Of prime concern is how to facilitate conditions that encourage students to maintain involvement with the on-line environment and ensuring it is beneficial to their learning. The findings also highlight the importance of regularly assessing the process and assessment of on-line activities.
This study has demonstrated that the use of secondary data from discussion forums can be an effective method to examine and reflect on the performance of on-line activities in a unit. While the assessment has been based on the number and timing of the contributions it is acknowledged that qualitative aspects such as student feedback and the nature and quality of contributions are equally important in evaluating the performance of on-line activities. Reporting these aspects was beyond the scope of this paper.
Bandura, A. (1971) Social Learning Theory. General Learning Press, New York.
Barrett, E. and Lally, V. (1999) "Gender differences in an on-line learning environment" Journal of Computer Assisted Learning, 15, pp. 48-60.
Bunker, A. and Ellis, R. (2001) "Using bulletin boards for learning: What do staff and students need to know in order to use boards effectively? In A. Herrmann and M. M. Kulski (Eds), Expanding Horizons in Teaching and Learning. Proceedings of the 10th Annual Teaching Learning Forum, 7-9 February 2001. Perth: Curtin University of Technology. Available Online [HREF8]
Chickering, A., & Gamson, Z. (1987) "Seven Principles of Good Practice in Undergraduate Education". AAHE Bulletin, 39, 3-7.
Davis, B.G. (1993) Tools for Teaching, Jossey-Bass Publishers: San Francisco.
Dysthe, O. (2002) "The learning potential of a web-mediated discussion in a university course". Studies in Higher Education, 27 (3), 339-352.
Gosper, M. (2001) "Assessment of Online Participation. A Summary of the issues raised by the Centre for Flexible Learning Teaching", Online @ Macquarie, 4 April 2001. Available Online [HREF10]
Graham, C., Cagiltay, K., Craner, J., Lim, B., & Duffy, T. M. (2000) "Teaching in a web-based distance learning environment: An evaluation summary based on four courses". Center for Research on Learning and Technology Technical Report No. 13-00. Indiana University Bloomington. Available online [HREF9]
Hallet, K. & Cummings, J. (1997) The virtual classroom as authentic experience: colaborative, problem-based learning in a WWW environment. Competition-Connection-Collaboration: Proceedings of the Annual Conference on Distance Teaching and Learning, Madison, WI: University of Wisconsin-Madison, 103-107.
Hayes, S. (1999) "Assessment Online" Assessment On-line Symposium, Learning and Teaching Services, Swinburne University of Technology. 27 August. Available online [HREF7]
Heath, E. F. (1998). "Two pints of cheer and a pint of worry: an on-line course in political and social philosophy". Journal of Asynchronous Learning Networks. Available online [HREF12]
Harasim, L. (1987). "Teaching and learning on-line: issues in computer-mediated graduate courses". Canadian Journal of Educational Communication, 16 (2), 117-135.
Ho, Suzanne 2002, "Evaluating students' participation in on-line discussions" Proceedings of AUSWEB02 The Eighth Australian World Wide Web Conference, Twin Waters Resort, Sunshine Coast Queensland 6 - 10 July 2002. Available online [HREF14]
Kibby, Marj (2002) "Assessing Students Online" School of Social Sciences, The University of Newcastle - Australia. June 20. Available online [HREF6]
Levenburg & Major (2000) "Motivating the Online Learner: The Effect of Frequency of Online Postings and Time Spent Online on Achievement of Learning Goals and Objectives" Proceedings of the International Online Conference On Teaching Online in Higher Education, 13-14 November 2000. Indiana University-Purdue University: Fort Wayne.
Lacoss, J. and Chylack, J. (1998) "What makes a discussion section productive?
Teaching Concerns, Fall. Available online [HREF11]
Moore, M. G. (1989) "Editorial: three types of interaction" The American Journal of Distance Education, v3 (2).
Morgan, A. (1993) Improving Your Students' Learning: Reflections on the Experience of Study. London: Kogan Page.
Muirhead, B. (2000) "Enhancing social interaction in computer-mediated distance education" Educational Technology & Society. 3, (4) 2000. Available online [HREF13]
O'Reilly, M. & Morgan, C. (1999) "Online assessment: creating communities and opportunities". In S. Brown, P. Race & J. Bull (eds), Computer Assisted Assessment in Higher Education. London: Kogan Page SEDA, 149-161.
O'Reilly, M. & Newton, D. (2001) "Why interact online if it's not assessed?" Academic Exchange, Winter, pp. 70-76.
Ramsden, P. (1992) Learning to Teach in Higher Education. London: Routledge.
Sringam, C. & Greer, R. (2000). "An investigation of an instrument for analysis of student-led electronic discussions" in R. Sims, M. O'Reilly & S. Sawkins (eds) Learning to Choose - Choosing to Learn, Proceedings of 17th Annual ASCILITE conference, Coffs Harbour NSW: Southern Cross University Press, 81-91.
Stecher, B., Rahn, M.L., Ruby, A., Alt, M. N. Robyn, A. and Ward, B. (1996) Using Alternative Assessments in Vocational Education. Santa Monica, CA: Rand Corporation.
Williams, J. B. 2002, "Assessment in an on-line environment: whither the examination hall?" Teaching and Educational Development Institute Conference. Available online [HREF5]