Examining the relationship between higher-order learning and students' perceived sense of community in an online learning environment

Dr. Peggy A. Ertmer [HREF1], Associate Professor, School of Education [HREF2], Purdue University [HREF3], 100 N. University St., West Lafayette, IN, 47907-2098. pertmer@purdue.edu

Dr. Donald A. Stepich [HREF4] Associate Professor, Department of Instructional and Performance Technology [HREF5] Boise State University [HREF6], 1910 University Dr., Boise, ID, 83725-2070. dstepich@boisestate.edu

Abstract

Using specific strategies, recommended in the literature, for building a sense of community in online learning environments, we connected students at two universities through electronic discussions and video conferencing. To determine the effectiveness of our approach, we examined the relationships among students' perceptions of community, their perceived learning, and measures of higher-order thinking. Results support the hypothesis that perceptions of community and perceived learning are related. However, evidence of a relationship between perceptions of community and higher-order thinking was not found, suggesting that these relationships may be more complex and difficult to discern.

Introduction

There has been a great deal of discussion, in recent years, of the importance of establishing a sense of community in online learning environments in order to maintain student motivation and increase learning (Palloff & Pratt, 1999; Wilson, 2001). Garrison, Anderson, and Archer (2001) observed that the community is ''extremely valuable, if not essential'' (p. 7) in the development of higher-order learning because it provides the social context in which learning occurs. Bielaczyc and Collins (1999) concurred, noting that community creates a culture of learning that can promote the kinds of higher-order learning currently being advocated in the literature (Bransford, 1993; Resnick, 1987). According to Rovai (2002b) a classroom community is defined as a ''social community of learners who share knowledge, values, and goals'' (p. 322) and is comprised of two main components: 1) connectedness-students' feelings of ''cohesion, spirit, trust, and interdependence'' (p. 325), and 2) learning-students' feelings ''regarding the extent to which their learning goals and expectations are satisfied'' (p. 325).

Recent research has established a clear link between a sense of classroom community and perceived learning (Picciano, 1998; Rovai, 2002b). According to Lipman (1991), participants in a community promote learning by: 1) listening to one another with respect, 2) trying to identify one another's assumptions, 3) challenging one another to supply reasons for otherwise, unsupported opinions, 4) building on one another's ideas, and 5) assisting each other in drawing inferences from what was said. This implies that a critical element in a community of learners is the interaction that takes place among its members. For example, Arbaugh (2000) investigated 5 Internet-based MBA courses and found that students' perceptions of learning were most closely associated with the instructor's emphasis on interaction within the course, ease of interaction, and classroom dynamics. Cunningham (1992) also emphasized the importance of interaction, noting that it was the dialog among community members that promoted learning.

Various authors have described strategies for building this kind of community within online learning environments. For example, Haythornthwaite, Kazmer, Robins, and Shoemaker (2000) suggested three basic strategies: 1) promote initial bonding, 2) monitor and support continual participation, and 3) provide multiple means of communication. Over the past several years, our work has been directed at applying these community-building strategies within our graduate-level instructional design (ID) courses (Stepich & Ertmer, 2003). Using strategies specifically designed to build a sense of community among students at our two universities, we have gathered data to determine the extent to which students' perceptions of community relate to their perceptions of learning in this collaborative online learning environment. In addition, we have gathered data to determine whether a similar relationship exists between a sense of community and actual learning outcomes (i.e., change in knowledge or skill), as determined by measures other than self-report. As Williams and Ceci (1997) suggested, students' ratings of perceived learning do not always correlate with actual achievement scores. Thus, we examined evidence of learning, in addition to evidence of perceived learning.

Purpose

The purpose of this study, then, was to extend our previous work by clarifying the relationships among sense of community, perceived learning, and learning outcomes for students enrolled in a case-based instructional design (ID) hybrid course. Using ''the development of students' higher-order thinking'' as the intended learning outcome, we examined changes in students' thinking and problem solving skills during the semester and investigated how these developing skills related to their judgments of perceived learning, as well as their perceptions of community.

Methods and Procedures

Participants included 11 graduate students (3 PhD; 8 MS) enrolled in an advanced ID course at a midwestern (n = 5) or northwestern (n = 6) university during fall 2003. Students ranged in age from 27-52 years (x = 38) and represented a broad range of backgrounds, knowledge, and experiences. Throughout the semester, students from the 2 universities collaborated in 12 online case discussions via an electronic bulletin board and in 7 face-to-face discussions via Internet-based video conferencing.

Data Collection and Analysis

Perceived Learning

At the beginning and end of the course, students completed a self-assessment questionnaire in which they rated themselves on 16 ID competencies (IPSTPI, 1984) from 1 (weak) to 5 (strong). Sample competencies included: Determine projects that are appropriate for instructional design, conduct a needs assessment, design instructional materials, and so on. A gain score was calculated to determine changes in students' judgments of competency, thus serving as a measure of perceived learning.

Sense of Community

At the end of the semester, students completed the Classroom Community Scale (Rovai, 2002a), in which they rated levels of agreement from 1 (strongly disagree) to 5 (strongly agree) on items related to feelings of connectedness (e.g., ''I trust others in this course.'') and learning (e.g., ''I feel that I am given ample opportunity to learn.''). Ratings for each item were totaled to obtain a scale score ranging from 20 to 100, with scores for each of the two subscales (connectedness, learning) ranging from 10 to 50. Higher scores reflected a stronger sense of classroom community.

Learning Outcomes

Two data sources provided measures of higher-order learning: students' weekly bulletin board postings and their pre- and post-case analyses of published instructional design case studies. An independent rater scored each posting on the bulletin board (n = 631), based on the level of cognitive skill demonstrated, using Bloom's taxonomy. Postings at the knowledge, comprehension, and application levels received 1 point; postings demonstrating analysis, synthesis, or evaluation received 2 points; non-substantive postings were tallied but received no points. The rater was a graduate assistant for the course whose main responsibility was to read and score students' messages on the bulletin board using a rubric, including sample postings and scores, developed by one of the authors.

Throughout the course, students analyzed a series of case studies involving two basic tasks: 1) analyzing problems and issues in the case and 2) making recommendations for identified issues. Case responses were typically posted online as part of the weekly discussions. However, initial and final case analyses were submitted by students individually as written assignments. These analyses related to the same case study and, thus, served as pre- and post-learning measures. After the graduate assistant removed all identifying information (including whether the response was a pre- or post-course assignment), the authors scored each response using a rubric based on six primary characteristics of expert-novice problem solvers described in the literature (e.g., conceptualization of the issues, impact and implications of recommendations; see Table 1) and outlined previously (Stepich, Ertmer, & Lane, 2001). An independent score, from 0-3, was assigned to students' responses, based on each of the 6 characteristics, using the following criteria: If a response exhibited no expert-like elements related to a specific characteristic, the response was given a score of 0 (no evidence) for that characteristic. If the response included at least one expert-like element, but there were more novice-like elements than expert-like elements, it was given a score of 1 (some evidence; mostly novice approach); if a response included some expert-like elements but there were more expert-like elements than novice-like elements, it earned a score of 2 (some evidence; mostly expert approach); finally, if a response exhibited a substantial number of expert-like elements, and the prominent approach taken by the student was expert-like, it was given 3 points (a lot of evidence; prominent approach). To increase the consistency of our ratings, we identified examples from students' responses that seemed to clearly illustrate the different ratings for each characteristic. These examples, then, were used as templates to guide our continued analysis efforts. After consensus was reached on all ratings, scores were totaled for each analysis. Possible scores ranged from 0 (no expert qualities) to 18 (mostly expert-like). Following this, responses were identified as belonging to specific individuals, sorted into pre- and post-course responses, and used for further analyses (e.g., correlations, t-tests, etc.).

Secondary Data

Course assignments (including students' additional case analyses), discussion board postings, and mid- and end of course evaluations served as secondary data sources and were used to triangulate survey findings.

Results and Discussion

Perceived Learning

Results of the self-assessment questionnaire showed a significant increase (t = 4.16; p = .001) in students' perceptions of their ID skills from pre- to post course. On average, students' ratings changed .51 points from 3.38 (neither weak nor strong) to 3.85 (somewhat strong).

Sense of Community

Students' scores on the Community Scale averaged 83/100 points suggesting that, in general, students ''agreed'' that they felt connected to each other (x = 39/50) and ''strongly agreed'' that the community enabled them to reach their learning goals (x = 44/50). Students' comments at the end of the course supported this: ''I found it very valuable to collaborate with a diverse group ­ I gained a much better understanding of ID in the field.''

Learning Outcomes: Bulletin Board Postings

Scores reflecting the quality of students' postings also showed steady increase over the course of the semester, ranging from a total of five 2-point responses (13%) during the first discussion to 19 (53%) during the last week. Similarly, 0-point responses decreased from 11 during the first week (29%) to zero during the last week. Two-point responses accounted for 38% of the total postings; 0-point responses equaled 7.5%. To determine if the change in the number of 2-point postings from pre- to post-course was significant, a t-test was performed using the number of 2-point posts, for each student, from weeks 1-3 as the pre-measure, and the number of 2-point posts from weeks 11-13 as the post-measure. This was based on the rationale that one week of postings (e.g., the first and last weeks of the semester) might not be as representative of a students' work as a series of weeks. Results of the t-test indicated a significant increase (t = 8.23; p < .0000) in the number of 2-point posts. On average, students made 1.8 two-point postings during the first three weeks of the semester, while they made an average of 6 two-point postings during the latter weeks of the semester. Furthermore the standard deviation increased from 1.32 during the first 3 weeks to 2.61, suggesting that some students improved more than others in their higher-order thinking skills, as represented by the upper levels of Bloom's taxonomy.

Learning Outcomes: Students' Case Analyses

Analyses of students' case responses showed no significant difference from pre to post course. Students' scores on the pre-case analysis ranged from 1 - 15 points, with an average of 6.1 points; scores on the post-case analysis ranged from 3 - 16, with an average of 6.3 points. Whereas some students increased their ratings from pre- to post-course by 2 or 3 points, others decreased their ratings by a similar amount, thus, canceling out any noticeable gain, overall, in expert-like thinking. It is important to point out, however, that these averages do not include the scores from one student who obtained a post-course score that was 11 points lower than her pre-course score. Despite being one of the strongest contributors in the class and having the strongest rating on her pre-course analysis, she submitted a very short and fairly superficial response for the post-case analysis. This was possibly due to the fact that she was simultaneously involved in studying for and taking the comprehensive exit exam for her master's degree. Given these circumstances, we decided to exclude her scores from further analysis, as it appeared that her lower score on the post case analysis was most likely due, not to a lack of expertise (she obtained 15/18 points on the pre-case analysis), but to her lack of attention to this particular task at this point in her program. Furthermore, because this score represented such a drastic departure from the results obtained from the other ten students, it would have skewed the outcomes to such an extent that it would have been difficult, if not impossible, to interpret our findings.

Comparisons between students' pre- and post-case analyses, then, suggest that students' ability to solve case problems did not improve, to any measurable extent, during the course. Students who started the course with fairly expert-like approaches, ended the course with fairly expert-like approaches; similarly, students who started with fairly novice-like approaches, ended with fairly novice-like approaches. While it is certainly possible that students' higher-order thinking skills did not improve during the course, the significant increase noted in the quality of students' online postings suggests otherwise. A number of other possible interpretations bear further consideration. First, our qualitative analysis techniques may not have been sophisticated enough to detect the changes that occurred in students' case analysis approaches. While our analysis categories were based on expert characteristics described in the literature, application of these categories to widely divergent case write-ups is a challenging process, and one that we continue to refine. Second, it is possible that while students' ability to analyze case problems improved, they did not demonstrate this to the extent to which they were capable. In other words, the assignments might not have been structured well enough to allow students to demonstrate the skills they had. Third, other variables may have negatively impacted students' post-case responses, particularly other commitments or pressures that tend to occur at the end of a semester. The point-value assigned to the completion of this final case analysis was weighted relatively low in relationship to the total number of possible points in the course (7/150 points) and students may have given it low priority compared to other tasks needing to be completed. Future work includes continued refinement of our analysis techniques as well as a plan to change the structure and the value of the pre and post-case assignments.

Relationships among Variables

Pearson product correlations (see Table 2) showed a significant relationship between perceived learning (i.e., gain scores on the ID self-assessment) and the community learning subscale (r = .64; p < .05), but not between perceived learning and the connectedness subscale (r = .39) or the total scale (r = .52). While students with greater gains in perceived learning were more likely to feel that the community contributed to their learning, students who felt more connected didn't necessarily feel that they learned more. These results suggest that perhaps what is most important to students is the perception that they can learn from the community, whether they feel a strong sense of cohesion with the group or not. Alternatively, the sense of connectedness may have a less direct effect on perceived learning. This alternative view is supported by student comments: ''We certainly benefited from the range of experience and numbers of the other class. The community-spirit/connection was less apparent than in the face to face situation.'' Future work might try to tease apart which of these community perceptions are most critical to the learning experience.

While previous researchers have suggested that measures of perceived learning are an adequate substitute for measures of actual learning outcomes (Rovai, 2001a, 2001b), the results of this study suggest the need to examine this practice more closely. That is, this study found no systematic relationships between the measures of community and measures of learning (see Table 2). Neither did we find significant relationships between our measure of perceived learning and our two measures of learning. At least in this study, it appears that these instruments were measuring different things. Future work suggests the importance of validating current measures.

While we did not find a significant relationship between measures of perceived learning and measures of learning, we did find positive and, in some cases, significant relationships between the various measures of students' higher-order thinking used in this study (see Table 3). That is, students' pre- and post-case analyses scores were significantly correlated to the number of 2-point posts (r = .62 and .67, respectively; p < .01), suggesting that these two measures were assessing similar cognitive abilities.

Implications

Taken together these results suggest that while students' perceptions of community appear to relate to their perceptions of learning, actual learning outcomes may not, indeed, be influenced by these perceptions, contrary to what has been suggested in the literature (Picciano, 1998; Rovai, 2002b). Although the small number of participants in this study warrants caution in interpretation and in generalizing to other contexts, it does suggest that more research needs to be conducted in which measures of students' learning and/or achievement are obtained.

Even though students in this study demonstrated increases in higher order thinking skills, as well as increases in their judgments of ID competencies, it does not appear, at least from the results obtained here, that these changes were systematically related to their perceived connectedness to others in the class. This suggests the need for additional work in this area. As Bonk (2003-2004) cautions, ''Although this quest for community may be considered the penultimate goal of online discussion, we must ask research questions about the learning benefits of such communities. … Is there always a satisfactory marriage'' (p. 100)?

Conclusion

Many online programs specifically support activities to develop feelings of community among participants (Swan, 2002) based on the assumption that this will increase both student motivation and learning. While the results of this study lend some support to this assumption, additional work is needed. As suggested by Swan, ''Researchers should explore those unique characteristics of asynchronous online environments that matter, or can be made to matter, in learning and instruction'' (p. 26). Furthermore, this implies the need to continue to look beyond perceived learning to determine whether a similar relationship exists with actual learning outcomes (i.e., change in knowledge or skill), as determined by measures other than self-report. Research in this area is still incredibly young; however, given the large number of students enrolled in online courses (and the speed with which this number is growing), this is an important area for our continued efforts.

References

Arbaugh, J. B. (2000). ''How classroom environment and student engagement affect learning in Internet-based MBA courses'' in Business Communication Quarterly 2000 v.63 n.4 p.9-26.

Bielczyc, K., and Collins, A. (1999). ''Learning communities in classrooms: A reconceptualization of educational practice.'' In C. M. Reigeluth (Ed.). Instructional-design theories and models (Vol. II): A new paradigm of instructional theory (pp. 269-292). Mahwah, NJ: Lawrence Erlbaum.

Bonk, C. (2003-2004). ''I should have known this was coming: Computer-mediated discussions in teacher education'' in Journal of Research on Technology in Education 2003-2004 v.36 p.95-102.

Bransford, J. D. (1993). ''Who ya gonna call? Thoughts about teaching problem solving.'' In P. Hallingen, K. Leithwood, and J. Murphy (Eds.). Cognitive perspective on educational leadership (pp. 171-191). New York: Teachers College Press.

Cunningham, R.D. (1992). ''Beyond educational psychology: Steps toward an educational semiotic'' in Educational Psychology Review 1992 v.4 p.165-194.

Garrison, D. R., Anderson, T., and Archer, W. (2001). ''Critical thinking, cognitive presence, and computer conferencing in distance education'' in The American Journal of Distance Education 2001 v.15 n.1 p.7-23.

Haythornthwaite, C., Kazmer, M. M., Robins, J., and Shoemaker, S. (2000). ''Community development among distance learners: Temporal and technological dimensions'' in Journal of Computer Mediated Communication 2000 v.6 n.1. Available online [HREF7].

International Board of Standards for Training, Performance, and Instruction (IBSTPI) (1984). Instructional design: The standards. Washington DC: Author.

Lipman, M. (1991). Thinking in education. Cambridge: Cambridge University Press.

Palloff, R. M., and Pratt, K. (1999). Building learning communities in cyberspace: Effective strategies for the online classroom. San Francisco: Jossey-Bass.

Picciano, A. G. (1998). ''Developing an asynchronous course model at a large urban university'' in Journal of Asynchronous Learning Networks 1998 v.2 n.1. p.1-14. Available online [HREF8].

Resnick, L. (1987). ''Learning in school and out'' in Educational Researcher 1987 v.16 n.9 p.13-20.

Rovai, A. P. (2002a). ''Development of an instrument to measure classroom community'' in Internet and Higher Education 2002 v.5 p.197-211.

Rovai, A. P. (2002b). ''Sense of community, perceived cognitive learning, and persistence in asynchronous learning networks'' in Internet and Higher Education 2002 v.5 p.319-332.

Stepich, D. A., & Ertmer, P. A. (2003). ''Building community as a critical element of online course design'' in Educational Technology 2003 v.43 n.5 p.33-43.

Stepich, D. A., Ertmer, P. A., & Lane, M. M. (2001). ''Problem solving in a case-based course: Strategies for facilitating coached expertise'' in Educational Technology Research and Development v.49 n.3 p.53-69.

Swan, K. (2002). ''Learning effectiveness online: What the research tells us'' in J. Bourne and J. C. Moore (Eds.), Elements of quality online education: Practice and direction (Vol 4, Sloan-C Series), pp. 13-46. Needham, MA: Sloan Consortium.

Williams, W. M., and Ceci, S. J. (1997). ''How'm I doing? Problems with student ratings of instructors and courses.'' in Change 1997 v.29 p.12-23.

Wilson, B. G. (2001). Sense of community as a valued outcome for electronic courses, cohorts, and programs. Available online [HREF9].

Hypertext References

HREF1

http://www.edci.purdue.edu

HREF2

http://www.soe.purdue.edu

HREF3

http://www.purdue.edu

HREF4

http://ipt.boisestate.edu/faculty/DStepich.htm

HREF5

http://ipt.boisestate.edu/

HREF6

http://www.boisestate.edu

HREF7

http://www.ascusc.org/jcmc/vol6/issue1/haythornthwaite.html

HREF8

http://www.aln.org/publications/jaln/v2n1/index.asp

HREF9

http://carbon.cudenver.edu/~bwilson/SenseofCommunity.html

Copyright

Peggy A. Ertmer & Donald A. Stepich © 2004. The authors assign to Southern Cross University and other educational and non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The authors also grant a non-exclusive licence to Southern Cross University to publish this document in full on the World Wide Web and on CD-ROM and in printed form with the conference papers and for the document to be published on mirrors on the World Wide Web.

Table 1

Categories Used to Rate Student Case Responses

Table 2

Sense of Community: Correlations with Perceived Learning, and Learning Outcomes

Table 3

Correlations among Measures of Higher-order Thinking

* Significant at the .05 level; ** Significant at the .01 level