Article: Reay, Li, and Bao (2008)
The news article I mentioned in my last post referred to a recent Ohio State study on the impact of classroom response system use in physics courses. Since this study has received other press, I thought I would comment on it here.
Reference: Reay, N. W., Li, Pengfei, & Bao, Lei. (2008). Testing a new voting machine methodology. American Journal of Physics, 72(2), 171-178.
Summary: This study examines the effect of the use of classroom response systems on student performance in introductory electricity and magnetism courses. For three consecutive semesters, student performance, as measured by results on common exam questions and on a particular concept inventory (the Conceptual Survey of Electricity and Magnetism), was compared between two sections, one in which classroom response systems were used and one in which they were not used.
For the authors, use of clickers typically involved question sequences. Each question sequence typically focused on a single concept explored in multiple contexts and was this designed to promote transfer of knowledge between contexts. The authors describe most of their question sequences as either “easy-difficult-difficult” sequences or “rapid-fire” sequences. The latter type featured relatively easy questions that students were required to answer quickly. The authors include examples of each type of question sequence.
Students in clicker sections answered 72%, 68%, and 63% of common exam questions correctly over the three semesters. Students in non-clicker sections answered 64%, 56%, and 52% of exam questions correctly, yielding a difference of 8%, 11%, and 11% between the two sections and indicating that student performance was improved by the use of clicker question sequences.
The concept inventory used was administered at the start and end of each semester. During the first and third semesters, students in the clicker sections performed better on the post-test than students in the non-clicker sections. The differences were statistically significant with p-values of 0.005 and 0.009, respectively. During the second semester, there was not a statistically significant difference between the two sections, although attendance in the second semester clicker section was low, averaging around 50%. (Normalized gains between the pre-tests and post-tests were higher in the clicker sections for all three semesters, although the authors didn’t compute these gains or associated p-values.)
The authors also note that the gain from pre-test to post-test on the concept inventory was statistically the same for male and female students in the clicker sections. However, the gain for male students was statistically greater than the gain for female students in the non-clicker sections. The authors conclude that “the voting machines reduced the gap between male and female student performances on tests.” They note that they hope to explore this finding further in future research.
Student response to the use of clickers was generally very positive on end-of-semester surveys. The responses were not as positive during the third semester, and the authors hypothesize that these results might be due to overuse of clicker questions during that semester or to the fact that clicker questions that semester were “lightly graded based on attendance.”
Comments: These results are very persuasive, and the finding about gender differences is particularly encouraging. The authors did a great job of explaining their research and teaching methods and the complications that arose in their study. The example question sequences, of both the “easy-difficult-difficult” and “rapid-fire” types, the authors included are appreciated, since they give a concrete sense of what kinds of clicker questions were used in the courses. As a result, the authors provide a useful description of how clicker questions can be used to promote transfer of knowledge between contexts, an important but difficult-to-achieve learning goal.
My main concern with this study is that it doesn’t seem to assess the impact of the use of classroom response systems as much as it seems to assess the impact of an entire pedagogy, one that happens to use classroom response systems. The “control” sections not only didn’t use clickers, but it’s unclear if they used any version of the pedagogical approach used in the “experimental” sections to promote transfer of knowledge. The authors note that “non-voting machine lecturers had access to question sequence material, but otherwise taught in a traditional manner.” No details are provided on what a “traditional manner” means in this instance.
As a result, I think this study rather successfully argues that a pedagogy involving (a) question sequences designed to promote transfer, (b) variations on the standard peer instruction method, and (c) classroom response systems can be a very effective pedagogy. That’s actually a great result and I hope that instructors in physics and other disciplines pay attention to it.
However, these findings don’t isolate the impact of the clicker technology itself. For instance, what about the fact that the technology allows all students to respond to questions independently and be held accountable for responding? Or the fact that instructors can view results of questions during class and make “on the fly” teaching decisions based on those results? Or the fact that students can see where they stand in relation to their peers via the results display? These are properties of the technology itself, and it’s unclear from this particular study what role they play in student learning.