A Crowdsourced Rubric for Evaluating Infographics

I’m teaching a statistics course this spring. It’s the fourth time I’ve taught this course, and, in an effort to make it more interesting to me and more useful for my students, I added a module on data visualization to the course this time around. I find that the data visualization chapters in most statistics textbooks look like they could have been written in the 1950s, featuring stem-and-leaf plots, box-and-whisker plots, and other very analog visualization tools. Modern data visualization tools are much more sophisticated than these, and, more importantly, these modern tools are needed to handle the multi-dimensional data sets that digital tools now make available. Check out, for instance, this bubble chart displaying executive salaries or this live heatmap of tweets made around the world or this interactive treemap of earthquakes since 1900.

I’m not expecting my students to create data visualizations as slick as those examples, but I do want them to think outside the boxplot. For their final project in the course, I’m asking them to apply the statistical techniques they’ve learned to some “real world” problem and communicate their findings through an infographic. (What are infographics? See these examples.) I’ve given students a version of this assignment in the past, but back then I asked them to write up their findings in five-page papers. Although these students (most of whom are engineers) are likely to write reports in their future careers, I think that having them design infographics will give them a valuable opportunity to learn how to communicate statistical data visually, which I believe to be an increasingly important skill for those working in quantitative fields. Even so, I was on the fence regarding this assignment until I learned that Sidneyeve Matrix, a media and film professor at Queen’s University, gave her students a similar assignment recently. You can read my assignment here, and Sidneyeve’s assignment here.

To prepare my students for their infographic assignment, I first asked them to find examples of well-design infographics as part of their social bookmarking assignments in the course. I asked asked them to read “Ending the Infographic Plague” by Atlantic editor Megan McArdle. I figured the students would need a bit more instruction about this genre, so I used a technique I first used years ago while helping an engineering professor teach his students how to conduct effective oral presentations: I asked students to crowdsource a rubric for evaluating infographics.

Step 1: First I showed students a few of the infographics they had found for the social bookmarking assignment. I asked the students who bookmarked these infographics to share reasons they felt the infographics were effective. I added a few comments, and, between us, we surfaced several aspects of effective infographic design.

Step 2: Then I shared with my students part of the rubric I had used in the past for this application project. Each row of this rubric focused on a different aspect of the content of the project: questions, methods, assumptions, computations. For each of these components, the rubric included descriptors of work that was of poor, acceptable, good, or excellent quality. (Rubrics with this kind of structure are called analytic rubrics, in contrast to holistic rubrics which don’t have categories and instead have one descriptor for each grade an assignment might receive.) Sharing this old rubric helped me communicate my expectations for this assignment to my students, and gave them some understanding of how rubrics can be constructed.

Step 3: I asked students brainstorm additional rows for this rubric, ones that focused on the communication of statistical results through infographics, using the discussion from Step 1 as inspiration. Here’s where the crowdsourcing came in: I encouraged students to add their ideas to this shared Google doc, synchronously during class. Here’s where I had a crowdsourcing #fail: I didn’t ask my students to bring their laptops to class this day. Maybe a dozen of my students were able to contribute directly to the Google doc. The rest turned in their rubric ideas on paper. That was fine for them, but it meant I ended up typing a lot of their contributions into the Google doc after class. Next time I try something like this, I’ll make sure to have students bring their mobile devices to class!

Step 4: After looking through all the students contributions after class, I was able to group most of them into eight different categories: purpose, organization, data visualization, use of charts, use of color, use of text, sources, and aesthetics. For each category I selected a student who had submitted a set of descriptors for that category that were at least pretty good, then copied those eight sets of descriptors into a draft infographic rubric that I shared with the class the next day. I made clear to them that this was only a draft, that I wouldn’t use it as-is to grade their final projects, and that some of the descriptors might be better than others.

Step 5: Here’s where my experience teaching a writing seminar paid off. I distributed hard copies of two infographics from those bookmarked by my students (one on robots in the workplace and one on shoplifting around the world), along with two blank copies of the draft infographic rubric. I asked students to work in groups and evaluate the two sample infographics using the rubric, marking each infographic as poor, acceptable, good, or excellent in each category. Once they had done so, I aggregated their evaluations using a series of clicker questions such as “How did you evaluate the robot infographics’ use of color?” For each of these questions, I asked students to provide reasons for their evaluations, given us a chance to discuss various aspects of effective infographic design and have something of a shared understanding of expectations for their final projects.

Note: I’ve exaggerated a bit on Step 5 there. That was what I had planned for class on Monday since that’s more or less what happened when I used this rubric-and-clickers activity in my writing seminar, but we only had time to look at two aspects (purpose and use of color) of one of the two sample infographics. I’ll continue this process in class today, hopefully getting to that bit about a shared understanding. Step 6 will then involve me revising that draft infographic rubric, using feedback from the students generated during Step 5, and sharing it with the students as the grading rubric for their final projects.

I’ll conclude by saying that I really like the idea of having students contribute to the rubric used to evaluate their work. I think doing so gives students more investment in the assignment and a clearer understanding of the standards of quality in a particular domain. I bet they learn as much from each other (hearing other students’ suggestions for the rubric) as they do from the instructor. This strategy seems particularly useful when the genre of the project is a little weird (like an infographic), but I think it’s also useful with more traditional assignments (like five-paper papers) because students often don’t understand the standards of quality in our disciplines as much as we’d like to think they do.

Image: “The largest QOTSA headline crowd to date,” Matthew Field, Flickr (CC)