Metrics for evaluating classroom interactivity in MOOCs

Madden, A. & Joyner, D. A. (2016). Metrics for evaluating classroom interactivity in MOOCs. In Proceedings of Learning with MOOCs III. Philadelphia, PA.

Student engagement with peers and instructors positively impacts learning in the traditional classroom, and preliminary studies suggest that the same is true for MOOCs (Hew 2015, Eynon 2016). In the traditional face-to-face, brick and mortar classroom, instructors often tacitly evaluate the quality of an instructional unit and student engagement based on classroom interaction, such as asking questions and engaging in discussions. Protocols exist for more formal evaluation of these interactions and their impact on learning, such as the Reformed Teaching Observation Protocol (RTOP) (Sawada et al. 2002). Several items on the RTOP capture metrics regarding interactivity within the classroom, both among students and between students and instructors. High performance on these metrics show a high correlation with student success (Piburn & Sawada 2000), suggesting that instructors ought to actively spur classroom interactivity to promote learning outcomes.

Evaluating student engagement, peer interaction, and learning communities is challenging in the traditional classroom, but what about in a MOOC? This paper proposes a new approach to evaluating community interactivity based on the opportunities MOOCs present for new ways of doing learning research. In the traditional classroom, these protocols require considerable effort to administer to evaluate traditional classrooms because the target of the evaluation is verbal and unrecorded. MOOCs, however, offer new types of data which examine similar factors to the RTOP. Online classrooms built around forums take these classroom interactions and capture them in organized, textual data. To both extend these observation protocols to online classrooms and streamline the evaluation of online learning, we have devised metrics for automatically or near-automatically evaluating online classroom community interactivity. In doing so, we observe elements of the online medium, such as the open and asynchronous forum structure, independently support classroom interactivity in ways traditionally reliant on the instructor (Joyner, Goel, & Isbell 2016).

Metrics for Evaluating Interaction

Our metrics attempt to replicate subscales 4 and 5 from the RTOP (Student-Student Interaction and Student Instructor Interaction, respectively), which capture interaction and engagement. We propose deriving the following metrics via quantitative and qualitative analysis, normalized for class size and class duration: quantity of student-initiated discussions; quantity of student direct replies to student-initiated discussions; quantity of instructor direct replies to student-initiated discussions; quantity of instructor-student interactions within replies to other discussions; and quantity of student-student interactions within the replies to instructor-initiated discussions. We propose that this framework for evaluating classroom community interaction is in line with the conclusions derived from the RTOP, and that this framework can be executed automatically or with minimal human analysis in forum-based online courses. We hypothesize, in line with the conclusions of the analysis of the RTOP, that classrooms that experience higher levels of student-student interaction, or classrooms in which instructors participate more often in student discussions, lead to desirable learning outcomes. We also propose that the same holds true in MOOCs, and thus learning in MOOCs can be evaluated using this methodology with some success. We lastly hypothesize that this methodology will elucidate inherent affordances of online courses that lend themselves to greater classroom interactivity.


Eynon, R., Hjorth, I., Yasseri, T., & Gillani, N. (2016). How can social interaction support learning in a MOOC?. Policy2014, 2012.

Hew, K. F. (2016). Promoting engagement in online courses: What strategies can we learn from three highly rated MOOCS. British Journal of Educational Technology, 47(2), 320-341.

Joyner, D. A., Goel, A., & Isbell, C. (2016). The Unexpected Pedagogical Benefits of Making Higher Education Accessible. In Proceedings of the Third Annual ACM Conference on Learning at Scale. Edinburgh, Scotland.

Piburn, M., & Sawada, D. (2000). Reformed Teaching Observation Protocol (RTOP) Reference Manual. Technical Report.

Sawada, D., Piburn, M. D., Judson, E., Turley, J., Falconer, K., Benford, R., & Bloom, I. (2002). Measuring reform practices in science and mathematics classrooms: The reformed teaching observation protocol. School Science and Mathematics, 102(6), 245-253.

Full Version

Leave a Reply