Goel, A. & Joyner, D. A. (2016). Formative Assessment and Implicit Feedback in Online Learning. In Proceedings of Learning with MOOCs III. Philadelphia, PA.
The importance of formative assessment to effective learning is well known (Black & William 1998). While formative assessment is equally important to effective learning in online education, mechanisms used in residential learning, such as extensive one-on-one expert feedback, do not scale up to MOOCs (Joyner et al. 2016). How, then, can we conduct frequent formative assessment and provide extensive feedback in MOOCs?
In the Georgia Tech online CS7637 class on Knowledge-Based Artificial Intelligence (dubbed “KBAI”; http://www.omscs.gatech.edu/cs-7637-knowledge-based-artificial-intelligence-cognitive-systems; Goel & Joyner 2016), we have deployed several practices for providing frequent and extensive feedback. While some of these practices provide feedback explicitly (for example, through exercises, projects and examinations), the depth of the feedback is modest because of limited resources. Thus, we have also deployed practices for providing implicit feedback, such as having students write self-reflection reports on programming projects and sharing exemplary reports, exercises, and examinations.
Forms of Implicit Feedback
Design-and-programming projects represent an important learning component of the KBAI class. For each project, we first require each student to write a reflection report, encouraging the student to perform a meta-analysis of the project’s goals, designs, strategies, experiments, and outcomes. Second, students participate in peer review throughout the semester, viewing over a dozen randomly selected classmates’ reports. Third, students provide peer review on their classmates’ reports, leveraging the idea of learning by teaching. Fourth, we select the best few reports, obtain permission from the authors of the exemplary reports, and share the exemplary reports with the whole class. We encourage all students to read the exemplary reports in advance of the next project and perform a comparative analysis with their own reports. Thus, students are equipped with cases of both excellent and average work against which to evaluate their own efforts.
Through their use in the KBAI class each term since Fall 2014, we have found that these reflective practices and implicit feedback to be both very scalable and quite effective. They are scalable because they shift part of the responsibility of formative assessment and feedback from the teacher to the student, and thus do not require extensive one-on-one interaction or any additional human effort as class sizes increase. They are effective because they inculcate the practices of self-regulated learning that have been shown to characterize effective learning outcomes (Nicol & MacFarlane-Dick 2006).
The use of implicit assessment and feedback for MOOCs, however, has some tradeoffs. On one hand, the practices of implicit assessment and feedback such as reflection, sharing, comparative analysis, and self-critiquing are scalable and effective. On the other hand, we expect that the students with stronger meta-cognitive skills are more likely to do well in such a learning environment than others. Thus, explicit attention must be paid to help students develop metacognitive ability rather than merely assuming it as a prerequisite.
Black, P., & William, D. (1998) Inside the Black Box: Raising Standards through Classroom Assessment. Phi Delta Kappa.
Goel, A., & Joyner, D. (2016). An Experiment in Teaching Artificial Intelligence Online. International Journal for the Scholarship of Technology-Enhanced Learning 1:1-28.
Joyner, D. A., Ashby, W., Irish, L., Lam, Y., Langston, J., Lupiani, I., Lustig, M., Pettoruto, P., Sheahen, D., Smiley, A., Bruckman, A., & Goel, A. (2016). Graders as Meta-Reviewers: Simultaneously Scaling and Improving Expert Evaluation for Large Online Classrooms. In Proceedings of the Third Annual ACM Conference on Learning at Scale. Edinburgh, Scotland.
Nicol, D., & Macfalane-Dick, D. (2006) Formative Assessment And Self‐Regulated Learning: A Model And Seven Principles Of Good Feedback Practice. Studies in Higher Education 31(2): 199-218.