2021

Content-Neutral Immersive Environments for Cultivating Scalable Camaraderie

Authors: David A. Joyner, Akhil Mavilakandy, Ishaani Mittal, Denise Kutnick, and Blair MacIntyre

Venue: Learning @ Scale 2021

Download Full Paper

Abstract: Drawbacks of online education, especially at scale, include the isolation and loss of community that come with taking classes from one’s home rather than in a typical coordinated classroom. In this series of studies, we explore the potential of an immersive virtual environment to recreate shared educational contexts in an online program, emphasizing those that focus on universal dynamics like discussions, Q&A, and lecture-watching. Using Mozilla Hubs-which offers virtual reality as well as a browser-based immersive environment for participants without VR headsets-we implement three common, student-driven on-campus environments: a lecture hall, a student lounge, and a poster session. We test these environments with students in an online graduate program through surveys and a controlled experiment. We find significant interest in the potential of these environments, but hurdles to their adoption as well.

With or Without EU: Navigating GDPR Constraints in Human Subjects Research in an Education Environment

Authors: Alex Duncan and David A. Joyner

Venue: Learning @ Scale 2021

Download Full Paper

Abstract: The General Data Protection Regulation (GDPR), passed in 2018, outlined new data privacy standards applying to residents of the European Union (EU). The impact of this law stretches beyond the EU to anyone – such as researchers across the world – collecting or processing data from EU residents. Researchers have had to augment their methodologies to ensure GDPR compliance. This initiative intersects with at-scale educational programs, which often enroll EU students and are also often the subject of institutional research. While creating a study to research students in an online Master of Science in Computer Science (MSCS) program, some of whom are in the EU, we encountered difficulties with ensuring GDPR compliance. This paper discusses the implications of the GDPR related to both general research and our specific study. We discuss the challenges of interpreting the GDPR and integrating it into our methodology as well as potential solutions, our ultimate resolution, and practical recommendations, and we consider what the future of data privacy legislation means for researchers.

Toward Reshaping the Syllabus for Education at Scale

Authors: Bobbie Eicher and David A. Joyner

Venue: Learning @ Scale 2021

Download Full Paper

Abstract: Ensuring that students are fully informed about course content and policies is always a challenge, but online education at scale adds additional complications. In this paper we present observations about the place of the syllabus in education at scale, based on the actual syllabus documents from 48 courses in a Computer Science Master’s degree program offered online and at scale. On the basis of these observations, we offer preliminary recommendations for factors that instructors should keep in mind when they compile a syllabus for similar courses.

Components of Assessments and Grading at Scale

Authors: Bobbie Eicher and David A. Joyner

Venue: Learning @ Scale 2021

Download Full Paper

Abstract: One of the major criticisms of efforts towards offering education at scale has been the Trap of Routine Assessment, the risk that student assessment will suffer from becoming excessively simplified in service of automation and scale. In this research, we examine the ways that students in an at-scale graduate program in computer science were assessed during their degrees. The program in question has scaled to over 10,000 students in only a few years, but awards a traditional Master’s degree, providing the opportunity to investigate whether scale was achieved by transitioning to more routine assessment or by bringing scale to traditional strategies. To do this, we investigate the syllabi of 52 classes offered through the program to identify the types of assessments used, and we survey teaching teams for their approaches to evaluating these assessments. We merge this data with historical enrollment data to gain an overall summary of the kinds of assessments and evaluations received during their degrees. We ultimately find the program’s scale has been managed by scaling up traditional assessment and evaluation strategies as the majority of grades are generated by human teaching teams based on projects and homeworks, with a relatively smaller portion generated exclusively by automated evaluation of exams.

Mechanisms for Supporting Emergency Remote Classes: Towards a Distributed Classroom

Authors: David A. Joyner

Venue: Moving Horizontally: The New Dimensions of At-Scale Learning at the Time of COVID-19

Download Full Paper

Abstract: During the rapid emergency transition to remote classes in 2020, our online Master of Science in Computer Science program supported the newly remote traditional classes in several ways. In this chapter, we go over some of those ways, including providing direct feedback, opening up remote instructional resources, reassigning classes to remote instructors, and providing material for the formation of local cohorts. We then investigate how these mechanisms are small steps toward a broader, more fundamental reimagining of classrooms as distributed across time and space.

Towards Mutual Theory of Mind in Human-AI Interaction: How Language Reflects What Students Perceive About a Virtual Teaching Assistant

Authors: Qiaosi Wang, Koustuv Saha, Eric Gregori, David A. Joyner and Ashok Goel

Venue: CHI 2021

Download Full Paper

Abstract: Building conversational agents that can conduct natural and prolonged conversations has been a major technical and design challenge, especially for community-facing conversational agents. We posit Mutual Theory of Mind as a theoretical framework to design for natural long-term human-AI interactions. From this perspective, we explore a community’s perception of a question-answering conversational agent through self-reported surveys and computational linguistic approach in the context of online education. We first examine long-term temporal changes in students’ perception of Jill Watson (JW), a virtual teaching assistant deployed in an online class discussion forum. We then explore the feasibility of inferring students’ perceptions of JW through linguistic features extracted from student-JW dialogues. We find that students’ perception of JW’s anthropomorphism and intelligence changed significantly over time. Regression analyses reveal that linguistic verbosity, readability, sentiment, diversity, and adaptability reflect student perception of JW. We discuss implications for building adaptive community-facing conversational agents as long-term companions and designing towards Mutual Theory of Mind in human-AI interaction.

2020

The Search for the MOOC Credit Hour

Authors: James J. Lohse, Filipe Altoe, Jasmine Jose, Andrew Nowotarski, Farrukh Rahman, Robert Tuck and David A. Joyner

Venue: Learning with MOOCs 2020

Download Full Paper

Abstract: Over the past decade, MOOCs have risen to prominence in part because most do not carry heavy requirements regarding accreditation, scope, and evaluation. While this has allowed MOOCs to proliferate, it has led to difficulty under-standing what completing a particular MOOC means. Individual MOOCs can vary tremendously in their quantity of content, their depth of assessment, and their mechanisms to ensure academic integrity. In order for MOOC credentials to have value in academia and the workplace, however, there must be some trust in what a particular credential means about the learner. In this work, we perform a study to understand the variety of MOOCs available, then pro¬pose a MOOC Content Matrix as a short, objective way to summarize what a particular MOOC entails and, in turn, what value its certificate gives.

The Synchronicity Paradox in Online Education

Authors: David A. Joyner, Qiaosi Wang, Suyask Thakare, Shan Jing, Ashok Goel and Blair McIntyre

Venue: Learning @ Scale 2020

Download Full Paper

Abstract: As online education proliferates, one concern that has been raised is that it may fail to capture desirable emergent phenomena from on-campus programs. Student community is one example of such a phenomenon: on-campus student communities thrive based on synchronous collocation. An online program might be designed to capture all deliberate constructs in an on-campus program, but there may be beneficial side effects of synchronous collocation that are not apparent. In this work, we examine the issue of social isolation in an online graduate program. By happenstance, three studies were conducted in relative isolation looking at social isolation from different angles. The first study examined trajectories in social presence as a semester proceeded. The second study developed an understanding of students’ needs with regard to community in an online program. The third study tested out an immersive virtual environment to try to improve students’ sense of connectedness. Combining their findings, we find compelling evidence of the existence of a Synchronicity Paradox in online education: students desire synchronicity to form strong social communities, and yet part of the chief appeal of these online pro-grams is their asynchronicity. In light of this finding, we provide design guidelines for how synchronicity may be reintroduced into asynchronous programs without sacrificing the benefits of asynchronicity. More specifically, we propose that scale itself may be the key to building emergent synchronicity.

Sensing Affect to Empower Students: Learner Perspectives on Affect-Sensitive Technology in Large Educational Contexts

Authors: Qiaosi Wang, Shan Jing, David A. Joyner, Lauren Wilcox, Hong Li, Thomas Ploetz and Betsy DiSalvo

Venue: Learning @ Scale 2020

Download Full Paper

Abstract: Large-scale educational settings have been common domains for affect detection and recognition research. Most research emphasizes improvements in the accuracy of affect measurement to enhance instructors’ efficiency in managing large numbers of students. However, these technologies are not designed from students’ perspectives, nor designed for students’ own usage. To identify the unique design considerations for affect sensors that consider student capacities and challenges, and explore the potential of affect sensors to support students’ self-learning, we conducted semi-structured interviews and surveys with both online students and on-campus students enrolled in large in-person classes. Drawing on these studies we: (a) propose using affect data to support students’ self-regulated learning behaviors through a “scaling for empowerment” design perspective, (b) identify design guidelines to mitigate students’ concerns regarding the use of affect data at scale, (c) provide design recommendations for the physical design of affect sensors for large educational settings.

Informal Learning Communities: The Other Massive Open Online ‘C’

Authors: Will Hudgins, Michael Lynch, Ash Schmal, Harsh Sikka, Michael Swenson and David A. Joyner

Venue: Learning @ Scale 2020

Download Full Paper

Abstract: While the literature on learning at scale has largely focused on MOOCs, online degree programs, and AI techniques for supporting scalable learning experiences, informal learning communities have been relatively underrepresented. Nonetheless, these massive open online learning communities regularly draw far more engaged users than the typical MOOC. Their informal structure, however, makes them significantly more difficult to study. In this work, we take a first step toward attempting to understand these communities specifically from the perspective of scale. Taking a sample of 62 such communities, we develop a tagging system for understanding the specific features and how they relate to scale. For example, just as a MOOC cannot manually grade every assignment, so also an informal learning community cannot approve every contribution; and just as MOOCs therefore employ autograding, informal learning communities employ crowd-sourced moderation or plat-form-driven enforcement. Using these tags, we then select several communities for deeper case studies. We also use these tags to make sense of learning-based subreddits from the popular community site Reddit, which offers an API for programmatic analysis. Based on these techniques, we offer findings about the performance of informal learning communities at scale and issue a call to include these environments more fully in future research on learning at scale.

Affordable Degrees at Scale: New Phenomenon or New Hype?

Authors: David S. Park, Robert W. Schmidt, Charankumar Akiri, Stephanie Kwak, David A. Joyner

Venue: Learning @ Scale 2020

Download Full Paper

Abstract: Following the initial proliferation of Massive Open Online Courses (MOOCs), a more recent trend has emerged toward offering “Affordable Degrees at Scale” or “Large, Internet-Mediated Asynchronous Degrees”. In this research, we set out to understand this space: the range in tuition costs for these programs, the variety of admissions standards, and the types of assessments used to evaluate these non-traditional students. In the process, however, we found that in many ways, these programs may not be as new as we initially perceived: similarly-priced online programs have existed from traditional universities for years. In this research, we explore these two questions: what are these new degrees at scale, and how do they actually differ from traditional programs? To explore this, we collected materials for 35 MOOC-based graduate degrees and numerous non-MOOC-based comparable degrees. We then explored the patterns in tuition, admissions requirements, and syllabus information. In this paper, we report the trends we identified in MOOC-based degrees, and attempt to answer the question: what makes these programs different from non-MOOC-based online programs of the past? Ultimately, we find that this new era of programs is similar in many observable ways.

Peripheral and Semi-Peripheral Community: A New Design Challenge for Learning at Scale

Authors: David A. Joyner

Venue: Learning @ Scale 2020

Download Full Paper

Abstract: Existing attempts to foster a greater sense of community in online education have largely focused on direct interactions among students in peer review, forums, and other mechanisms. In this paper, we pose a new design challenge for learning at scale: peripheral community. Peripheral community is the sense of community derived from peripheral interactions in which a student has visibility into others’ behaviors without a direct, intentional interaction occurring between the students. We argue for the value of peripheral community by examining opportunities for such visibility in residential learning environments. We then explore possible ways to supply peripheral community, both in the form of new initiatives and in reinterpretations of existing interventions as fostering peripheral community.

SAGA: Curricula Optimization

Authors: Anneli LeFranc and David A. Joyner

Venue: Learning @ Scale 2020

Download Full Paper

Abstract: This paper presents two approaches using Simulated Annealing and a genetic algorithm to create optimal curricula. The method generates a customized course selection and schedule for individual students enrolled in a large online graduate program in computer science offered by a major public research institution in the United States.

Challenges of Online Learning in Nigeria

Authors: Kabir Abdulmajeed, David A. Joyner and Christine McManus

Venue: Learning @ Scale 2020

Download Full Paper

Abstract: Education has traditionally been administered via physical interactions between teachers and students in classrooms. Through technological advancement in communications and digital devices, online education has been developed with the potential to scale education, making it affordable and accessible. With an internet connection and a laptop or mobile phone, learners can access massive open online courses (MOOCs) for free. Nonetheless, the opportunity to scale education and the advantages of online learning are not always fulfilled due to certain challenges. In this work, Socioeconomic, Sociocultural, and IT infrastructural factors are categorized as challenges hindering the adoption of online learning in Nigeria. Although some factors mitigating online learning have been identified in the past, there is relatively little empirical evidence indicating the reality and severity of these challenges. Since scaling education involves worldwide reach, local contexts such as found in Nigeria and other developing countries become critical. The objective of this work, therefore, is to understand these challenges, present empirical evidence through a questionnaire survey, rank these challenges in order of severity, and propose solutions.

Attitudinal Trajectories in an Online CS1 Class: Demographic and Performance Trends

Authors: David A. Joyner, Lily Bernstein, Maria-Isabelle Dittamo, Ben Engelman, Alysha Naran, Amber Ott, Jasmine Suh and Abby Thien

Venue: Innovation and Technology in Computer Science Education (ITiCSE) 2020

Download Full Paper

Abstract: In this research, we investigate the trajectory of attitudinal change towards computer science among students in an online CS1 class. We perform this investigation to address several trends in modern computer science education. First, as computer science increasingly becomes a required class for all majors, how do students’ first experiences with the subject impact their attitudes? Second, as online education continues to expand, how does enrolling in CS1 online specifically affect audiences that may be marginalized in both CS classes and in online learning environments, such as women and underrepresented minorities? Third, can we intervene to improve attitudes towards computer science, especially among those marginalized audiences? In this research, we poll students in an online for-credit CS1 class four times to observe the change in their attitudes towards computer science over time and intervene with some students to try to improve their perception of computer science. We find that attitudes towards computer science improve with initial exposure, that women’s attitudes towards CS begin less positive but follow the same trajectory, and that mid-semester regression in attitudes toward computer science may predict eventual struggles to perform well in the class.

Jill Watson SA: Design and Evaluation of a Virtual Agent to Build Communities Among Online Learners

Authors:  Qiaosi Wang, Shan Jing, Ida Camacho, David A. Joyner, and Ashok Goel

Venue: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems

Download Full Paper

Abstract: Despite being accessible and affordable, online education presents numerous challenges for online learners due to the absence of face-to-face interactions. Lack of community-belongingness, in particular, negatively impacts online learners’ learning outcomes and learning experience. To help online learners build communities and foster connections with their peers, we designed and deployed Jill Watson SA (stands for Social Agent). Jill Watson SA is a virtual agent who can match students with shared identity, defined by similarities in location, timezone, hobby, class schedule, etc., on the Piazza class discussion forum. We implemented Jill Watson SA in two online classes and conducted three short surveys with online students to evaluate Jill Watson SA.

Enrollment Motivations in an Online Graduate CS Program: Trends & Gender- and Age-Based Differences

Authors: Alex Duncan, Bobbie Eicher and David A. Joyner

Venue: ACM SIGCSE 2020

Download Full Paper

Abstract: Demand for CS education has risen, leading to numerous new programs, such as the rise of affordable online degrees. Research shows these programs meet an otherwise untapped audience of working professionals seeking graduate level CS education. In this study, we examine the motivations for enrollment among students in one such online MSCS program. Based responses to an open ended question, we develop a typology of motivations, including goals (e.g. career transition), opportunities (e.g. enrolling without taking time off work), and assurances that their goals will be met (e..g the program’s accreditation). We then issue a closed survey question to a new group of students to further explore these motivations. In this paper, we discuss both aggregate and demographic trends in motivations, including the different motivations of men and women and what they imply about the program’s impact on the gender divide in computing. We also examine older students’ tendency towards intrinsic motivation to pursue an MSCS degree.

Advising of the Students, by the Students, for the Students: The Case of a Student-Owned Peer Advising Community

Authors: Alex Duncan and David A. Joyner

Venue: Hawaii International Conference on Education 2020

Download Full Paper

Abstract: Peer advising, where students receive advice from other students about courses, can be difficult to study due to its private, spontaneous nature or to potential response bias. However, peer advising can help create and strengthen student communities, and the information students provide can help improve course quality. We first present a case study on a student-run course review website developed to support an online graduate program in computer science. We discuss the evolution of the website and examine its usefulness as a student-organized community providing peer advising at scale. In our second study, we develop a coding scheme and use it to analyze reviews from the website. Although students provide mostly evaluative information, they also provide advice, context for their reviews, course descriptions, and feedback for the instructional team. This research explores the importance of student-organized communities in higher education and provides useful insights into peer advising at scale.

Eroding Investment in Repeated Peer Review: A Reaction to Unrequited Aid?

Authors: David A. Joyner and Alex Duncan

Venue: Hawaii International Conference on Education 2020

Download Full Paper

Abstract: Peer evaluation in educational contexts has been well-researched, but there are open questions about the trends students follow over the course of repeated peer evaluation activities within the same course or semester. We hypothesize a trend whereby reviewers who initially heavily invest in giving their classmates strong peer reviews lower their performance over time due in part to disillusionment with the lower-quality feedback they receive. To test this hypothesis, we investigated a dataset of over 50,000 peer assignment evaluations gathered from three semesters of two different courses, totaling over 79 assignments. We examine whether class performance across four quantitative variables drops over the course of the semester, and whether those drops are specifically more prevalent among high-performing reviewers. We find evidence that reviewers who begin the semester committed suffer a greater drop in
performance over time, and propose potential causal mechanisms for this drop as well as plans to potentially prevent it.

2019

Surveying the MOOC Data Set Universe

Authors: James J. Lohse, Christine A. McManus and David A. Joyner

Venue: 2019 IEEE Learning With MOOCS (LWMOOCS)

Download Full Paper

Abstract: This paper is a survey of the availability of open data sets generated from Massively Open Online Courses (MOOCs). This log data allows researchers to analyze and predict student performance. Often, the goal of the analysis is to focus on at-risk students who are not likely to finish a course. There is a growing gap between the average researcher (who does not have access to proprietary data) and the ready availability of data sets for analysis. Most research papers studying and predicting student performance in MOOCs are done on proprietary data sets that are not anonymized (deidentified) or released for general study. There are no standardized tools that provide a gateway to access usable data sets; instead, the researcher must navigate a maze of sites with different data structures and varying data access policies. To our knowledge, no open data sets are being produced, and have not been since 2016. The authors survey the history of MOOC data sharing, identify the few available open data sets, and discuss a path forward to increase the reproducibility of MOOC research.

Annotation-free Automatic Examination Essay Feedback Generation

Authors: Filipe Altoe and David A. Joyner

Venue: 2019 IEEE Learning With MOOCS (LWMOOCS)

Download Full Paper

Abstract: Many of the demands of scale appear at odds with quality, such as in the case of online MOOC degree programs. This paper presents research that focuses on reducing this dichotomy. Examination essays are important learning tools in for-credit courses. Psychology studies show they drive deeper learning strategies and invoke higher levels of cognitive processing in students. High-quality feedback is another critical aspect of students’ learning in advanced education. Key components of high-quality feedback are how expeditiously feedback is presented and how unambiguous it is. Examination essays are time-consuming for educators to grade, and at scale this may create an incentive for superficial feedback that may also reach students in a less than ideal turnaround time. This imposes clear scalability constraints and a potential decrease in course quality in online for-credit MOOC programs. We propose an annotation-free artificial intelligence-based approach for the automatic generation of examination essay rubrics and its subsequent utilization as part of high-quality feedback to students. It utilizes a combination of the natural language processing techniques TextRank and semantic similarity for automatic rubric generation and concept map creation for feedback presentation. The feedback is immediately available upon essay submission and offers the student a conceptual analysis of the submitted essay against the assignment’s learning objectives.

Creating Cadence: Fostering Persistent Engagement in Asynchronous Online Courses

Authors: David A. Joyner

Venue: Engaged Student Learning: Essays on Best Practices in the University System of Georgia

Download Full Paper

Abstract: The current boon of affordable, scalable online degrees is supported by the modern internet as a medium to construct high-quality, fully asynchronous learning experiences. The asynchronous nature of these programs accommodates working professionals who are unable to carve out consistent and prescribed times for the pursuit of a degree, but who nonetheless have the dedication and ability to succeed at the programs’ content. However, these classes’ asynchronous nature breaks some of students’ assumptions about the structure of college courses. Scheduled meetings do more than just support disseminating lecture material or facilitating synchronous activities; they establish a classroom cadence and set students’ expectations for the pace and routine of the course. Without meetings, these expectations are lost. Moreover, the persistent availability of asynchronous material removes scarcity from the experience, and while this is one of the medium’s strengths, it may also lure students into unhealthy procrastination. Thus, asynchronous online classes must actively accomplish that which traditional classes accomplish passively through required lecture attendance: creating cadence and incentivizing persistent engagement.

Master’s at Scale: Five Years in a Scalable Online Graduate Degree

Authors: David A. Joyner and Charles Isbell

Venue: Learning at Scale 2019

Download Full Paper

Abstract: In 2014, Georgia Tech launched the first for-credit MOOC-based graduate degree program. In the five years since, the program has proven generally successful, enrolling over 14,000 unique students, and several other similar programs have followed in its footsteps. Existing research on the program has focused largely on details of individual classes; program-level research, however, has been scarce. In this paper, we delve into the program-level details of an at-scale Master’s degree, from the story of its creation through the data generated by the program, including the numbers of applications, admissions, matriculations, and graduations; enrollment details including demographic information and retention patterns; trends in student grades and experience as compared to the on-campus student body; and alumni perceptions. Among our findings, we note that the program has stabilized at a retention rate of around 70%; that the program’s growth has not slowed; that the program has not cannibalized its on-campus counterpart; and that the program has seen an upward trend in the number of women enrolled as well as a persistently higher number of underrepresented minorities than the on-campus program. Throughout this analysis, we abstract out distinct lessons that should inform the development and growth of similar programs.

Synchronous at Scale: Investigation and Implementation of a Semi-Synchronous Online Lecture Platform

Authors: Denise Kutnick and David A. Joyner

Venue: Learning at Scale 2019

Download Full Paper

Abstract: Online classes and degree programs continue to grow in popularity, in part due to the increased convenience and accessibility of education that technology has provided in recent years. As online education scales upwards and outwards, there is an increased need to provide students with an engaging and collaborative learning experience. In some online learning environments, student collaboration is perceived to be more difficult than it is in a physical classroom setting due to cultural or geographic distance between students. In particular, online class lectures often lack the collaborative spirit seen in most in-person classroom lectures. To improve upon the online classroom experience, this project first examines the benefits and drawbacks of several in-person and online lecture delivery techniques, then proposes an online lecture platform that allows students to facilitate their own collaborative classrooms on-demand through a semi-synchronous viewing area and chatroom.

Peer Advising at Scale: Content and Context of a Learner-Owned Course Evaluation System

Authors: Alex Duncan and David A. Joyner

Venue: Learning at Scale 2019

Download Full Paper

Abstract: Peer advising in education, which involves students providing fellow students with course advice, can be important in online student communities and can provide insights into potential course improvements. We examine reviews from a course review web site for online graduate programs. We develop a coding scheme to analyze the free text portion of the reviews and integrate those findings with students’ quantitative ratings of each course’s overall score, difficulty, and workload. While reviews focus on subjective evaluation of courses, students also provide feedback for instructors, personal context, advice for other students, and objective course descriptions. Additionally, the average review varies by course overall score, difficulty, and workload. Our research examines the importance of student communities in online education and peer advising at scale.

Designing and Developing Video Lessons for Online Learning: A Seven-Principle Model

Authors: Chaohua Ou, David A. Joyner and Ashok Goel

Venue: Online Learning, 23(2)

Download Full Paper

Abstract: Despite the ubiquitous use of instructional videos in both formal and informal learning settings, questions remain largely unanswered on how to design and develop video lessons that are often used as the primary method for delivering instruction in online courses. In this study, we experimented with a model of seven principles drawn from instructional design theories for designing and developing video lessons for an online graduate course. Feedback was collected from students through surveys on their perceptions of the effectiveness of the video lessons and the overall course quality for eight semesters. This paper shares the instructors’ experience on the design and development of the video lessons as well as the survey findings. Implications of the findings for instructional design and future research are also discussed.

Replicating and Unraveling Performance and Behavioral Differences between an Online and a Traditional CS Course

Authors: David A. Joyner and Melinda McDaniel

Venue: ACM Global Computing Education Conference 2019 (CompEd ’19)

Download Full Paper

Abstract: In January 2017, a major public research university launched an online version of CS1 targeted at on-campus students to address rising enrollments and provide students with flexibility in their schedules. Prior research on this class has found positive outcomes: students in the course achieve the same learning outcomes as those in a traditional course, while reporting a lower time investment to reach those outcomes and a high level of student satisfaction. This research builds on that prior work in two ways. First, it replicates the findings from that earlier semester with an entirely new semester of students. Second, it delves deeper into the student experience within the online course and its traditional counterpart. This deeper analysis focuses specifically on the differing ways in which students in each section allocated their time, whether or not students in either section accessed the opposite section’s material, and their future preferences in online vs. residential CS classes.

Five Years of Graduate CS Education Online and at Scale

Authors: David A. Joyner, Charles Isbell, Thad Starner, and Ashok Goel

Venue: ACM Global Computing Education Conference 2019 (CompEd ’19)

Download Full Paper

Abstract: In 2014, Georgia Tech launched an online campus for its Master of Science in Computer Science program. The degree, equal in stature and accreditation to its on-campus counterpart, offered a notably lower cost of attendance. Its design emphasized flexibility in both geography and time, allowing students from around the world to earn a highly-ranked MSCS without taking time off work or moving to campus. Five years later, the program enrolls over 8000 students per semester and has graduated 1500 alumni. It is believed to be the largest program of its kind and has received recognition from national organizations on professional education. Existing research on the program has focused on challenges and opportunities to scale that are agnostic to the content itself. In this reflection, we look at the creation and growth of the program as it relates to graduate-level CS instruction. In particular, we note a unique and powerful unity of content and platform: the online delivery of the program dovetails with the technical skillsets of the professors and students that it draws, putting both in the position to contribute and innovate.

Building Purposeful Online Learning: Outcomes from Blending CS1

Authors: David A. Joyner

Venue: Blended Learning in Practice

Download Full Paper

Abstract: This chapter describes a blended learning class used to teach a large Computer Science 1 course that serves 1000 students per year and is a bottleneck course that limits how many students can enroll be accepted into computing majors. The course uses a MOOC to provide lectures, practice, and assignments. In addition, students could get help from three sources: recitation with teaching assistants, a help desk, or office hours with the instructor. The chapter discusses the production of the MOOC, which was designed to be adaptable to multiple types of intro computing courses for multiple majors. The design team also surveyed online Intro to Computer Science courses and share the best practices that they experienced.

This course is blended – at least for some students. Based on the MIX taxonomy, the required parts of the course are delivered entirely online. The online components include content delivery and content application with feedback through Vocareum and McGraw-Hill’s adaptive Smartbook. The optional parts of the class provide in-person content delivery and feedback during content application. During recitation, teaching assistants provided lectures and guidance on activities. For additional guidance on activities, the students could go to the help desk or the instructor’s office hours.

The research method is a quasi-experimental design. The two experimental groups, students who took the face-to-face class and students who took the blended class, took the class during the same semester. Student performance was measured through a standardized computer science assessment. Student perceptions, demographic information, and expectations for the course were measured through surveys.

Collaboration versus Cheating: Reducing Code Plagiarism in an Online MS Computer Science Program

Authors: Tony Mason, Ada Gavrilovska and David A. Joyner

Venue: ACM Technical Symposium on Computer Science Education (SIGCSE) 2019

Download Full Paper

Abstract: We outline how we detected programming plagiarism in an introductory online course for a master’s of science in computer science program, how we achieved a statistically significant reduction in programming plagiarism by combining a clear explanation of university and class policy on academic honesty reinforced with a short but formal assessment, and how we evaluated plagiarism rates before and after implementing our policy and assessment.

From Clusters to Content: Using Code Clustering for Course Improvement

Authors: David A. Joyner, Ryan Arrison, Mehnaz Ruksana, Evi Salguero, Zida Wang, Ben Wellington and Kevin Yin

Venue: ACM Technical Symposium on Computer Science Education (SIGCSE) 2019

Download Full Paper

Abstract: Large undergraduate CS courses receive thousands of code submissions per term. To help make sense of the large quantities of submissions, projects have emerged to dynamically cluster student submissions by approach for writing scalable feedback, tailoring hints, and conducting research. However, relatively little attention has been paid to the value of these tools for informing revision to core course materials and delivery methods. In this work, we applied one such technology—Sense, the eponymous product of its company—to an online CS1 class delivered simultaneously for credit to on-campus students and for free to MOOC students. Using Sense, we clustered student submissions to around 70 problems used throughout the course. In this work, we discuss the value of such clustering, the surprising trends we discovered through this process, and the changes made or planned to the course based on the results. We also discuss broader ideas on injecting clustering results into course design.