Abstract: The Trap of Routine Assessment refers to the risk that over time, schools become overly focused on those skills that are easy to automatically assess in order to achieve large-scale, rapid feedback. In this research, we look at an at-scale graduate program in computer science that has achieved scale, and we examine whether scale was largely achieved due to Routine Assessment, due to sophisticated automatic evaluation, or due to scaling more traditional approaches to assessment, such as human grading. To do this, we harvested syllabi from 52 classes in the program to see what types of assignments students complete as part of their enrollment, and we survey the teaching assistants responsible for administering those courses to understand how those assignments are evaluated. Combining this with historical data on course grades and enrollments, we derive a snapshot of the average student’s experience in the program, what assignments they complete, and where their evaluation comes from. In the end, we note that the program’s scale has largely been achieved by streamlining the ability to generate more traditional assessment and evaluation, with the large fraction of student work continuing to be open-ended and human-evaluated.

Abstract: Proponents of at-scale education initiatives often tout the benefits of these programs to the students who enroll in them, but anecdotally, faculty who participate in creating at-scale courses have often commented on how their participation positively impacted their subsequent in-person teaching. In this study, we investigate this potential phenomenon more thoroughly to assess how generalizable this perspective is among participants in these initiatives. We conduct three studies: we interview 78 faculty teaching in at-scale degree programs, examine offering histories for 70 courses offered in one such program, and survey 153 faculty who have developed massive open online courses. Based on these three studies, we propose a list of ways in which in-person instruction benefits from the existence of at-scale teaching initiatives. We further discuss remaining perceived drawbacks of at-scale education as derived from these interviews and surveys.

Abstract: Research on how students prepare for graduate computer science programs typically focuses on single, subject-specific interventions or relates to preparation for life as a graduate student. Preparatory work completed prior to enrolling in such a program can be particularly important for underrepresented minorities and those without technical backgrounds. We use
survey data from incoming students in a large online graduate computer science program to answer three research questions: What are the backgrounds of students entering the program? How do students prepare for the program? And how does student preparation differ based on demographics and prior experience? We find that: male students are more likely than female students to enter the program with computer science qualifications; older students, female students, and those with non-technical degrees are more likely to pursue preparation; and students with no online learning experience are less likely to pursue preparation. These findings highlight the importance of student backgrounds when creating preparatory courses and indicate the value of preparatory courses in increasing diversity in large online graduate programs.

Abstract: Using the Online Master of Science in Computer Science program in the College of Computing at Georgia Tech as a case study, this paper explores the ways in which student life initiatives can humanize the online student experience at scale through a holistic and student care-centric approach. By analyzing ten student life initiatives from the program, we highlight ways to ensure belonging, facilitate engagement, and create community at scale and mitigate the isolation and social disconnectedness students may feel in large online programs.

Abstract: We examine trends in the Learning at Scale conference from 2014 through 2021. We use an original coding scheme to classify all 142 full papers from five angles: setting, approach, pedagogical strategy, population of interest, and dependent variable. We observe a decline of research on MOOCs, an increase in number of settings studied over time, and a consistent focus on assessment strategies. We then examine other conferences to which Learning at Scale authors contribute research. This paper contributes an original coding scheme to classify future Learning at Scale papers; an analysis of the research focuses at Learning at Scale over time; and an evaluation of the mutual influence between Learning at Scale and other venues. These latter two contributions contextualize learning at scale research within Learning at Scale and the broader research community to show how the field has evolved and to help predict its future directions.

Abstract: The emerging field of affordable degrees at scale relies in large part on a linear relationship between the number of students who enroll and the number of teaching assistants who are employed. As programs grow, however, identifying good candidates can become untenable: research has found that the fraction of students who apply for TA roles far exceeds the number of actual available roles, forcing instructors to comb through hundreds of applications for a tiny number of positions. Most of these applicants are themselves former students in the class as well, meaning that there is abundant data from their assignments, grades, peer review behaviors, and forum participation to evaluate candidates, but navigating and using this data requires significant time. In this work, we utilize machine learning to predict students’ aptitudes for teaching assistant roles. We train a neural network model on the available data and aggregate meaningful features to determine if a student possesses the qualities associated with being a good TA. This paper covers implementation details of the training dataset, neural network model training, model result and analysis. We also present a verification method and suggest future improvements.

Abstract: Online education is convenient and accessible, but also brings new challenges to retention. Prior research has found that students in online programs—including CS programs—are more likely to withdraw both from individual classes as well as the program as a whole than students in traditional programs. In this study, we delve deeper into retention at the level of individual courses. We analyze three courses offered as part of a large online CS graduate degree program taught at a major research university in the United States. We obtained voluntary data from students across 12 semesters—including gender, prior CS experience, and anticipated workload for the course—and analyzed these variables in the context of course retention. While the average course drop rate for the examined courses in the program was 7%, variations in this rate could be predicted by a number of different attributes: for instance, we observed that women, native English speakers, and students with less prior education were all more likely to complete a course that they had started. Importantly, these predictors can be measured before the class begins, supporting early intervention.

Abstract: A/B experiments are commonly used in research to compare the effects of changing one or more variables in two different experimental groups—a control group and a treatment group. While the benefits of using A/B experiments are widely known and accepted in education, there is less agreement on an approach to creating software infrastructure systems to assist in rapidly conducting such experiments in the field. To assist in alleviating this gap, we are creating a software infrastructure for A/B experiments that allows researchers to conduct experiments and automatically analyze their results for an education-focused ecology-based conceptual modeling platform.

Abstract: ChatGPT has taken the world by storm, with educators reeling from its implication for curricula and assessment. This article examines how ChatGPT resembles earlier technologies and predicts how we can expect it to impact education going forward.

“ChatGPT brings with it a new set of skills to be learned, and along with those skills the possibility to elevate our expectations for how well students fulfill our learning goals.”

Abstract: Since their inception, MOOCs have had a complicated relationship with traditional models of high-stakes, high-value college credit. Early MOOCs were modeled after for-credit courses but were required to be deliberately differentiated from actual for-credit enrollment. Later MOOCs actively separated from for-credit classes, leaving more leeway to experiment with topic, scope, and audience, albeit without the incentive introduced by a connection to a college curriculum. Since then, though, there have been several efforts to relink MOOCs to some form of heavier credential. In this paper, we take inventory of the current landscape of MOOCs and their varying links to college credit. We draw on the idea of credit substitutability as a way of understanding how far from credit a particular course may be. We articulate multiple factors that contribute to substitutability, including scope, assessment, and integrity. Using these factors, we illustrate a spectrum of credit substitutability in MOOCs that includes large classes in affordable degrees at scale; MOOCs with attached mechanisms for credit exchange like MicroMaster’s programs, MOOCs that build on content used for for-credit experiences; and MOOCs offered through international platforms with a clearer focus on inter-university credit exchange.

Abstract: “MOOC-based” degrees are degree programs often offered in partnership with MOOC providers that provide the flexibility and scale of MOOCs while also awarding accredited degrees. This positioning between MOOCs and degrees raises interesting questions regarding retention: MOOCs are famous for their low completion rates, but accredited degree programs often strive for high retention rates. This paper aims to answer the broad question: what does retention look like in a “MOOC-based” degree program? To answer this question, we analyze retention at two levels: first at the program level, then at the course level. We find that retention is far higher than in MOOCs, but notably lower than in traditional in-person programs, both when looking at the program as a whole and at individual courses. We provide discuss several hypotheses for this phenomenon, as well as implications for program evaluation and course design.

Abstract: Past research on the ways that students evaluate their courses has focused largely on how those evaluations relate to the specific course instructor. This research examines a set of data from a public site where students unofficially rate the courses in a very large online graduate program operating at scale. We examine the relationship between the unofficial scores students give to their classes with data on enrollment trends over time and the assessment strategies used within the courses themselves to examine additional actors that shape the ratings students choose, as well as how they use those ratings to choose what courses to take in the future. We find several different notable relationships: reviews in this context are largely impervious to the extreme response bias prevalent on other review sites; review content does not appear to significantly influence enrollment trends; more difficult classes tend to receive more favorable ratings overall, although individual students do not rate difficult classes more favorably; and project-based classes are perceived by students to be less difficult.

Abstract: Students have developed their own platforms for sharing their evaluations of courses over the Internet. These are typically focused on the needs and experience of students in traditional undergraduate programs, but the rise of online programs operating at scale has made it practical for students to develop such a platform dedicated to their particular program. We have used a survey to gather information from students in such a program at a major research institution in the United States. Through this data we explore how many students are using the site, how they use the information, and also how often and why they write reviews. The ultimate goal is to gather information that could help students to decide how to critically assess such reviews and successfully use them to make better decisions.

Abstract: Plagiarism is a very serious offence in academic institutions. Yet there is some reluctance to address plagiarism by educators as its enforcement can require a significant time commitment if not handled wisely. Handling plagiarism at scale has the potential to exacerbate this problem. This article explores the challenges educators face when it comes to enforcing plagiarism from within the course environment, presents a solution that addresses these challenges, and provides guidelines for preventing and detecting plagiarism in our at‐scale courses. A workflow is shared which was created from our experience of handling plagiarism within our large online computer science graduate level program. We present an empirical study of this workflow that shows the overall prevalence of misconduct. Results demonstrate this workflow being effectively and efficiently applied to address plagiarism across four courses and three semesters by a single individual working half time. We have observed that warnings on low-stakes assignments can be very effective at deterring future misconduct. We have also observed that cases come to a quicker resolution when students readily acknowledge their own misconduct after being promptly notified. Plagiarism should not only be managed through institution wide policies, but also through effective strategies implemented in the course environment. This is particularly important when handling plagiarism at scale. Addressing plagiarism starts in the course and therefore effective strategies for handling it there are needed.

Abstract: It is important for institutions of higher education to maintain academic integrity, both for students and the institutions themselves. Proctoring is one way of accomplishing this, and with the increasing popularity of online courses—along with the sudden shift to online education sparked by the COVID-19 pandemic—digital proctoring has seen an increase in use. However, there are privacy and bias concerns related to digital proctoring, so it is important to critically examine its role in higher education—when it should and should not be used, and how it is perceived among those who interact with it. In this paper, we: examine the features of and concerns about digital proctoring; analyse the results of a survey regarding student and teaching assistant (TA) attitudes towards digital proctoring; and present alternatives to digital proctoring and a framework for evaluating the need for a digital proctoring tool.

Abstract: The term “double-edged sword” refers to something that may have both favorable and unfavorable consequences. We posit that allowing students to post anonymously in a CS course forum may fit this metaphor with regard to gender and belongingness. In this work, we test a theory that patterns of anonymous posting in a course forum for a CS1 class may reinforce gender stereotypes even as the underlying patterns of interaction debunk those stereotypes. We examine forum interactions from a CS1 class with an even gender split and find that women engage in anonymous posting more often than men; thus, a student’s view of the class’s gender distribution is different from the actual distribution. We hypothesize this is a missed opportunity to combat stereotypes of gender in computer science. Possible solutions and further work are discussed.

Abstract: Drawbacks of online education, especially at scale, include the isolation and loss of community that come with taking classes from one’s home rather than in a typical coordinated classroom. In this series of studies, we explore the potential of an immersive virtual environment to recreate shared educational contexts in an online program, emphasizing those that focus on universal dynamics like discussions, Q&A, and lecture-watching. Using Mozilla Hubs-which offers virtual reality as well as a browser-based immersive environment for participants without VR headsets-we implement three common, student-driven on-campus environments: a lecture hall, a student lounge, and a poster session. We test these environments with students in an online graduate program through surveys and a controlled experiment. We find significant interest in the potential of these environments, but hurdles to their adoption as well.

Abstract: Drawbacks of online education, especially at scale, include the isolation and loss of community that come with taking classes from one’s home rather than in a typical coordinated classroom. In this series of studies, we explore the potential of an immersive virtual environment to recreate shared educational contexts in an online program, emphasizing those that focus on universal dynamics like discussions, Q&A, and lecture-watching. Using Mozilla Hubs-which offers virtual reality as well as a browser-based immersive environment for participants without VR headsets-we implement three common, student-driven on-campus environments: a lecture hall, a student lounge, and a poster session. We test these environments with students in an online graduate program through surveys and a controlled experiment. We find significant interest in the potential of these environments, but hurdles to their adoption as well.

Abstract: The General Data Protection Regulation (GDPR), passed in 2018, outlined new data privacy standards applying to residents of the European Union (EU). The impact of this law stretches beyond the EU to anyone – such as researchers across the world – collecting or processing data from EU residents. Researchers have had to augment their methodologies to ensure GDPR compliance. This initiative intersects with at-scale educational programs, which often enroll EU students and are also often the subject of institutional research. While creating a study to research students in an online Master of Science in Computer Science (MSCS) program, some of whom are in the EU, we encountered difficulties with ensuring GDPR compliance. This paper discusses the implications of the GDPR related to both general research and our specific study. We discuss the challenges of interpreting the GDPR and integrating it into our methodology as well as potential solutions, our ultimate resolution, and practical recommendations, and we consider what the future of data privacy legislation means for researchers.

Abstract: Ensuring that students are fully informed about course content and policies is always a challenge, but online education at scale adds additional complications. In this paper we present observations about the place of the syllabus in education at scale, based on the actual syllabus documents from 48 courses in a Computer Science Master’s degree program offered online and at scale. On the basis of these observations, we offer preliminary recommendations for factors that instructors should keep in mind when they compile a syllabus for similar courses.

Abstract: One of the major criticisms of efforts towards offering education at scale has been the Trap of Routine Assessment, the risk that student assessment will suffer from becoming excessively simplified in service of automation and scale. In this research, we examine the ways that students in an at-scale graduate program in computer science were assessed during their degrees. The program in question has scaled to over 10,000 students in only a few years, but awards a traditional Master’s degree, providing the opportunity to investigate whether scale was achieved by transitioning to more routine assessment or by bringing scale to traditional strategies. To do this, we investigate the syllabi of 52 classes offered through the program to identify the types of assessments used, and we survey teaching teams for their approaches to evaluating these assessments. We merge this data with historical enrollment data to gain an overall summary of the kinds of assessments and evaluations received during their degrees. We ultimately find the program’s scale has been managed by scaling up traditional assessment and evaluation strategies as the majority of grades are generated by human teaching teams based on projects and homeworks, with a relatively smaller portion generated exclusively by automated evaluation of exams.

Abstract: During the rapid emergency transition to remote classes in 2020, our online Master of Science in Computer Science program supported the newly remote traditional classes in several ways. In this chapter, we go over some of those ways, including providing direct feedback, opening up remote instructional resources, reassigning classes to remote instructors, and providing material for the formation of local cohorts. We then investigate how these mechanisms are small steps toward a broader, more fundamental reimagining of classrooms as distributed across time and space.

Abstract: Building conversational agents that can conduct natural and prolonged conversations has been a major technical and design challenge, especially for community-facing conversational agents. We posit Mutual Theory of Mind as a theoretical framework to design for natural long-term human-AI interactions. From this perspective, we explore a community’s perception of a question-answering conversational agent through self-reported surveys and computational linguistic approach in the context of online education. We first examine long-term temporal changes in students’ perception of Jill Watson (JW), a virtual teaching assistant deployed in an online class discussion forum. We then explore the feasibility of inferring students’ perceptions of JW through linguistic features extracted from student-JW dialogues. We find that students’ perception of JW’s anthropomorphism and intelligence changed significantly over time. Regression analyses reveal that linguistic verbosity, readability, sentiment, diversity, and adaptability reflect student perception of JW. We discuss implications for building adaptive community-facing conversational agents as long-term companions and designing towards Mutual Theory of Mind in human-AI interaction.

Abstract: Over the past decade, MOOCs have risen to prominence in part because most do not carry heavy requirements regarding accreditation, scope, and evaluation. While this has allowed MOOCs to proliferate, it has led to difficulty under-standing what completing a particular MOOC means. Individual MOOCs can vary tremendously in their quantity of content, their depth of assessment, and their mechanisms to ensure academic integrity. In order for MOOC credentials to have value in academia and the workplace, however, there must be some trust in what a particular credential means about the learner. In this work, we perform a study to understand the variety of MOOCs available, then pro¬pose a MOOC Content Matrix as a short, objective way to summarize what a particular MOOC entails and, in turn, what value its certificate gives.

Abstract: As online education proliferates, one concern that has been raised is that it may fail to capture desirable emergent phenomena from on-campus programs. Student community is one example of such a phenomenon: on-campus student communities thrive based on synchronous collocation. An online program might be designed to capture all deliberate constructs in an on-campus program, but there may be beneficial side effects of synchronous collocation that are not apparent. In this work, we examine the issue of social isolation in an online graduate program. By happenstance, three studies were conducted in relative isolation looking at social isolation from different angles. The first study examined trajectories in social presence as a semester proceeded. The second study developed an understanding of students’ needs with regard to community in an online program. The third study tested out an immersive virtual environment to try to improve students’ sense of connectedness. Combining their findings, we find compelling evidence of the existence of a Synchronicity Paradox in online education: students desire synchronicity to form strong social communities, and yet part of the chief appeal of these online pro-grams is their asynchronicity. In light of this finding, we provide design guidelines for how synchronicity may be reintroduced into asynchronous programs without sacrificing the benefits of asynchronicity. More specifically, we propose that scale itself may be the key to building emergent synchronicity.

Abstract: Large-scale educational settings have been common domains for affect detection and recognition research. Most research emphasizes improvements in the accuracy of affect measurement to enhance instructors’ efficiency in managing large numbers of students. However, these technologies are not designed from students’ perspectives, nor designed for students’ own usage. To identify the unique design considerations for affect sensors that consider student capacities and challenges, and explore the potential of affect sensors to support students’ self-learning, we conducted semi-structured interviews and surveys with both online students and on-campus students enrolled in large in-person classes. Drawing on these studies we: (a) propose using affect data to support students’ self-regulated learning behaviors through a “scaling for empowerment” design perspective, (b) identify design guidelines to mitigate students’ concerns regarding the use of affect data at scale, (c) provide design recommendations for the physical design of affect sensors for large educational settings.

Abstract: While the literature on learning at scale has largely focused on MOOCs, online degree programs, and AI techniques for supporting scalable learning experiences, informal learning communities have been relatively underrepresented. Nonetheless, these massive open online learning communities regularly draw far more engaged users than the typical MOOC. Their informal structure, however, makes them significantly more difficult to study. In this work, we take a first step toward attempting to understand these communities specifically from the perspective of scale. Taking a sample of 62 such communities, we develop a tagging system for understanding the specific features and how they relate to scale. For example, just as a MOOC cannot manually grade every assignment, so also an informal learning community cannot approve every contribution; and just as MOOCs therefore employ autograding, informal learning communities employ crowd-sourced moderation or plat-form-driven enforcement. Using these tags, we then select several communities for deeper case studies. We also use these tags to make sense of learning-based subreddits from the popular community site Reddit, which offers an API for programmatic analysis. Based on these techniques, we offer findings about the performance of informal learning communities at scale and issue a call to include these environments more fully in future research on learning at scale.

Abstract: Following the initial proliferation of Massive Open Online Courses (MOOCs), a more recent trend has emerged toward offering “Affordable Degrees at Scale” or “Large, Internet-Mediated Asynchronous Degrees”. In this research, we set out to understand this space: the range in tuition costs for these programs, the variety of admissions standards, and the types of assessments used to evaluate these non-traditional students. In the process, however, we found that in many ways, these programs may not be as new as we initially perceived: similarly-priced online programs have existed from traditional universities for years. In this research, we explore these two questions: what are these new degrees at scale, and how do they actually differ from traditional programs? To explore this, we collected materials for 35 MOOC-based graduate degrees and numerous non-MOOC-based comparable degrees. We then explored the patterns in tuition, admissions requirements, and syllabus information. In this paper, we report the trends we identified in MOOC-based degrees, and attempt to answer the question: what makes these programs different from non-MOOC-based online programs of the past? Ultimately, we find that this new era of programs is similar in many observable ways.

Abstract: Existing attempts to foster a greater sense of community in online education have largely focused on direct interactions among students in peer review, forums, and other mechanisms. In this paper, we pose a new design challenge for learning at scale: peripheral community. Peripheral community is the sense of community derived from peripheral interactions in which a student has visibility into others’ behaviors without a direct, intentional interaction occurring between the students. We argue for the value of peripheral community by examining opportunities for such visibility in residential learning environments. We then explore possible ways to supply peripheral community, both in the form of new initiatives and in reinterpretations of existing interventions as fostering peripheral community.

Abstract: This paper presents two approaches using Simulated Annealing and a genetic algorithm to create optimal curricula. The method generates a customized course selection and schedule for individual students enrolled in a large online graduate program in computer science offered by a major public research institution in the United States.

Abstract: Education has traditionally been administered via physical interactions between teachers and students in classrooms. Through technological advancement in communications and digital devices, online education has been developed with the potential to scale education, making it affordable and accessible. With an internet connection and a laptop or mobile phone, learners can access massive open online courses (MOOCs) for free. Nonetheless, the opportunity to scale education and the advantages of online learning are not always fulfilled due to certain challenges. In this work, Socioeconomic, Sociocultural, and IT infrastructural factors are categorized as challenges hindering the adoption of online learning in Nigeria. Although some factors mitigating online learning have been identified in the past, there is relatively little empirical evidence indicating the reality and severity of these challenges. Since scaling education involves worldwide reach, local contexts such as found in Nigeria and other developing countries become critical. The objective of this work, therefore, is to understand these challenges, present empirical evidence through a questionnaire survey, rank these challenges in order of severity, and propose solutions.

Abstract: In this research, we investigate the trajectory of attitudinal change towards computer science among students in an online CS1 class. We perform this investigation to address several trends in modern computer science education. First, as computer science increasingly becomes a required class for all majors, how do students’ first experiences with the subject impact their attitudes? Second, as online education continues to expand, how does enrolling in CS1 online specifically affect audiences that may be marginalized in both CS classes and in online learning environments, such as women and underrepresented minorities? Third, can we intervene to improve attitudes towards computer science, especially among those marginalized audiences? In this research, we poll students in an online for-credit CS1 class four times to observe the change in their attitudes towards computer science over time and intervene with some students to try to improve their perception of computer science. We find that attitudes towards computer science improve with initial exposure, that women’s attitudes towards CS begin less positive but follow the same trajectory, and that mid-semester regression in attitudes toward computer science may predict eventual struggles to perform well in the class.

Abstract: Despite being accessible and affordable, online education presents numerous challenges for online learners due to the absence of face-to-face interactions. Lack of community-belongingness, in particular, negatively impacts online learners’ learning outcomes and learning experience. To help online learners build communities and foster connections with their peers, we designed and deployed Jill Watson SA (stands for Social Agent). Jill Watson SA is a virtual agent who can match students with shared identity, defined by similarities in location, timezone, hobby, class schedule, etc., on the Piazza class discussion forum. We implemented Jill Watson SA in two online classes and conducted three short surveys with online students to evaluate Jill Watson SA.

Abstract: Demand for CS education has risen, leading to numerous new programs, such as the rise of affordable online degrees. Research shows these programs meet an otherwise untapped audience of working professionals seeking graduate level CS education. In this study, we examine the motivations for enrollment among students in one such online MSCS program. Based responses to an open ended question, we develop a typology of motivations, including goals (e.g. career transition), opportunities (e.g. enrolling without taking time off work), and assurances that their goals will be met (e..g the program’s accreditation). We then issue a closed survey question to a new group of students to further explore these motivations. In this paper, we discuss both aggregate and demographic trends in motivations, including the different motivations of men and women and what they imply about the program’s impact on the gender divide in computing. We also examine older students’ tendency towards intrinsic motivation to pursue an MSCS degree.

Abstract: Peer advising, where students receive advice from other students about courses, can be difficult to study due to its private, spontaneous nature or to potential response bias. However, peer advising can help create and strengthen student communities, and the information students provide can help improve course quality. We first present a case study on a student-run course review website developed to support an online graduate program in computer science. We discuss the evolution of the website and examine its usefulness as a student-organized community providing peer advising at scale. In our second study, we develop a coding scheme and use it to analyze reviews from the website. Although students provide mostly evaluative information, they also provide advice, context for their reviews, course descriptions, and feedback for the instructional team. This research explores the importance of student-organized communities in higher education and provides useful insights into peer advising at scale.

Abstract: Peer evaluation in educational contexts has been well-researched, but there are open questions about the trends students follow over the course of repeated peer evaluation activities within the same course or semester. We hypothesize a trend whereby reviewers who initially heavily invest in giving their classmates strong peer reviews lower their performance over time due in part to disillusionment with the lower-quality feedback they receive. To test this hypothesis, we investigated a dataset of over 50,000 peer assignment evaluations gathered from three semesters of two different courses, totaling over 79 assignments. We examine whether class performance across four quantitative variables drops over the course of the semester, and whether those drops are specifically more prevalent among high-performing reviewers. We find evidence that reviewers who begin the semester committed suffer a greater drop in
performance over time, and propose potential causal mechanisms for this drop as well as plans to potentially prevent it.

Abstract: This paper is a survey of the availability of open data sets generated from Massively Open Online Courses (MOOCs). This log data allows researchers to analyze and predict student performance. Often, the goal of the analysis is to focus on at-risk students who are not likely to finish a course. There is a growing gap between the average researcher (who does not have access to proprietary data) and the ready availability of data sets for analysis. Most research papers studying and predicting student performance in MOOCs are done on proprietary data sets that are not anonymized (deidentified) or released for general study. There are no standardized tools that provide a gateway to access usable data sets; instead, the researcher must navigate a maze of sites with different data structures and varying data access policies. To our knowledge, no open data sets are being produced, and have not been since 2016. The authors survey the history of MOOC data sharing, identify the few available open data sets, and discuss a path forward to increase the reproducibility of MOOC research.

Abstract: Many of the demands of scale appear at odds with quality, such as in the case of online MOOC degree programs. This paper presents research that focuses on reducing this dichotomy. Examination essays are important learning tools in for-credit courses. Psychology studies show they drive deeper learning strategies and invoke higher levels of cognitive processing in students. High-quality feedback is another critical aspect of students’ learning in advanced education. Key components of high-quality feedback are how expeditiously feedback is presented and how unambiguous it is. Examination essays are time-consuming for educators to grade, and at scale this may create an incentive for superficial feedback that may also reach students in a less than ideal turnaround time. This imposes clear scalability constraints and a potential decrease in course quality in online for-credit MOOC programs. We propose an annotation-free artificial intelligence-based approach for the automatic generation of examination essay rubrics and its subsequent utilization as part of high-quality feedback to students. It utilizes a combination of the natural language processing techniques TextRank and semantic similarity for automatic rubric generation and concept map creation for feedback presentation. The feedback is immediately available upon essay submission and offers the student a conceptual analysis of the submitted essay against the assignment’s learning objectives.

Abstract: The current boon of affordable, scalable online degrees is supported by the modern internet as a medium to construct high-quality, fully asynchronous learning experiences. The asynchronous nature of these programs accommodates working professionals who are unable to carve out consistent and prescribed times for the pursuit of a degree, but who nonetheless have the dedication and ability to succeed at the programs’ content. However, these classes’ asynchronous nature breaks some of students’ assumptions about the structure of college courses. Scheduled meetings do more than just support disseminating lecture material or facilitating synchronous activities; they establish a classroom cadence and set students’ expectations for the pace and routine of the course. Without meetings, these expectations are lost. Moreover, the persistent availability of asynchronous material removes scarcity from the experience, and while this is one of the medium’s strengths, it may also lure students into unhealthy procrastination. Thus, asynchronous online classes must actively accomplish that which traditional classes accomplish passively through required lecture attendance: creating cadence and incentivizing persistent engagement.

Abstract: In 2014, Georgia Tech launched the first for-credit MOOC-based graduate degree program. In the five years since, the program has proven generally successful, enrolling over 14,000 unique students, and several other similar programs have followed in its footsteps. Existing research on the program has focused largely on details of individual classes; program-level research, however, has been scarce. In this paper, we delve into the program-level details of an at-scale Master’s degree, from the story of its creation through the data generated by the program, including the numbers of applications, admissions, matriculations, and graduations; enrollment details including demographic information and retention patterns; trends in student grades and experience as compared to the on-campus student body; and alumni perceptions. Among our findings, we note that the program has stabilized at a retention rate of around 70%; that the program’s growth has not slowed; that the program has not cannibalized its on-campus counterpart; and that the program has seen an upward trend in the number of women enrolled as well as a persistently higher number of underrepresented minorities than the on-campus program. Throughout this analysis, we abstract out distinct lessons that should inform the development and growth of similar programs.

Abstract: Online classes and degree programs continue to grow in popularity, in part due to the increased convenience and accessibility of education that technology has provided in recent years. As online education scales upwards and outwards, there is an increased need to provide students with an engaging and collaborative learning experience. In some online learning environments, student collaboration is perceived to be more difficult than it is in a physical classroom setting due to cultural or geographic distance between students. In particular, online class lectures often lack the collaborative spirit seen in most in-person classroom lectures. To improve upon the online classroom experience, this project first examines the benefits and drawbacks of several in-person and online lecture delivery techniques, then proposes an online lecture platform that allows students to facilitate their own collaborative classrooms on-demand through a semi-synchronous viewing area and chatroom.

Abstract: Peer advising in education, which involves students providing fellow students with course advice, can be important in online student communities and can provide insights into potential course improvements. We examine reviews from a course review web site for online graduate programs. We develop a coding scheme to analyze the free text portion of the reviews and integrate those findings with students’ quantitative ratings of each course’s overall score, difficulty, and workload. While reviews focus on subjective evaluation of courses, students also provide feedback for instructors, personal context, advice for other students, and objective course descriptions. Additionally, the average review varies by course overall score, difficulty, and workload. Our research examines the importance of student communities in online education and peer advising at scale.

Abstract: Despite the ubiquitous use of instructional videos in both formal and informal learning settings, questions remain largely unanswered on how to design and develop video lessons that are often used as the primary method for delivering instruction in online courses. In this study, we experimented with a model of seven principles drawn from instructional design theories for designing and developing video lessons for an online graduate course. Feedback was collected from students through surveys on their perceptions of the effectiveness of the video lessons and the overall course quality for eight semesters. This paper shares the instructors’ experience on the design and development of the video lessons as well as the survey findings. Implications of the findings for instructional design and future research are also discussed.

Abstract: In January 2017, a major public research university launched an online version of CS1 targeted at on-campus students to address rising enrollments and provide students with flexibility in their schedules. Prior research on this class has found positive outcomes: students in the course achieve the same learning outcomes as those in a traditional course, while reporting a lower time investment to reach those outcomes and a high level of student satisfaction. This research builds on that prior work in two ways. First, it replicates the findings from that earlier semester with an entirely new semester of students. Second, it delves deeper into the student experience within the online course and its traditional counterpart. This deeper analysis focuses specifically on the differing ways in which students in each section allocated their time, whether or not students in either section accessed the opposite section’s material, and their future preferences in online vs. residential CS classes.

Abstract: In 2014, Georgia Tech launched an online campus for its Master of Science in Computer Science program. The degree, equal in stature and accreditation to its on-campus counterpart, offered a notably lower cost of attendance. Its design emphasized flexibility in both geography and time, allowing students from around the world to earn a highly-ranked MSCS without taking time off work or moving to campus. Five years later, the program enrolls over 8000 students per semester and has graduated 1500 alumni. It is believed to be the largest program of its kind and has received recognition from national organizations on professional education. Existing research on the program has focused on challenges and opportunities to scale that are agnostic to the content itself. In this reflection, we look at the creation and growth of the program as it relates to graduate-level CS instruction. In particular, we note a unique and powerful unity of content and platform: the online delivery of the program dovetails with the technical skillsets of the professors and students that it draws, putting both in the position to contribute and innovate.

Abstract: This chapter describes a blended learning class used to teach a large Computer Science 1 course that serves 1000 students per year and is a bottleneck course that limits how many students can enroll be accepted into computing majors. The course uses a MOOC to provide lectures, practice, and assignments. In addition, students could get help from three sources: recitation with teaching assistants, a help desk, or office hours with the instructor. The chapter discusses the production of the MOOC, which was designed to be adaptable to multiple types of intro computing courses for multiple majors. The design team also surveyed online Intro to Computer Science courses and share the best practices that they experienced.

This course is blended – at least for some students. Based on the MIX taxonomy, the required parts of the course are delivered entirely online. The online components include content delivery and content application with feedback through Vocareum and McGraw-Hill’s adaptive Smartbook. The optional parts of the class provide in-person content delivery and feedback during content application. During recitation, teaching assistants provided lectures and guidance on activities. For additional guidance on activities, the students could go to the help desk or the instructor’s office hours.

The research method is a quasi-experimental design. The two experimental groups, students who took the face-to-face class and students who took the blended class, took the class during the same semester. Student performance was measured through a standardized computer science assessment. Student perceptions, demographic information, and expectations for the course were measured through surveys.

Abstract: We outline how we detected programming plagiarism in an introductory online course for a master’s of science in computer science program, how we achieved a statistically significant reduction in programming plagiarism by combining a clear explanation of university and class policy on academic honesty reinforced with a short but formal assessment, and how we evaluated plagiarism rates before and after implementing our policy and assessment.

Abstract: Large undergraduate CS courses receive thousands of code submissions per term. To help make sense of the large quantities of submissions, projects have emerged to dynamically cluster student submissions by approach for writing scalable feedback, tailoring hints, and conducting research. However, relatively little attention has been paid to the value of these tools for informing revision to core course materials and delivery methods. In this work, we applied one such technology—Sense, the eponymous product of its company—to an online CS1 class delivered simultaneously for credit to on-campus students and for free to MOOC students. Using Sense, we clustered student submissions to around 70 problems used throughout the course. In this work, we discuss the value of such clustering, the surprising trends we discovered through this process, and the changes made or planned to the course based on the results. We also discuss broader ideas on injecting clustering results into course design.

Abstract: As online offerings have matured and expanded, new efforts have recently been devoted to opening fully-accredited online versions of traditional on-campus classes. Such classes may be offered to on-campus students for greater flexibility during busy semesters, or to allow them to continue to make progress toward their degrees during internships or semesters away from campus. This trend intersects with a growing CS for All movement that sees more and more non-computer science majors enrolling in CS classes. As online offerings expand, it is important for us to understand who enrolls in online sections and the reasons for their choices, both to make sure that learning outcomes are similar across different delivery mechanisms, and to take advantage of opportunities to tailor course content to specific audiences. In this analysis, we look at two versions, one online and one traditional, of a CS1 class offered at a major public research university. We investigate demographic, motivational, and experiential components to identify which types of students are most likely to select each version and what implications this decision has for student success and course design.

Abstract: If a MOOC is to qualify for equal credit as an existing on-campus offering, students must achieve comparable outcomes, both educational and attitudinal. We have built a MOOC for teaching CS1 with the intent of offering it for degree credit. To test its eligibility for credit, we delivered it as an online for-credit course for two semesters to 197 on-campus students who selected the online version rather than a traditional version. We compared the demographics, outcomes, and experiences of these students to the 715 students in the traditional version. We found the online students more likely to be older; to be underrepresented minorities; and to have previously failed a CS class. We then found that our online students attained comparable learning outcomes to students in the traditional section. Finally, we found that our online students perceived the online course quality more positively and required less time to achieve those comparable learning outcomes.

Abstract: In recent years, non-credit options for learning at scale have outpaced for-credit options. To scale for-credit options, workflows and policies must be devised to preserve the characteristics of accredited higher education—such as the presumption of human evaluation and an assertion of academic integrity—despite increased scale. These efforts must follow as well with shifting from offering isolated courses (or informal collections thereof) to offering full degree programs with additional administrative elements. We see this shift as one from Massive Open Online Courses (MOOCs) to Large, Internet-Mediated Asynchronous Degrees (Limeades). In this work, we perform a qualitative research study on one such program that has scaled to 6,500 students while retaining full accreditation. We report a typology of policies and workflows employed by the individual classes to deliver this experience.

Abstract: Massive Open Online Courses (MOOCs) may reach a massive number of people, but few MOOCs count for credit. Scaling rigorous assessment, feedback, and integrity checks presents difficulties. We implemented an AI system for a CS1 MOOC-for-credit to address both scale and endorsement. In this analysis, we present the design of the system and an evaluation of the course. We observe that students in the online course achieve comparable learning outcomes, report a more positive student experience, and identify AI-equipped programming problems as the primary contributor to their experiences.

Abstract: We used a sentiment analysis tool, VADER (Valence Aware Dictionary and sEntiment Reasoner), to analyze Student Evaluations of Teaching (SET) of a single course from three different sources: official evaluations, forum comments from another course, and an unofficial “reviews” site maintained by students. We compared the positive and negative valences of these sites; identified frequently-used key words in SET comments and determined the impact on positivity/negativity of comments that included them; and determined positive/negative values by question on the official course SET comments. Many universities use similar questions, which may make this research useful for those analyzing comments at other institutions. Previous published studies of sentiment analysis in SET settings are rare.

Abstract: Traditionally, education relies on a linear relationship between enrollment and staff; rising enrollment dictates increases to staff with some expertise (such as teaching assistants, TAs) for evaluation. This relationship is expensive, so learning at scale has largely deemphasized expert evaluation and feedback. Two organizations, though, have used different models to scale up class size online while retaining this expert evaluation and feedback. In this paper, we analyze the methods these two organizations have used to increase enrollment while preserving scalability and feedback. We observe an academic program has scaled feedback with traditional TAs by relying on unique characteristics of its student body, while a commercial program has done so with a novel, network-based model. These successes show the potential of learning from experts at scale.

Abstract: In January 2017, Georgia Tech launched a new online section of its CS1301: Introduction to Computing class. The course, offered both as a for-credit course to on-ground students and as an open MOOC, built on four unique design principles: congruency, adaptivity, modularity, and personalization. In this short paper, we describe the background of the course, the definitions of these design principles, and their application to the course design.

Abstract: Large classes, both online and residential, typically demand many graders for evaluating students’ written work. Some classes attempt to use autograding or peer grading, but these both present challenges to assigning grades at for-credit institutions, such as the difficulty of autograding to evaluate free-response answers and the lack of expert oversight in peer grading. In a large, online class at Georgia Tech in Summer 2015, we experimented with a new approach to grading: framing graders as meta-reviewers, charged with evaluating the original work in the context of peer reviews. To evaluate this approach, we conducted a pair of controlled experiments and a handful of qualitative analyses. We found that having access to peer reviews improves the perceived quality of feedback provided by graders without decreasing the graders’ efficiency and with only a small influence on the grades assigned.

Abstract: Many ongoing efforts in online education aim to increase accessibility through affordability and flexibility, but some critics have noted that pedagogy often suffers during these efforts. In contrast, in the low-cost for-credit Georgia Tech Online Masters of Science in Computer Science (OMSCS) program, we have observed that the features that make the program accessible also lead to pedagogical benefits. In this paper, we discuss the pedagogical benefits, and draw a causal link between those benefits and the factors that increase the program’s accessibility.

Abstract: In October 2014, one-time MOOC developer Udacity completed its transition from primarily producing massive, open online courses to producing job-focused, project-based microcredentials called “Nanodegree” programs. With this transition came a challenge: whereas MOOCs focus on automated assessment and peer-to-peer grading, project-based microcredentials would only be feasible with expert evaluation. With dreams of enrolling tens of thousands of students at a time, the major obstacle became project evaluation. To address this, Udacity developed a system for hiring external experts as project reviewers. A year later, this system has supported project evaluation on a massive scale: 61,000 projects have been evaluated in 12 months, with 50% evaluated within 2.5 hours (and 88% within 24 hours) of submission. More importantly, students rate the feedback they receive very highly at 4.8/5.0. In this paper, we discuss the structure of the project review system, including the nature of the projects, the structure of the feedback, and the data described above.

Abstract: Cheating in computer science classes can damage the reputation of institutions and their students. It is therefore essential to routinely authenticate student submissions with available software plagiarism detection algorithms such as Measure of Software Similarity (MOSS). Scaling this task for large classes where assignments are repeated each semester adds complexity and increases the instructor workload. The MOSS Tool for Addressing Plagiarism at Scale (MOSS-TAPS), organizes the MOSS submission task in courses that repeat coding assignments. In a recent use-case in the Online Master of Science in Computer Science (OMSCS) program at the Georgia Institute of Technology, the instructor time spent was reduced from 50 hours to only 10 minutes using the managed submission tool design presented here. MOSS-TAPS provides persistent configuration, supports a mixture of software languages and file organizations, and is implemented in pure Java for cross-platform compatibility.

Abstract: Despite the ubiquitous use of videos in online learning and enormous literature on designing online learning, there has been relatively little research on what pedagogical strategies should be used to make the most of video lessons and what constitutes an effective video for student learning. We experimented with a model of incorporating four pedagogical strategies, four instructional phases, and four production guidelines-in designing and developing video lessons for an online graduate course. In this paper, we share our experience as well as students’ perceptions of their effectiveness. We also discuss what needs to be done for future research.

Abstract: In Fall 2014 we offered an online course on Knowledge-Based Artificial Intelligence (KBAI) to about 200 students as part of the Georgia Tech Online MS in CS program. By now we have offered the course to more than 1000 students. We describe the design, development and delivery of the online KBAI class in Fall 2014.

Abstract: The Raven’s Progressive Matrices (RPM) test is a commonly used test of intelligence. The literature suggests a variety of problem-solving methods for addressing RPM problems. For a graduate-level artificial intelligence class in Fall 2014, we asked students to develop intelligent agents that could address 123 RPM-inspired problems, essentially crowdsourcing RPM problem solving. The students in the class submitted 224 agents that used a wide variety of problem-solving methods. In this paper, we first report on the aggregate results of those 224 agents on the 123 problems, then focus specifically on four of the most creative, novel, and effective agents in the class. We find that the four agents, using four very different problem-solving methods, were all able to achieve significant success. This suggests the RPM test may be amenable to a wider range of problem- solving methods than previously reported. It also suggests that human computation might be an effective strategy for collecting a wide variety of methods for creative tasks.

Abstract: Science education nowadays emphasizes authentic science practices mimicking the creative discovery processes of real scientists. How, then, can we build creativity support tools for student learning about scientific discovery processes? We summarize several epistemic views of ideation in scientific discovery and find that the ideation techniques provide few guarantees of correctness of scientific hypotheses, indicating the need for supporting hypothesis evaluation. We describe an interactive tool called MILA−S that enables students to elaborate hypotheses about ecological phenomena into conceptual models and evaluate conceptual models through agent-based simulations. We report on a pilot experiment with 50 middle school students who used MILA−S to discover causal explanations for an ecological phenomenon. Preliminary results from the study indicate that use of MILA–S had a significant impact both on the creative process of model construction and the nature of the constructed models. We posit that the computational support for model construction, evaluation and revision embodied in MILA–S fosters student creativity in learning about scientific discovery processes.

Abstract: Metacognitive skills are critical in learning but difficult to teach. Thus the question becomes how can we facilitate metacognitive tutoring? We present an exploratory learning environment called MILA-T with embedded metacognitive tutors imitating five functional roles of teachers in classrooms. We tested MILA–T in a controlled experiment with 237 middle school students. We examine the impact of MILA–T on the models of a natural phenomenon constructed by the students. We find that students with access to MILA–T wrote better evidential justifications for their models, and thus, deliver better-justified models for the phenomenon. We also find that these improvements persisted during a transfer task. These results lend support for organizing metacognitive tutoring around the functional roles of teachers for supporting inquiry-driven modeling.

Abstract: This paper presents the design and evaluation of a set of intelligent tutoring agents constructed to teach teams of students an authentic process of inquiry-driven modeling. The paper first presents the theoretical grounding for inquiry-driven modeling as both a teaching strategy and a learning goal, and then presents the need for guided instruction to improve learning of this skill. However, guided instruction is difficulty to provide in a one-to-many classroom environment, and thus, this paper makes the case that interaction with a metacognitive tutoring system can help students acquire the skill. The paper then describes the design of an exploratory learning environment, the Modeling and Inquiry Learning Application (MILA), and an accompanying set of metacognitive tutors (MILA–T). These tools were used in a controlled experiment with 84 teams (237 total students) in which some teams received and interacted with the tutoring system while other teams did not. The effect of this experiment on teams’ demonstration of inquiry-driven modeling are presented.

Abstract: MILA–T (MILA–Tutoring) is constructed to give students explicit instruction on scientific modeling and inquiry, intending in part to help cultivate positive attitudes toward science. The results of a two-week controlled experiment using MILA–T in middle school classroom show a significant effect of MILA–T on students’ attitudes towards science.

Abstract: Scientists use both conceptual models and executable simulations to help them make sense of the world. Models and simulations each have unique affordances and limitations, and it is useful to leverage their affordances to mitigate their respective limitations. One way to do this is by generating the simulations based on the conceptual models, preserving the capacity for rapid revision and knowledge sharing allowed by the conceptual models while extending them to provide the repeated testing and feedback of the simulations. In this paper, we present an interactive system called MILAfiS for generating agent-based simulations from conceptual models of ecological systems. Designed with STEM education in mind, this user-centered interface design allows the user to construct a Component-Mechanism-Phenomenon conceptual model of a complex system, and then compile the conceptual model into an executable NetLogo simulation. In this paper, we present the results of a pilot study with this interface with about 50 middle school students in the context of learning about ecosystems.

Abstract: Scientists use both conceptual models and executable simulations to help them make sense of the world. Models and simulations each have unique affordances and limitations, and it is useful to leverage their affordances to mitigate their respective limitations. One way to do this is by generating the simulations based on the conceptual models, preserving the capacity for rapid revision and knowledge sharing allowed by the conceptual models while extending them to provide the repeated testing and feedback of the simulations. In this paper, we present an interactive system called MILAfiS for generating agent-based simulations from conceptual models of ecological systems. Designed with STEM education in mind, this user-centered interface design allows the user to construct a Component-Mechanism-Phenomenon conceptual model of a complex system, and then compile the conceptual model into an executable NetLogo simulation. In this paper, we present the results of a pilot study with this interface with about 50 middle school students in the context of learning about ecosystems.

Abstract: In order to tackle problems in the modern world, individuals must possess a strong ability to reason about and understand complex systems in a practical and useful way. Past research has indicated that experts and novices possess fundamentally different kinds of understanding of complex systems. Therefore, to adequately prepare students to address problems pertaining to complex systems, it is important to help them acquire an authentic expert- like understanding of these systems. We approach this problem from two angles: first, we create an interactive environment in which students may investigate complex systems in a manner similar to that of scientists and engineers; secondly, we embed metacognitive agents in the interactive environments such that the agents provide situated guidance towards expert-like understanding of complex systems. In this paper, we detail the design of these two systems, referred to as MILA and MeTA respectively, and the way in which they help students obtain a more authentic and advanced understanding of complex systems. We also briefly describe the deployments of these tools in a science summer camp for middle school students and preliminary results of students’ interaction with them.

Abstract: In this paper, we describe the evolution of an interactive technology called the Ecological Modeling Toolkit (EMT) that supports learning about complex ecological systems in middle school science. Authentic learning of science is facilitated by imitation, rehearsal and understanding of real-world scientific practices such as observation, experimentation, problem formulation, hypothesis testing, and model construction and revision. We illustrate how the tools in EMT work together to support many real-world scientific practices such as model construction, simulation and revision, and scaffold others such as observation, problem formulation and hypothesis testing.

Abstract: Structure-Behavior-Function (SBF) models of complex systems use functions as abstractions to organize knowledge of structural components and causal processes in a system. We describe an interactive learning environment called ACT (Aquarium Construction Toolkit) for constructing simple SBF models of classroom aquaria, and report on a case study on the use of SBF thinking and the ACT tool in middle school science classes. We present initial data indicating that SBF thinking supported in part by the ACT tool leads to enhanced understanding of functions and behaviors of aquaria.

Abstract: Learning about ecosystems is challenging because, like any complex system, they are simultaneously multidimensional and dynamic. Often, learners engage only with the visible components of an ecosystem and draw either single or linear causal connections between components. In this study, we explored how using a Structure-Behavior-Function framework supported middle school students’ conceptual and complex reasoning about the visible and invisible components of an ecosystem. Research shows that learners often engage only with the visible components of an ecosystem and draw linear/single causal connections between the components of the ecosystem. Our findings suggest that a combination of using structure, behavior, and function approach along with a set of carefully designed technology tools can push the students toward a better understanding of the ecosystem functioning. The results show that along with the visible components of the ecosystem, students have started to identify the invisible components of the ecosystem.

Abstract: This paper presents Tangible Tracking Table, an interactive tabletop display, and Optical Chess, a strategy game. We discuss the design and implementation of both systems and report our evaluation game play sessions with young adults, with a special focus on how the Tangible Tracking Table enhances interaction over a point-and-click interface.