Publications

The page lists my authored or co-authored book chapters, refereed journal articles, and peer-reviewed conference papers. For my books, see my dedicated Books page. For a full list of publications including workshop papers, symposium abstracts, and op/eds, see my full CV.

2021

Beyond Instruction: Scaling Support for a Large Online Master’s Program

David A. Joyner

International Perspectives on Supporting and Engaging Online Learners

Abstract: Drawbacks of online education, especially at scale, include the isolation and loss of community that come with taking classes from one’s home rather than in a typical coordinated classroom. In this series of studies, we explore the potential of an immersive virtual environment to recreate shared educational contexts in an online program, emphasizing those that focus on universal dynamics like discussions, Q&A, and lecture-watching. Using Mozilla Hubs-which offers virtual reality as well as a browser-based immersive environment for participants without VR headsets-we implement three common, student-driven on-campus environments: a lecture hall, a student lounge, and a poster session. We test these environments with students in an online graduate program through surveys and a controlled experiment. We find significant interest in the potential of these environments, but hurdles to their adoption as well.

Content-Neutral Immersive Environments for Cultivating Scalable Camaraderie

David A. Joyner, Akhil Mavilakandy, Ishaani Mittal, Denise Kutnick, and Blair MacIntyre

Proceedings of the Eighth ACM Conference on Learning @ Scale

Download Full Paper

Abstract: Drawbacks of online education, especially at scale, include the isolation and loss of community that come with taking classes from one’s home rather than in a typical coordinated classroom. In this series of studies, we explore the potential of an immersive virtual environment to recreate shared educational contexts in an online program, emphasizing those that focus on universal dynamics like discussions, Q&A, and lecture-watching. Using Mozilla Hubs-which offers virtual reality as well as a browser-based immersive environment for participants without VR headsets-we implement three common, student-driven on-campus environments: a lecture hall, a student lounge, and a poster session. We test these environments with students in an online graduate program through surveys and a controlled experiment. We find significant interest in the potential of these environments, but hurdles to their adoption as well.

With or Without EU: Navigating GDPR Constraints in Human Subjects Research in an Education Environment

Alex Duncan and David A. Joyner

Proceedings of the Eighth ACM Conference on Learning @ Scale

Download Full Paper

Abstract: The General Data Protection Regulation (GDPR), passed in 2018, outlined new data privacy standards applying to residents of the European Union (EU). The impact of this law stretches beyond the EU to anyone – such as researchers across the world – collecting or processing data from EU residents. Researchers have had to augment their methodologies to ensure GDPR compliance. This initiative intersects with at-scale educational programs, which often enroll EU students and are also often the subject of institutional research. While creating a study to research students in an online Master of Science in Computer Science (MSCS) program, some of whom are in the EU, we encountered difficulties with ensuring GDPR compliance. This paper discusses the implications of the GDPR related to both general research and our specific study. We discuss the challenges of interpreting the GDPR and integrating it into our methodology as well as potential solutions, our ultimate resolution, and practical recommendations, and we consider what the future of data privacy legislation means for researchers.

Toward Reshaping the Syllabus for Education at Scale

Bobbie Eicher and David A. Joyner

Proceedings of the Eighth ACM Conference on Learning @ Scale

Download Full Paper

Abstract: Ensuring that students are fully informed about course content and policies is always a challenge, but online education at scale adds additional complications. In this paper we present observations about the place of the syllabus in education at scale, based on the actual syllabus documents from 48 courses in a Computer Science Master’s degree program offered online and at scale. On the basis of these observations, we offer preliminary recommendations for factors that instructors should keep in mind when they compile a syllabus for similar courses.

Components of Assessments and Grading at Scale

Bobbie Eicher and David A. Joyner

Proceedings of the Eighth ACM Conference on Learning @ Scale

Download Full Paper

Abstract: One of the major criticisms of efforts towards offering education at scale has been the Trap of Routine Assessment, the risk that student assessment will suffer from becoming excessively simplified in service of automation and scale. In this research, we examine the ways that students in an at-scale graduate program in computer science were assessed during their degrees. The program in question has scaled to over 10,000 students in only a few years, but awards a traditional Master’s degree, providing the opportunity to investigate whether scale was achieved by transitioning to more routine assessment or by bringing scale to traditional strategies. To do this, we investigate the syllabi of 52 classes offered through the program to identify the types of assessments used, and we survey teaching teams for their approaches to evaluating these assessments. We merge this data with historical enrollment data to gain an overall summary of the kinds of assessments and evaluations received during their degrees. We ultimately find the program’s scale has been managed by scaling up traditional assessment and evaluation strategies as the majority of grades are generated by human teaching teams based on projects and homeworks, with a relatively smaller portion generated exclusively by automated evaluation of exams.

Mechanisms for Supporting Emergency Remote Classes: Towards a Distributed Classroom

David A. Joyner

Moving Horizontally: The New Dimensions of At-Scale Learning at the Time of COVID-19

Download Full Paper

Abstract: During the rapid emergency transition to remote classes in 2020, our online Master of Science in Computer Science program supported the newly remote traditional classes in several ways. In this chapter, we go over some of those ways, including providing direct feedback, opening up remote instructional resources, reassigning classes to remote instructors, and providing material for the formation of local cohorts. We then investigate how these mechanisms are small steps toward a broader, more fundamental reimagining of classrooms as distributed across time and space.

Towards Mutual Theory of Mind in Human-AI Interaction: How Language Reflects What Students Perceive About a Virtual Teaching Assistant

Qiaosi Wang, Koustuv Saha, Eric Gregori, David A. Joyner and Ashok Goel

Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

Download Full Paper

Abstract: Building conversational agents that can conduct natural and prolonged conversations has been a major technical and design challenge, especially for community-facing conversational agents. We posit Mutual Theory of Mind as a theoretical framework to design for natural long-term human-AI interactions. From this perspective, we explore a community’s perception of a question-answering conversational agent through self-reported surveys and computational linguistic approach in the context of online education. We first examine long-term temporal changes in students’ perception of Jill Watson (JW), a virtual teaching assistant deployed in an online class discussion forum. We then explore the feasibility of inferring students’ perceptions of JW through linguistic features extracted from student-JW dialogues. We find that students’ perception of JW’s anthropomorphism and intelligence changed significantly over time. Regression analyses reveal that linguistic verbosity, readability, sentiment, diversity, and adaptability reflect student perception of JW. We discuss implications for building adaptive community-facing conversational agents as long-term companions and designing towards Mutual Theory of Mind in human-AI interaction.

2020

The Search for the MOOC Credit Hour

James J. Lohse, Filipe Altoe, Jasmine Jose, Andrew Nowotarski, Farrukh Rahman, Robert Tuck and David A. Joyner

Proceedings of Learning With MOOCs 2020

Download Full Paper

Abstract: Over the past decade, MOOCs have risen to prominence in part because most do not carry heavy requirements regarding accreditation, scope, and evaluation. While this has allowed MOOCs to proliferate, it has led to difficulty under-standing what completing a particular MOOC means. Individual MOOCs can vary tremendously in their quantity of content, their depth of assessment, and their mechanisms to ensure academic integrity. In order for MOOC credentials to have value in academia and the workplace, however, there must be some trust in what a particular credential means about the learner. In this work, we perform a study to understand the variety of MOOCs available, then pro¬pose a MOOC Content Matrix as a short, objective way to summarize what a particular MOOC entails and, in turn, what value its certificate gives.

The Synchronicity Paradox in Online Education

David A. Joyner, Qiaosi Wang, Suyask Thakare, Shan Jing, Ashok Goel and Blair McIntyre

Proceedings of the Seventh ACM Conference on Learning @ Scale

Download Full Paper

Abstract: As online education proliferates, one concern that has been raised is that it may fail to capture desirable emergent phenomena from on-campus programs. Student community is one example of such a phenomenon: on-campus student communities thrive based on synchronous collocation. An online program might be designed to capture all deliberate constructs in an on-campus program, but there may be beneficial side effects of synchronous collocation that are not apparent. In this work, we examine the issue of social isolation in an online graduate program. By happenstance, three studies were conducted in relative isolation looking at social isolation from different angles. The first study examined trajectories in social presence as a semester proceeded. The second study developed an understanding of students’ needs with regard to community in an online program. The third study tested out an immersive virtual environment to try to improve students’ sense of connectedness. Combining their findings, we find compelling evidence of the existence of a Synchronicity Paradox in online education: students desire synchronicity to form strong social communities, and yet part of the chief appeal of these online pro-grams is their asynchronicity. In light of this finding, we provide design guidelines for how synchronicity may be reintroduced into asynchronous programs without sacrificing the benefits of asynchronicity. More specifically, we propose that scale itself may be the key to building emergent synchronicity.

Sensing Affect to Empower Students: Learner Perspectives on Affect-Sensitive Technology in Large Educational Contexts

Qiaosi Wang, Shan Jing, David A. Joyner, Lauren Wilcox, Hong Li, Thomas Ploetz and Betsy DiSalvo

Proceedings of the Seventh ACM Conference on Learning @ Scale

Download Full Paper

Abstract: Large-scale educational settings have been common domains for affect detection and recognition research. Most research emphasizes improvements in the accuracy of affect measurement to enhance instructors’ efficiency in managing large numbers of students. However, these technologies are not designed from students’ perspectives, nor designed for students’ own usage. To identify the unique design considerations for affect sensors that consider student capacities and challenges, and explore the potential of affect sensors to support students’ self-learning, we conducted semi-structured interviews and surveys with both online students and on-campus students enrolled in large in-person classes. Drawing on these studies we: (a) propose using affect data to support students’ self-regulated learning behaviors through a “scaling for empowerment” design perspective, (b) identify design guidelines to mitigate students’ concerns regarding the use of affect data at scale, (c) provide design recommendations for the physical design of affect sensors for large educational settings.

Informal Learning Communities: The Other Massive Open Online ‘C’

Will Hudgins, Michael Lynch, Ash Schmal, Harsh Sikka, Michael Swenson and David A. Joyner

Proceedings of the Seventh ACM Conference on Learning @ Scale

Download Full Paper

Abstract: While the literature on learning at scale has largely focused on MOOCs, online degree programs, and AI techniques for supporting scalable learning experiences, informal learning communities have been relatively underrepresented. Nonetheless, these massive open online learning communities regularly draw far more engaged users than the typical MOOC. Their informal structure, however, makes them significantly more difficult to study. In this work, we take a first step toward attempting to understand these communities specifically from the perspective of scale. Taking a sample of 62 such communities, we develop a tagging system for understanding the specific features and how they relate to scale. For example, just as a MOOC cannot manually grade every assignment, so also an informal learning community cannot approve every contribution; and just as MOOCs therefore employ autograding, informal learning communities employ crowd-sourced moderation or plat-form-driven enforcement. Using these tags, we then select several communities for deeper case studies. We also use these tags to make sense of learning-based subreddits from the popular community site Reddit, which offers an API for programmatic analysis. Based on these techniques, we offer findings about the performance of informal learning communities at scale and issue a call to include these environments more fully in future research on learning at scale.

Affordable Degrees at Scale: New Phenomenon or New Hype?

David S. Park, Robert W. Schmidt, Charankumar Akiri, Stephanie Kwak, David A. Joyner

Proceedings of the Seventh ACM Conference on Learning @ Scale

Download Full Paper

Abstract: Following the initial proliferation of Massive Open Online Courses (MOOCs), a more recent trend has emerged toward offering “Affordable Degrees at Scale” or “Large, Internet-Mediated Asynchronous Degrees”. In this research, we set out to understand this space: the range in tuition costs for these programs, the variety of admissions standards, and the types of assessments used to evaluate these non-traditional students. In the process, however, we found that in many ways, these programs may not be as new as we initially perceived: similarly-priced online programs have existed from traditional universities for years. In this research, we explore these two questions: what are these new degrees at scale, and how do they actually differ from traditional programs? To explore this, we collected materials for 35 MOOC-based graduate degrees and numerous non-MOOC-based comparable degrees. We then explored the patterns in tuition, admissions requirements, and syllabus information. In this paper, we report the trends we identified in MOOC-based degrees, and attempt to answer the question: what makes these programs different from non-MOOC-based online programs of the past? Ultimately, we find that this new era of programs is similar in many observable ways.

Peripheral and Semi-Peripheral Community: A New Design Challenge for Learning at Scale

David A. Joyner

Proceedings of the Seventh ACM Conference on Learning @ Scale

Download Full Paper

Abstract: Existing attempts to foster a greater sense of community in online education have largely focused on direct interactions among students in peer review, forums, and other mechanisms. In this paper, we pose a new design challenge for learning at scale: peripheral community. Peripheral community is the sense of community derived from peripheral interactions in which a student has visibility into others’ behaviors without a direct, intentional interaction occurring between the students. We argue for the value of peripheral community by examining opportunities for such visibility in residential learning environments. We then explore possible ways to supply peripheral community, both in the form of new initiatives and in reinterpretations of existing interventions as fostering peripheral community.

SAGA: Curricula Optimization

Anneli LeFranc and David A. Joyner

Proceedings of the Seventh ACM Conference on Learning @ Scale

Download Full Paper

Abstract: This paper presents two approaches using Simulated Annealing and a genetic algorithm to create optimal curricula. The method generates a customized course selection and schedule for individual students enrolled in a large online graduate program in computer science offered by a major public research institution in the United States.

Challenges of Online Learning in Nigeria

Kabir Abdulmajeed, David A. Joyner and Christine McManus

Proceedings of the Seventh ACM Conference on Learning @ Scale

Download Full Paper

Abstract: Education has traditionally been administered via physical interactions between teachers and students in classrooms. Through technological advancement in communications and digital devices, online education has been developed with the potential to scale education, making it affordable and accessible. With an internet connection and a laptop or mobile phone, learners can access massive open online courses (MOOCs) for free. Nonetheless, the opportunity to scale education and the advantages of online learning are not always fulfilled due to certain challenges. In this work, Socioeconomic, Sociocultural, and IT infrastructural factors are categorized as challenges hindering the adoption of online learning in Nigeria. Although some factors mitigating online learning have been identified in the past, there is relatively little empirical evidence indicating the reality and severity of these challenges. Since scaling education involves worldwide reach, local contexts such as found in Nigeria and other developing countries become critical. The objective of this work, therefore, is to understand these challenges, present empirical evidence through a questionnaire survey, rank these challenges in order of severity, and propose solutions.

Attitudinal Trajectories in an Online CS1 Class: Demographic and Performance Trends

David A. Joyner, Lily Bernstein, Maria-Isabelle Dittamo, Ben Engelman, Alysha Naran, Amber Ott, Jasmine Suh and Abby Thien

Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE)

Download Full Paper

Abstract: In this research, we investigate the trajectory of attitudinal change towards computer science among students in an online CS1 class. We perform this investigation to address several trends in modern computer science education. First, as computer science increasingly becomes a required class for all majors, how do students’ first experiences with the subject impact their attitudes? Second, as online education continues to expand, how does enrolling in CS1 online specifically affect audiences that may be marginalized in both CS classes and in online learning environments, such as women and underrepresented minorities? Third, can we intervene to improve attitudes towards computer science, especially among those marginalized audiences? In this research, we poll students in an online for-credit CS1 class four times to observe the change in their attitudes towards computer science over time and intervene with some students to try to improve their perception of computer science. We find that attitudes towards computer science improve with initial exposure, that women’s attitudes towards CS begin less positive but follow the same trajectory, and that mid-semester regression in attitudes toward computer science may predict eventual struggles to perform well in the class.

Jill Watson SA: Design and Evaluation of a Virtual Agent to Build Communities Among Online Learners

Qiaosi Wang, Shan Jing, Ida Camacho, David A. Joyner, and Ashok Goel

Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems

Download Full Paper

Abstract: Despite being accessible and affordable, online education presents numerous challenges for online learners due to the absence of face-to-face interactions. Lack of community-belongingness, in particular, negatively impacts online learners’ learning outcomes and learning experience. To help online learners build communities and foster connections with their peers, we designed and deployed Jill Watson SA (stands for Social Agent). Jill Watson SA is a virtual agent who can match students with shared identity, defined by similarities in location, timezone, hobby, class schedule, etc., on the Piazza class discussion forum. We implemented Jill Watson SA in two online classes and conducted three short surveys with online students to evaluate Jill Watson SA.

Enrollment Motivations in an Online Graduate CS Program: Trends & Gender- and Age-Based Differences

Alex Duncan, Bobbie Eicher and David A. Joyner

Proceedings of the 51st ACM Technical Symposium on Computer Science Education

Download Full Paper

Abstract: Demand for CS education has risen, leading to numerous new programs, such as the rise of affordable online degrees. Research shows these programs meet an otherwise untapped audience of working professionals seeking graduate level CS education. In this study, we examine the motivations for enrollment among students in one such online MSCS program. Based responses to an open ended question, we develop a typology of motivations, including goals (e.g. career transition), opportunities (e.g. enrolling without taking time off work), and assurances that their goals will be met (e..g the program’s accreditation). We then issue a closed survey question to a new group of students to further explore these motivations. In this paper, we discuss both aggregate and demographic trends in motivations, including the different motivations of men and women and what they imply about the program’s impact on the gender divide in computing. We also examine older students’ tendency towards intrinsic motivation to pursue an MSCS degree.

Advising of the Students, by the Students, for the Students: The Case of a Student-Owned Peer Advising Community

Alex Duncan and David A. Joyner

2020 Conference Proceedings of the Hawaii International Conference on Education

Download Full Paper

Abstract: Peer advising, where students receive advice from other students about courses, can be difficult to study due to its private, spontaneous nature or to potential response bias. However, peer advising can help create and strengthen student communities, and the information students provide can help improve course quality. We first present a case study on a student-run course review website developed to support an online graduate program in computer science. We discuss the evolution of the website and examine its usefulness as a student-organized community providing peer advising at scale. In our second study, we develop a coding scheme and use it to analyze reviews from the website. Although students provide mostly evaluative information, they also provide advice, context for their reviews, course descriptions, and feedback for the instructional team. This research explores the importance of student-organized communities in higher education and provides useful insights into peer advising at scale.

Eroding Investment in Repeated Peer Review: A Reaction to Unrequited Aid?

David A. Joyner and Alex Duncan

2020 Conference Proceedings of the Hawaii International Conference on Education

Download Full Paper

Abstract: Peer evaluation in educational contexts has been well-researched, but there are open questions about the trends students follow over the course of repeated peer evaluation activities within the same course or semester. We hypothesize a trend whereby reviewers who initially heavily invest in giving their classmates strong peer reviews lower their performance over time due in part to disillusionment with the lower-quality feedback they receive. To test this hypothesis, we investigated a dataset of over 50,000 peer assignment evaluations gathered from three semesters of two different courses, totaling over 79 assignments. We examine whether class performance across four quantitative variables drops over the course of the semester, and whether those drops are specifically more prevalent among high-performing reviewers. We find evidence that reviewers who begin the semester committed suffer a greater drop in
performance over time, and propose potential causal mechanisms for this drop as well as plans to potentially prevent it.

2019

Surveying the MOOC Data Set Universe

James J. Lohse, Christine A. McManus and David A. Joyner

Proceedings of Learning With MOOCs 2019

Download Full Paper

Abstract: This paper is a survey of the availability of open data sets generated from Massively Open Online Courses (MOOCs). This log data allows researchers to analyze and predict student performance. Often, the goal of the analysis is to focus on at-risk students who are not likely to finish a course. There is a growing gap between the average researcher (who does not have access to proprietary data) and the ready availability of data sets for analysis. Most research papers studying and predicting student performance in MOOCs are done on proprietary data sets that are not anonymized (deidentified) or released for general study. There are no standardized tools that provide a gateway to access usable data sets; instead, the researcher must navigate a maze of sites with different data structures and varying data access policies. To our knowledge, no open data sets are being produced, and have not been since 2016. The authors survey the history of MOOC data sharing, identify the few available open data sets, and discuss a path forward to increase the reproducibility of MOOC research.

Annotation-free Automatic Examination Essay Feedback Generation

Filipe Altoe and David A. Joyner

Proceedings of Learning With MOOCs 2019

Download Full Paper

Abstract: Many of the demands of scale appear at odds with quality, such as in the case of online MOOC degree programs. This paper presents research that focuses on reducing this dichotomy. Examination essays are important learning tools in for-credit courses. Psychology studies show they drive deeper learning strategies and invoke higher levels of cognitive processing in students. High-quality feedback is another critical aspect of students’ learning in advanced education. Key components of high-quality feedback are how expeditiously feedback is presented and how unambiguous it is. Examination essays are time-consuming for educators to grade, and at scale this may create an incentive for superficial feedback that may also reach students in a less than ideal turnaround time. This imposes clear scalability constraints and a potential decrease in course quality in online for-credit MOOC programs. We propose an annotation-free artificial intelligence-based approach for the automatic generation of examination essay rubrics and its subsequent utilization as part of high-quality feedback to students. It utilizes a combination of the natural language processing techniques TextRank and semantic similarity for automatic rubric generation and concept map creation for feedback presentation. The feedback is immediately available upon essay submission and offers the student a conceptual analysis of the submitted essay against the assignment’s learning objectives.

Creating Cadence: Fostering Persistent Engagement in Asynchronous Online Courses

David A. Joyner

Engaged Student Learning: Essays on Best Practices in the University System of Georgia

Download Full Paper

Abstract: The current boon of affordable, scalable online degrees is supported by the modern internet as a medium to construct high-quality, fully asynchronous learning experiences. The asynchronous nature of these programs accommodates working professionals who are unable to carve out consistent and prescribed times for the pursuit of a degree, but who nonetheless have the dedication and ability to succeed at the programs’ content. However, these classes’ asynchronous nature breaks some of students’ assumptions about the structure of college courses. Scheduled meetings do more than just support disseminating lecture material or facilitating synchronous activities; they establish a classroom cadence and set students’ expectations for the pace and routine of the course. Without meetings, these expectations are lost. Moreover, the persistent availability of asynchronous material removes scarcity from the experience, and while this is one of the medium’s strengths, it may also lure students into unhealthy procrastination. Thus, asynchronous online classes must actively accomplish that which traditional classes accomplish passively through required lecture attendance: creating cadence and incentivizing persistent engagement.

Master’s at Scale: Five Years in a Scalable Online Graduate Degree

David A. Joyner and Charles Isbell

Proceedings of the Sixth (2019) ACM Conference on Learning @ Scale

Download Full Paper

Abstract: In 2014, Georgia Tech launched the first for-credit MOOC-based graduate degree program. In the five years since, the program has proven generally successful, enrolling over 14,000 unique students, and several other similar programs have followed in its footsteps. Existing research on the program has focused largely on details of individual classes; program-level research, however, has been scarce. In this paper, we delve into the program-level details of an at-scale Master’s degree, from the story of its creation through the data generated by the program, including the numbers of applications, admissions, matriculations, and graduations; enrollment details including demographic information and retention patterns; trends in student grades and experience as compared to the on-campus student body; and alumni perceptions. Among our findings, we note that the program has stabilized at a retention rate of around 70%; that the program’s growth has not slowed; that the program has not cannibalized its on-campus counterpart; and that the program has seen an upward trend in the number of women enrolled as well as a persistently higher number of underrepresented minorities than the on-campus program. Throughout this analysis, we abstract out distinct lessons that should inform the development and growth of similar programs.

Synchronous at Scale: Investigation and Implementation of a Semi-Synchronous Online Lecture Platform

Denise Kutnick and David A. Joyner

Proceedings of the Sixth (2019) ACM Conference on Learning @ Scale

Download Full Paper

Abstract: Online classes and degree programs continue to grow in popularity, in part due to the increased convenience and accessibility of education that technology has provided in recent years. As online education scales upwards and outwards, there is an increased need to provide students with an engaging and collaborative learning experience. In some online learning environments, student collaboration is perceived to be more difficult than it is in a physical classroom setting due to cultural or geographic distance between students. In particular, online class lectures often lack the collaborative spirit seen in most in-person classroom lectures. To improve upon the online classroom experience, this project first examines the benefits and drawbacks of several in-person and online lecture delivery techniques, then proposes an online lecture platform that allows students to facilitate their own collaborative classrooms on-demand through a semi-synchronous viewing area and chatroom.

Peer Advising at Scale: Content and Context of a Learner-Owned Course Evaluation System

Alex Duncan and David A. Joyner

Proceedings of the Sixth (2019) ACM Conference on Learning @ Scale

Download Full Paper

Abstract: Peer advising in education, which involves students providing fellow students with course advice, can be important in online student communities and can provide insights into potential course improvements. We examine reviews from a course review web site for online graduate programs. We develop a coding scheme to analyze the free text portion of the reviews and integrate those findings with students’ quantitative ratings of each course’s overall score, difficulty, and workload. While reviews focus on subjective evaluation of courses, students also provide feedback for instructors, personal context, advice for other students, and objective course descriptions. Additionally, the average review varies by course overall score, difficulty, and workload. Our research examines the importance of student communities in online education and peer advising at scale.

Designing and Developing Video Lessons for Online Learning: A Seven-Principle Model

Chaohua Ou, David A. Joyner and Ashok Goel

Online Learning, 23(2)

Download Full Paper

Abstract: Despite the ubiquitous use of instructional videos in both formal and informal learning settings, questions remain largely unanswered on how to design and develop video lessons that are often used as the primary method for delivering instruction in online courses. In this study, we experimented with a model of seven principles drawn from instructional design theories for designing and developing video lessons for an online graduate course. Feedback was collected from students through surveys on their perceptions of the effectiveness of the video lessons and the overall course quality for eight semesters. This paper shares the instructors’ experience on the design and development of the video lessons as well as the survey findings. Implications of the findings for instructional design and future research are also discussed.

Replicating and Unraveling Performance and Behavioral Differences between an Online and a Traditional CS Course

David A. Joyner and Melinda McDaniel

Proceedings of the ACM Conference on Global Computing Education (CompEd ’19)

Download Full Paper

Abstract: In January 2017, a major public research university launched an online version of CS1 targeted at on-campus students to address rising enrollments and provide students with flexibility in their schedules. Prior research on this class has found positive outcomes: students in the course achieve the same learning outcomes as those in a traditional course, while reporting a lower time investment to reach those outcomes and a high level of student satisfaction. This research builds on that prior work in two ways. First, it replicates the findings from that earlier semester with an entirely new semester of students. Second, it delves deeper into the student experience within the online course and its traditional counterpart. This deeper analysis focuses specifically on the differing ways in which students in each section allocated their time, whether or not students in either section accessed the opposite section’s material, and their future preferences in online vs. residential CS classes.

Five Years of Graduate CS Education Online and at Scale

David A. Joyner, Charles Isbell, Thad Starner, and Ashok Goel

Proceedings of the ACM Conference on Global Computing Education (CompEd ’19)

Download Full Paper

Abstract: In 2014, Georgia Tech launched an online campus for its Master of Science in Computer Science program. The degree, equal in stature and accreditation to its on-campus counterpart, offered a notably lower cost of attendance. Its design emphasized flexibility in both geography and time, allowing students from around the world to earn a highly-ranked MSCS without taking time off work or moving to campus. Five years later, the program enrolls over 8000 students per semester and has graduated 1500 alumni. It is believed to be the largest program of its kind and has received recognition from national organizations on professional education. Existing research on the program has focused on challenges and opportunities to scale that are agnostic to the content itself. In this reflection, we look at the creation and growth of the program as it relates to graduate-level CS instruction. In particular, we note a unique and powerful unity of content and platform: the online delivery of the program dovetails with the technical skillsets of the professors and students that it draws, putting both in the position to contribute and innovate.

Building Purposeful Online Learning: Outcomes from Blending CS1

David A. Joyner

Blended Learning in Practice

Download Full Paper

Abstract: This chapter describes a blended learning class used to teach a large Computer Science 1 course that serves 1000 students per year and is a bottleneck course that limits how many students can enroll be accepted into computing majors. The course uses a MOOC to provide lectures, practice, and assignments. In addition, students could get help from three sources: recitation with teaching assistants, a help desk, or office hours with the instructor. The chapter discusses the production of the MOOC, which was designed to be adaptable to multiple types of intro computing courses for multiple majors. The design team also surveyed online Intro to Computer Science courses and share the best practices that they experienced.

This course is blended – at least for some students. Based on the MIX taxonomy, the required parts of the course are delivered entirely online. The online components include content delivery and content application with feedback through Vocareum and McGraw-Hill’s adaptive Smartbook. The optional parts of the class provide in-person content delivery and feedback during content application. During recitation, teaching assistants provided lectures and guidance on activities. For additional guidance on activities, the students could go to the help desk or the instructor’s office hours.

The research method is a quasi-experimental design. The two experimental groups, students who took the face-to-face class and students who took the blended class, took the class during the same semester. Student performance was measured through a standardized computer science assessment. Student perceptions, demographic information, and expectations for the course were measured through surveys.

Collaboration versus Cheating: Reducing Code Plagiarism in an Online MS Computer Science Program

Tony Mason, Ada Gavrilovska and David A. Joyner

Proceedings of the 50th ACM Technical Symposium on Computer Science Education

Download Full Paper

Abstract: We outline how we detected programming plagiarism in an introductory online course for a master’s of science in computer science program, how we achieved a statistically significant reduction in programming plagiarism by combining a clear explanation of university and class policy on academic honesty reinforced with a short but formal assessment, and how we evaluated plagiarism rates before and after implementing our policy and assessment.

From Clusters to Content: Using Code Clustering for Course Improvement

David A. Joyner, Ryan Arrison, Mehnaz Ruksana, Evi Salguero, Zida Wang, Ben Wellington and Kevin Yin

Proceedings of the 50th ACM Technical Symposium on Computer Science Education

Download Full Paper

Abstract: Large undergraduate CS courses receive thousands of code submissions per term. To help make sense of the large quantities of submissions, projects have emerged to dynamically cluster student submissions by approach for writing scalable feedback, tailoring hints, and conducting research. However, relatively little attention has been paid to the value of these tools for informing revision to core course materials and delivery methods. In this work, we applied one such technology—Sense, the eponymous product of its company—to an online CS1 class delivered simultaneously for credit to on-campus students and for free to MOOC students. Using Sense, we clustered student submissions to around 70 problems used throughout the course. In this work, we discuss the value of such clustering, the surprising trends we discovered through this process, and the changes made or planned to the course based on the results. We also discuss broader ideas on injecting clustering results into course design.

Online or In Person? Student MOtivations in the Choice of a CS1 Experience

Melinda McDaniel and David A. Joyner

Proceedings of the 50th ACM Technical Symposium on Computer Science Education

Download Full Paper

Abstract: As online offerings have matured and expanded, new efforts have recently been devoted to opening fully-accredited online versions of traditional on-campus classes. Such classes may be offered to on-campus students for greater flexibility during busy semesters, or to allow them to continue to make progress toward their degrees during internships or semesters away from campus. This trend intersects with a growing CS for All movement that sees more and more non-computer science majors enrolling in CS classes. As online offerings expand, it is important for us to understand who enrolls in online sections and the reasons for their choices, both to make sure that learning outcomes are similar across different delivery mechanisms, and to take advantage of opportunities to tailor course content to specific audiences. In this analysis, we look at two versions, one online and one traditional, of a CS1 class offered at a major public research university. We investigate demographic, motivational, and experiential components to identify which types of students are most likely to select each version and what implications this decision has for student success and course design.

2018

Toward CS1 at Scale: Building and Testing a MOOC-for-Credit Candidate

David A. Joyner

Proceedings of the Fifth Annual ACM Conference on Learning at Scale

Download Full Paper

Abstract: If a MOOC is to qualify for equal credit as an existing on-campus offering, students must achieve comparable outcomes, both educational and attitudinal. We have built a MOOC for teaching CS1 with the intent of offering it for degree credit. To test its eligibility for credit, we delivered it as an online for-credit course for two semesters to 197 on-campus students who selected the online version rather than a traditional version. We compared the demographics, outcomes, and experiences of these students to the 715 students in the traditional version. We found the online students more likely to be older; to be underrepresented minorities; and to have previously failed a CS class. We then found that our online students attained comparable learning outcomes to students in the traditional section. Finally, we found that our online students perceived the online course quality more positively and required less time to achieve those comparable learning outcomes.

Squeezing the Limeade: Policies and Workflows for Scalable Online Degrees

David A. Joyner

Proceedings of the Fifth Annual ACM Conference on Learning at Scale

Download Full Paper

Abstract: In recent years, non-credit options for learning at scale have outpaced for-credit options. To scale for-credit options, workflows and policies must be devised to preserve the characteristics of accredited higher education—such as the presumption of human evaluation and an assertion of academic integrity—despite increased scale. These efforts must follow as well with shifting from offering isolated courses (or informal collections thereof) to offering full degree programs with additional administrative elements. We see this shift as one from Massive Open Online Courses (MOOCs) to Large, Internet-Mediated Asynchronous Degrees (Limeades). In this work, we perform a qualitative research study on one such program that has scaled to 6,500 students while retaining full accreditation. We report a typology of policies and workflows employed by the individual classes to deliver this experience.

Intelligent Evaluation and Feedback in Support of a Credit-Bearing MOOC

David A. Joyner

Artificial Intelligence in Education
19th International Conference Proceedings

Download Full Paper

Abstract: Massive Open Online Courses (MOOCs) may reach a massive number of people, but few MOOCs count for credit. Scaling rigorous assessment, feedback, and integrity checks presents difficulties. We implemented an AI system for a CS1 MOOC-for-credit to address both scale and endorsement. In this analysis, we present the design of the system and an evaluation of the course. We observe that students in the online course achieve comparable learning outcomes, report a more positive student experience, and identify AI-equipped programming problems as the primary contributor to their experiences.

Sentiment Analysis of Student Evaluations of Teaching

Heather Newman and David A. Joyner

Artificial Intelligence in Education
19th International Conference Proceedings

Download Full Paper

Abstract: We used a sentiment analysis tool, VADER (Valence Aware Dictionary and sEntiment Reasoner), to analyze Student Evaluations of Teaching (SET) of a single course from three different sources: official evaluations, forum comments from another course, and an unofficial “reviews” site maintained by students. We compared the positive and negative valences of these sites; identified frequently-used key words in SET comments and determined the impact on positivity/negativity of comments that included them; and determined positive/negative values by question on the official course SET comments. Many universities use similar questions, which may make this research useful for those analyzing comments at other institutions. Previous published studies of sentiment analysis in SET settings are rare.

2017

Scaling Expert Feedback: Two Case Studies

David A. Joyner

Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale

Download Full Paper

Abstract: Traditionally, education relies on a linear relationship between enrollment and staff; rising enrollment dictates increases to staff with some expertise (such as teaching assistants, TAs) for evaluation. This relationship is expensive, so learning at scale has largely deemphasized expert evaluation and feedback. Two organizations, though, have used different models to scale up class size online while retaining this expert evaluation and feedback. In this paper, we analyze the methods these two organizations have used to increase enrollment while preserving scalability and feedback. We observe an academic program has scaled feedback with traditional TAs by relying on unique characteristics of its student body, while a commercial program has done so with a novel, network-based model. These successes show the potential of learning from experts at scale.

Congruency, Adaptivity, Modularity, and Personalization: Four Experiments in Teaching Introduction to Computing

David A. Joyner

Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale

Download Full Paper

Abstract: In January 2017, Georgia Tech launched a new online section of its CS1301: Introduction to Computing class. The course, offered both as a for-credit course to on-ground students and as an open MOOC, built on four unique design principles: congruency, adaptivity, modularity, and personalization. In this short paper, we describe the background of the course, the definitions of these design principles, and their application to the course design.

2016

Graders as Meta-Reviewers: Simultaneously Scaling and Improving Expert Evaluation for Large Online Classrooms

David A. Joyner, Wade Ashby, Liam Irish, Yeeling Lam, Jacob Langston, Isabel Lupiani, Mike Lustig, Paige Pettoruto, Dana Sheahen, Angela Smiley, Amy Bruckman, and Ashok Goel

Proceedings of the Third (2016) ACM Conference on Learning @ Scale

Download Full Paper

Abstract: Large classes, both online and residential, typically demand many graders for evaluating students’ written work. Some classes attempt to use autograding or peer grading, but these both present challenges to assigning grades at for-credit institutions, such as the difficulty of autograding to evaluate free-response answers and the lack of expert oversight in peer grading. In a large, online class at Georgia Tech in Summer 2015, we experimented with a new approach to grading: framing graders as meta-reviewers, charged with evaluating the original work in the context of peer reviews. To evaluate this approach, we conducted a pair of controlled experiments and a handful of qualitative analyses. We found that having access to peer reviews improves the perceived quality of feedback provided by graders without decreasing the graders’ efficiency and with only a small influence on the grades assigned.

The Unexpected Pedagogical Benefits of Making Higher Education Accessible

David A. Joyner, Ashok Goel, and Charles Isbell

Proceedings of the Third (2016) ACM Conference on Learning @ Scale

Download Full Paper

Abstract: Many ongoing efforts in online education aim to increase accessibility through affordability and flexibility, but some critics have noted that pedagogy often suffers during these efforts. In contrast, in the low-cost for-credit Georgia Tech Online Masters of Science in Computer Science (OMSCS) program, we have observed that the features that make the program accessible also lead to pedagogical benefits. In this paper, we discuss the pedagogical benefits, and draw a causal link between those benefits and the factors that increase the program’s accessibility.

Expert Evaluation of 300 Projects Per Day

David A. Joyner

Proceedings of the Third (2016) ACM Conference on Learning @ Scale

Download Full Paper

Abstract: In October 2014, one-time MOOC developer Udacity completed its transition from primarily producing massive, open online courses to producing job-focused, project-based microcredentials called “Nanodegree” programs. With this transition came a challenge: whereas MOOCs focus on automated assessment and peer-to-peer grading, project-based microcredentials would only be feasible with expert evaluation. With dreams of enrolling tens of thousands of students at a time, the major obstacle became project evaluation. To address this, Udacity developed a system for hiring external experts as project reviewers. A year later, this system has supported project evaluation on a massive scale: 61,000 projects have been evaluated in 12 months, with 50% evaluated within 2.5 hours (and 88% within 24 hours) of submission. More importantly, students rate the feedback they receive very highly at 4.8/5.0. In this paper, we discuss the structure of the project review system, including the nature of the projects, the structure of the feedback, and the data described above.

TAPS: A MOSS Extension for Detecting Software Plagiarism at Scale

Dana Sheahen and David A. Joyner

Proceedings of the Third (2016) ACM Conference on Learning @ Scale

Download Full Paper

Abstract: Cheating in computer science classes can damage the reputation of institutions and their students. It is therefore essential to routinely authenticate student submissions with available software plagiarism detection algorithms such as Measure of Software Similarity (MOSS). Scaling this task for large classes where assignments are repeated each semester adds complexity and increases the instructor workload. The MOSS Tool for Addressing Plagiarism at Scale (MOSS-TAPS), organizes the MOSS submission task in courses that repeat coding assignments. In a recent use-case in the Online Master of Science in Computer Science (OMSCS) program at the Georgia Institute of Technology, the instructor time spent was reduced from 50 hours to only 10 minutes using the managed submission tool design presented here. MOSS-TAPS provides persistent configuration, supports a mixture of software languages and file organizations, and is implemented in pure Java for cross-platform compatibility.

Designing Videos with Pedagogical Strategies: Online Students’ Perceptions of Their Effectiveness

Chaohua Ou, Ashok Goel and David A. Joyner

Proceedings of the Third (2016) ACM Conference on Learning @ Scale

Download Full Paper

Abstract: Despite the ubiquitous use of videos in online learning and enormous literature on designing online learning, there has been relatively little research on what pedagogical strategies should be used to make the most of video lessons and what constitutes an effective video for student learning. We experimented with a model of incorporating four pedagogical strategies, four instructional phases, and four production guidelines-in designing and developing video lessons for an online graduate course. In this paper, we share our experience as well as students’ perceptions of their effectiveness. We also discuss what needs to be done for future research.

Design of an Online Course on Knowledge-Based AI

Ashok Goel and David A. Joyner

Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence

Download Full Paper

Abstract: In Fall 2014 we offered an online course on Knowledge-Based Artificial Intelligence (KBAI) to about 200 students as part of the Georgia Tech Online MS in CS program. By now we have offered the course to more than 1000 students. We describe the design, development and delivery of the online KBAI class in Fall 2014.

2015

Using Human Computation to Acquire Novel Methods for Addressing Visual Analogy Problems on Intelligence Tests

David A. Joyner, Darren Bedwell, Chris Graham, Warren Lemmon, Oscar Martinez, and Ashok Goel

Proceedings of the Sixth International Conference on Computational Creativity

Download Full Paper

Abstract: The Raven’s Progressive Matrices (RPM) test is a commonly used test of intelligence. The literature suggests a variety of problem-solving methods for addressing RPM problems. For a graduate-level artificial intelligence class in Fall 2014, we asked students to develop intelligent agents that could address 123 RPM-inspired problems, essentially crowdsourcing RPM problem solving. The students in the class submitted 224 agents that used a wide variety of problem-solving methods. In this paper, we first report on the aggregate results of those 224 agents on the 123 problems, then focus specifically on four of the most creative, novel, and effective agents in the class. We find that the four agents, using four very different problem-solving methods, were all able to achieve significant success. This suggests the RPM test may be amenable to a wider range of problem- solving methods than previously reported. It also suggests that human computation might be an effective strategy for collecting a wide variety of methods for creative tasks.

Impact of a Creativity Support Tool on Student Learning About Scientific Discovery Processes

Ashok Goel and David A. Joyner

Proceedings of the Sixth International Conference on Computational Creativity

Download Full Paper

Abstract: Science education nowadays emphasizes authentic science practices mimicking the creative discovery processes of real scientists. How, then, can we build creativity support tools for student learning about scientific discovery processes? We summarize several epistemic views of ideation in scientific discovery and find that the ideation techniques provide few guarantees of correctness of scientific hypotheses, indicating the need for supporting hypothesis evaluation. We describe an interactive tool called MILA−S that enables students to elaborate hypotheses about ecological phenomena into conceptual models and evaluate conceptual models through agent-based simulations. We report on a pilot experiment with 50 middle school students who used MILA−S to discover causal explanations for an ecological phenomenon. Preliminary results from the study indicate that use of MILA–S had a significant impact both on the creative process of model construction and the nature of the constructed models. We posit that the computational support for model construction, evaluation and revision embodied in MILA–S fosters student creativity in learning about scientific discovery processes.

Organizing Metacognitive Tutoring Around Functional Roles of Teachers

David A. Joyner and Ashok Goel

Proceedings of the 37th Annual Cognitive Science Society Meeting

Download Full Paper

Abstract: Metacognitive skills are critical in learning but difficult to teach. Thus the question becomes how can we facilitate metacognitive tutoring? We present an exploratory learning environment called MILA-T with embedded metacognitive tutors imitating five functional roles of teachers in classrooms. We tested MILA–T in a controlled experiment with 237 middle school students. We examine the impact of MILA–T on the models of a natural phenomenon constructed by the students. We find that students with access to MILA–T wrote better evidential justifications for their models, and thus, deliver better-justified models for the phenomenon. We also find that these improvements persisted during a transfer task. These results lend support for organizing metacognitive tutoring around the functional roles of teachers for supporting inquiry-driven modeling.

Improving Inquiry-Driven Modeling in Science Education through Interaction with Intelligent Tutoring Agents

David A. Joyner and Ashok Goel

Proceedings of the 20th International Conference on Intelligent User Interfaces

Download Full Paper

Abstract: This paper presents the design and evaluation of a set of intelligent tutoring agents constructed to teach teams of students an authentic process of inquiry-driven modeling. The paper first presents the theoretical grounding for inquiry-driven modeling as both a teaching strategy and a learning goal, and then presents the need for guided instruction to improve learning of this skill. However, guided instruction is difficulty to provide in a one-to-many classroom environment, and thus, this paper makes the case that interaction with a metacognitive tutoring system can help students acquire the skill. The paper then describes the design of an exploratory learning environment, the Modeling and Inquiry Learning Application (MILA), and an accompanying set of metacognitive tutors (MILA–T). These tools were used in a controlled experiment with 84 teams (237 total students) in which some teams received and interacted with the tutoring system while other teams did not. The effect of this experiment on teams’ demonstration of inquiry-driven modeling are presented.

Before 2015

Attitudinal Gains from Engagement with Metacognitive Tutors in an Exploratory Learning Environment

David A. Joyner and Ashok Goel

Intelligent Tutoring Systems 12th International Conference Proceedings

Download Full Paper

Abstract: MILA–T (MILA–Tutoring) is constructed to give students explicit instruction on scientific modeling and inquiry, intending in part to help cultivate positive attitudes toward science. The results of a two-week controlled experiment using MILA–T in middle school classroom show a significant effect of MILA–T on students’ attitudes towards science.

MILA–S: Generation of Agent-Based Simulations from Conceptual Models

David A. Joyner, Ashok Goel, and Nicholas Papin

Proceedings of the 19th International Conference on Intelligent User Interfaces

Download Full Paper

Abstract: Scientists use both conceptual models and executable simulations to help them make sense of the world. Models and simulations each have unique affordances and limitations, and it is useful to leverage their affordances to mitigate their respective limitations. One way to do this is by generating the simulations based on the conceptual models, preserving the capacity for rapid revision and knowledge sharing allowed by the conceptual models while extending them to provide the repeated testing and feedback of the simulations. In this paper, we present an interactive system called MILAfiS for generating agent-based simulations from conceptual models of ecological systems. Designed with STEM education in mind, this user-centered interface design allows the user to construct a Component-Mechanism-Phenomenon conceptual model of a complex system, and then compile the conceptual model into an executable NetLogo simulation. In this paper, we present the results of a pilot study with this interface with about 50 middle school students in the context of learning about ecosystems.

MILA–S: Generation of Agent-Based Simulations from Conceptual Models

David A. Joyner, Ashok Goel, and Nicholas Papin

Proceedings of the 19th International Conference on Intelligent User Interfaces

Download Full Paper

Abstract: Scientists use both conceptual models and executable simulations to help them make sense of the world. Models and simulations each have unique affordances and limitations, and it is useful to leverage their affordances to mitigate their respective limitations. One way to do this is by generating the simulations based on the conceptual models, preserving the capacity for rapid revision and knowledge sharing allowed by the conceptual models while extending them to provide the repeated testing and feedback of the simulations. In this paper, we present an interactive system called MILAfiS for generating agent-based simulations from conceptual models of ecological systems. Designed with STEM education in mind, this user-centered interface design allows the user to construct a Component-Mechanism-Phenomenon conceptual model of a complex system, and then compile the conceptual model into an executable NetLogo simulation. In this paper, we present the results of a pilot study with this interface with about 50 middle school students in the context of learning about ecosystems.

Facilitating Authentic Reasoning about Complex Systems in Middle School Science Education

David A. Joyner, David Majerich, and Ashok Goel

2013 Conference on Systems Engineering Research

Download Full Paper

Abstract: In order to tackle problems in the modern world, individuals must possess a strong ability to reason about and understand complex systems in a practical and useful way. Past research has indicated that experts and novices possess fundamentally different kinds of understanding of complex systems. Therefore, to adequately prepare students to address problems pertaining to complex systems, it is important to help them acquire an authentic expert- like understanding of these systems. We approach this problem from two angles: first, we create an interactive environment in which students may investigate complex systems in a manner similar to that of scientists and engineers; secondly, we embed metacognitive agents in the interactive environments such that the agents provide situated guidance towards expert-like understanding of complex systems. In this paper, we detail the design of these two systems, referred to as MILA and MeTA respectively, and the way in which they help students obtain a more authentic and advanced understanding of complex systems. We also briefly describe the deployments of these tools in a science summer camp for middle school students and preliminary results of students’ interaction with them.

Evolution of an Integrated Technology for Supporting Learning about Complex Systems

David A. Joyner, Ashok Goel, Spencer Rugaber, Cindy Hmelo-Silver, and Rebecca Jordan

2011 IEEE 11th International Conference on Advanced Learning Technologies

Download Full Paper

Abstract: In this paper, we describe the evolution of an interactive technology called the Ecological Modeling Toolkit (EMT) that supports learning about complex ecological systems in middle school science. Authentic learning of science is facilitated by imitation, rehearsal and understanding of real-world scientific practices such as observation, experimentation, problem formulation, hypothesis testing, and model construction and revision. We illustrate how the tools in EMT work together to support many real-world scientific practices such as model construction, simulation and revision, and scaffold others such as observation, problem formulation and hypothesis testing.

Functional and Causal Abstractions of Complex Systems

Ashok Goel, Swaroop Vattam, Spencer Rugaber, David A. Joyner, Cindy Hmelo-Silver, Rebecca Jordan, Sameer Honwad, Steven Gray, and Suparna Sinha

Proc. of the 32nd Annual Meeting of the Cognitive Science Society

Download Full Paper

Abstract: Structure-Behavior-Function (SBF) models of complex systems use functions as abstractions to organize knowledge of structural components and causal processes in a system. We describe an interactive learning environment called ACT (Aquarium Construction Toolkit) for constructing simple SBF models of classroom aquaria, and report on a case study on the use of SBF thinking and the ACT tool in middle school science classes. We present initial data indicating that SBF thinking supported in part by the ACT tool leads to enhanced understanding of functions and behaviors of aquaria.

Connecting the Visible to the Invisible: Helping Middle School Students Understand Complex Ecosystem Processes

Sameer Honwad, Cindy Hmelo-Silver, Rebecca Jordan, Catherine Eberbach, Steven Gray, Suparna Sinha, Ashok Goel, Swaroop Vattam, Spencer Rugaber, and David A. Joyner

Proc. of the 32nd Annual Meeting of the Cognitive Science Society

Download Full Paper

Abstract: Learning about ecosystems is challenging because, like any complex system, they are simultaneously multidimensional and dynamic. Often, learners engage only with the visible components of an ecosystem and draw either single or linear causal connections between components. In this study, we explored how using a Structure-Behavior-Function framework supported middle school students’ conceptual and complex reasoning about the visible and invisible components of an ecosystem. Research shows that learners often engage only with the visible components of an ecosystem and draw linear/single causal connections between the components of the ecosystem. Our findings suggest that a combination of using structure, behavior, and function approach along with a set of carefully designed technology tools can push the students toward a better understanding of the ecosystem functioning. The results show that along with the visible components of the ecosystem, students have started to identify the invisible components of the ecosystem.

Tangible Optical Chess: A Laser Strategy Game on an Interactive Tabletop

David A. Joyner, Chih-Sung (Andy) Wu, Ellen Yi-Luen Do

Proceedings of the 8th International Conference on Interaction Design and Children

Download Full Paper

Abstract: This paper presents Tangible Tracking Table, an interactive tabletop display, and Optical Chess, a strategy game. We discuss the design and implementation of both systems and report our evaluation game play sessions with young adults, with a special focus on how the Tangible Tracking Table enhances interaction over a point-and-click interface.