TY - JOUR
T1 - Predicting learning outcome in a first-year engineering course
T2 - 129th ASEE Annual Conference and Exposition: Excellence Through Diversity, ASEE 2022
AU - Castro, Laura Melissa Cruz
AU - Li, Tiantian
AU - Ciner, Leyla
AU - Douglas, Kerrie A.
AU - Brinton, Christopher Greg
N1 - Funding Information:
This material is based upon work supported by the Charles Koch Foundation.
Funding Information:
Dr. Kerrie Douglas, Assistant Professor of Engineering Education at Purdue, studies how to improve the quality of classroom assessments and evaluation of online learning in a variety of engineering education contexts. She holds a PhD in Educational Psychology and a M.A. in Educational Studies, with focus on school counseling. She is a co-PI on the SCALE project, leading the evaluation and assessment efforts. She recently received an NSF award to study engineering instructor decisions and student support during COVID-19 and impact the pandemic is having on engineering students. She also recently won the prestigious CAREER award from the U.S. National Science Foundation to study increasing the fairness of engineering assessments. In total, she has been on the leadership of more than $24 million dollars in research awards. Her research on evaluation of online learning (supported by two NSF awards #1544259,1935683, ) has resulted in more than 20 peer-reviewed conference and journal publications related to engineering learners in online courses. She was a FutureLearn Research Fellow from 2017-2019; a 2018 recipient of the FIE New Faculty Fellow Award and was the 2021 Program Chair for the Educational Research Methods Division of ASEE.
Publisher Copyright:
© American Society for Engineering Education, 2022.
PY - 2022/8/23
Y1 - 2022/8/23
N2 - First-year engineering courses are relatively large with several sections; thus, it can be rather difficult for an individual instructor to recognize when a particular student begins to lose engagement. Learning management systems (LMS) (e.g., Canvas, Blackboard, Brightspace) can be valuable tools to provide a consistent curriculum across several sections of a course and generate data regarding students' engagement with course materials. However, a human-centered approach to transform the data needs to be utilized to extract valuable insights from LMS data. The purpose of this Complete Research paper is to explore the following research questions: What type of LMS objects contain information to explain students' grades in a first-year engineering course? Is the inclusion of a human operator during the data transformation process significant to the analysis of learning outcomes? For this, data from LMS is used to predict the learning outcome of students in a FYE course. Two predictive models are compared. The first model corresponds to a usual predictive model, using the data from the LMS directly. The second model considers the specifics of the course, by transforming the data from aggregate user interaction to more granular categories related to the content of the class by a human operator. A logistic regression model is fitted using both datasets. The comparison between predictive measures such as precision, accuracy, and recall are then analyzed. The findings from the transformed dataset indicate that students' engagement with the career exploration curriculum was the strongest predictor of students' final grades in the course. This is a fascinating finding because the amount of weight the career assignments contributed to the overall course grade was relatively low. Additionally, while both models produced adequate fit indices, the human-informed model performed significantly better and resulted in more interpretable results.
AB - First-year engineering courses are relatively large with several sections; thus, it can be rather difficult for an individual instructor to recognize when a particular student begins to lose engagement. Learning management systems (LMS) (e.g., Canvas, Blackboard, Brightspace) can be valuable tools to provide a consistent curriculum across several sections of a course and generate data regarding students' engagement with course materials. However, a human-centered approach to transform the data needs to be utilized to extract valuable insights from LMS data. The purpose of this Complete Research paper is to explore the following research questions: What type of LMS objects contain information to explain students' grades in a first-year engineering course? Is the inclusion of a human operator during the data transformation process significant to the analysis of learning outcomes? For this, data from LMS is used to predict the learning outcome of students in a FYE course. Two predictive models are compared. The first model corresponds to a usual predictive model, using the data from the LMS directly. The second model considers the specifics of the course, by transforming the data from aggregate user interaction to more granular categories related to the content of the class by a human operator. A logistic regression model is fitted using both datasets. The comparison between predictive measures such as precision, accuracy, and recall are then analyzed. The findings from the transformed dataset indicate that students' engagement with the career exploration curriculum was the strongest predictor of students' final grades in the course. This is a fascinating finding because the amount of weight the career assignments contributed to the overall course grade was relatively low. Additionally, while both models produced adequate fit indices, the human-informed model performed significantly better and resulted in more interpretable results.
UR - http://www.scopus.com/inward/record.url?scp=85138241731&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85138241731&partnerID=8YFLogxK
M3 - Conference article
AN - SCOPUS:85138241731
SN - 2153-5965
JO - ASEE Annual Conference and Exposition, Conference Proceedings
JF - ASEE Annual Conference and Exposition, Conference Proceedings
Y2 - 26 June 2022 through 29 June 2022
ER -