Research

Academic Papers

RCTs can serve as a novel evaluation standard for knowledge tracing algorithms in real-world applications

January 2025 with Marius Köppel, Daniel Schunk, and Isabell Zipperle
Submitted to ACM FAccT
The individualization of learning contents using knowledge tracing algorithms within online learning environments promises large benefits – both for individual learners themselves and for society. However, it is far from clear how to best proceed when evaluating and optimizing knowledge tracing algorithms. More so, common evaluation approaches focus and restrict to testing of the model performance and do not evaluate the effectiveness with regard to real-world learning outcomes. We propose a comprehensive three-step evaluation approach for knowledge tracing algorithms including performance testing and a randomized controlled experiment trial (RCT) that we conduct on a large digital self-learning platform. We develop a knowledge tracing machine learning algorithm based on two convolutional neural networks (CNNs) that we use to assigns tasks to 4,365 learners based on historic learner-platform interactions. To test the effectiveness of our algorithm with regard to learning outcomes, learners are randomized into three groups: two treatment groups that get tasks assigned based on group-based and individual predictions of our algorithm and one control group that gets tasks assigned randomly. We analyze group differences with respect to the effort that learners provide and their performance on the platform. Even though our trained algorithm shows good performance in predicting learning outcomes on cross-validation splits and performs similar to commonly used knowledge tracing algorithms on prediction tasks in our and in benchmark datasets, we do not detect significant differences between the three randomized groups. Our results shed light on the importance of comprehensive evaluation of knowledge tracing algorithms and the multiple challenges associated with the algorithm-based individualization of learning paths.
Code

Predicting NOx Emissions in Biochar Production Plants Using Machine Learning

December 2024 with Marius Köppel, Tobias Schweitzer, Jochen Weber, Erdem Yilmaz, Niklas Witzig, Mattia Cerrato, Bernardo del Campo, Lissete Davila, and David Barrento
NeurIPS
The global Biochar Industry has witnessed a surge in biochar production, with a total of 350k mt/year production in 2023. With the pressing climate goals set and the potential of Biochar Carbon Removal (BCR) as a climate-relevant technology, scaling up the number of new plants to over 1000 facilities per year by 2030 becomes imperative. However, such a massive scale-up presents not only technical challenges but also control and regulation issues, ensuring maximal output of plants while conforming to regulatory requirements. In this paper, we present a novel method of optimizing the process of a biochar plant based on machine learning methods. We show how a standard Random Forest Regressor can be used to model the states of the pyrolysis machine, the physics of which remains highly complex. This model then serves as a surrogate of the machine — reproducing several key outcomes of the machine — in a numerical optimization. This, in turn, could enable us to reduce NOx emissions — a key regulatory goal in that industry — while achieving maximal output still. In a preliminary test our approach shows remarkable results, proves to be applicable on two different machines from different manufacturers, and can be implemented on standard Internet of Things (IoT) devices more generally.

Investment into Identity in the Field – Nudging Refugees’ Integration Effort

June 2023 with Nora Grote and Mario Scharfbillig
European Journal of Political Economics
Social identity greatly affects behavior. However, less is known about individual’s investment into identification, i.e. into belonging to a social group. We design a field experiment that allows us to make effort as an investment into a new group identity salient. The social identity in our treatment is refugee’s identification with the host society. We modified a mailing to 5600 refugees who use an online language-learning platform to learn the host countries’ language. These treatment emails make salient that improving the host country’s language ability increases the belonging to the host society. Our analysis reveals that the treatment has a significant positive effect on the effort exerted on the language-learning platform, leading to more completed exercises and more time spent learning the host country’s language. This suggests that refugees’ invest into being part of the host country’s society for its social identity component.

Rank Response Functions in an Online Learning Environment

December 2021 with Valentin Wagner and Isabell Zipperle
Economic Letters
We estimate rank response functions after receiving rank-order feedback in an online learning platform. We find that the shapes of the rank response functions depend on the outcome measure under consideration. For our effort measure, i.e., whether learners continue to interact with the platform, we can reject a linear rank response function and find significant evidence of a U-shaped relationship. For our performance measure, i.e., correctly solved exercises, we find no clear pattern overall but suggestive evidence for a linearly decreasing rank response function for individuals in the lower half of the ability distribution, i.e., the lower the rank the lower the performance.

Feedback in Homogeneous Ability Groups: A Field Experiment

June 2021
Relative performance feedback (RPF) often increases effort and performance on average. However, in the context of education, learners with low ability are observed to reduce effort and performance when RPF is provided. In a randomized field experiment with 7352 learners, we sort treated learners into anonymous homogeneous ability feedback groups. These learners receive RPF in groups of learners with the same ability level. Futher we observe a control group that does not receive feedback and a group that face RPF in anonymous heterogeneous ability feedback groups. We find that on average RPF increases learning effort by 0.11 standard deviations with no additional effect average effect of homogeneous ability groups. Further, we find that weak learners in homogeneous ability feedback groups learn insignificantly more and strong learners insignificantly less compared to learners in more heterogeneous groups. This is in line with a flatter skill formation curve or insignificantly less inequality due to feedback in homogeneous ability groups.d one control group. We analyze the difference between the three groups with respect to effort learners provide within the platform, their performance within the platform, formative assessments within the platform and their final high-stake standardized exams (summative assessment).

Shedding light on the effects of adaptive learning: A randomized field experiment in an online learning platform

Mai 2021 with Daniel Schunk
In the context of the digital transformation of teaching and learning, digitalization enables a high degree of adaptivity, thus potentially improving the effectiveness of education. However, conclusive evidence on the effect of adaptive digital teaching approaches on learning processes and learning success is still scarce. Here, we first develop an algorithm that operationalizes adaptive learning based on the difficulty levels of learning tasks. Then, we conduct a randomized controlled trial in the context of a large digital learning platform and compare a non-adaptive control group with two treatment groups that differ in their degree of adaptivity. We find that adaptivity significantly increased learning effort in both treatment groups but did not lead to better test scores in formative or summative assessments. We discuss reasons for these findings to shed light on the complex challenges associated with effectively operationalizing adaptive digital education approaches.

Conferences

  • Experimentation Elite: London, Jul 2023 and Dec 2024
  • Advances with Field Experiments: University of Chicago, Sep 2022
    Presenting “RCTs can serve as a novel evaluation standard for knowledge tracing algorithms in real-world applications”
  • Field Days: Experiments outside the laboratory, online, Dec 2020
    Presenting “Investment into Identity in the Field – Nudging Refugees’ Integration Effort”, online, Oct 2020
    Presenting “Feedback in Homogeneous Ability Groups: A Field Experiment”
  • Field Days and INSEAD RCT days: Experiments outside the laboratory, INSEAD Europe Fontainebleau, Nov 2019
    Presenting a poster on “Understanding Adaptive Learning with a Field Experiment”
  • Advances with Field Experiments, Boston University, Oct 2018
    Presenting “Preference for Identification in the Field – Nudging Refugees’ Integration Effort”
  • Nordic Conference on Behavioral and Experimental Economics, USD Odense, Sep 2018
    Presenting “Preference for Identification in the Field – Nudging Refugees’ Integration Effort”
  • Current Trends in Public Sector Research, Masaryk University, Brno, Jan 2018
    Presenting “Preference for Identification in the Field – Nudging Refugees’ Integration Effort”
  • International Workshop Economics of Education and Self-Regulation, University of Mainz, Oct 2015
    Participant