“Medium-n studies" in computing education conferences

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Good (Frequentist) statistical practice requires that statistical tests be performed in order to determine if the phenomenon being observed could plausibly occur by chance if the null hypothesis is false. Good practice also requires that a test is not performed if the study is underpowered: if the number of observations is not sufficiently large to be able to reliably detect the effect one hypothesizes, even if the effect exists [7]. Running underpowered studies runs the risk of false negative results. This creates tension in the guidelines and expectations for computer science education conferences: while things are clear for studies with a large number of observations, researchers should in fact not compute p-values and perform statistical tests if the number of observations is too small [2]. The issue is particularly live in CSed venues, since class sizes where those issues are salient are common We outline the considerations for when to compute and when not to compute p-values in different settings encountered by computer science education researchers. We survey the author and reviewer guidelines in different computer science education conferences (ICER, SIGCSE TS, ITiCSE, EAAI, CompEd, Koli Calling). We present summary data and make several preliminary observations about reviewer guidelines: guidelines vary from conference to conference; guidelines allow for qualitative studies, and, in some cases, experience reports, but guidelines do not generally explicitly indicate that a paper should have at least one of (1) an appropriately-powered statistical analysis or (2) rich qualitative descriptions. We present preliminary ideas for addressing the tension in the guidelines between small-n and large-n studies.

Original languageEnglish (US)
Title of host publicationProceedings of 23rd International Conference on Computing Education Research, Koli Calling 2023
PublisherAssociation for Computing Machinery
ISBN (Electronic)9798400716539
StatePublished - Nov 13 2023
Externally publishedYes
Event23rd International Conference on Computing Education Research, Koli Calling 2023 - Hybrid, Koli, Finland
Duration: Nov 13 2023Nov 19 2023

Publication series

NameACM International Conference Proceeding Series


Conference23rd International Conference on Computing Education Research, Koli Calling 2023
CityHybrid, Koli

All Science Journal Classification (ASJC) codes

  • Human-Computer Interaction
  • Computer Networks and Communications
  • Computer Vision and Pattern Recognition
  • Software


Dive into the research topics of '“Medium-n studies" in computing education conferences'. Together they form a unique fingerprint.

Cite this