The impact of prompting strategies on LLM-assisted assessment: Evaluative alignment and boundary conditions for student performance

Abstract

Large language models (LLMs) are increasingly applied in automated code assessment, yet their integration remains constrained by evaluative inconsistency and deficits in instructional specificity. To examine these challenges, this study employed a quasi-experimental design involving 93 novice programmers to investigate how prompting strategies relate to evaluative alignment, pedagogical characteristics and student feedback adoption. Analysis indicated that the few-shot prompting condition was associated with significantly higher scoring alignment with human instructors (90.5% versus 81.4%) and with generated feedback exhibiting greater criteria anchoring and actionability. Mixed-effects models revealed specific boundary conditions: the positive association of few-shot feedback with assignment scores and submission probability was most pronounced for students with moderate prior competence engaging in tasks of moderate complexity. Furthermore, mediation analysis indicated that perceived usefulness, rather than perceived ease of use, functioned as a significant correlate of substantive revision behaviours and subsequent code quality. These findings suggest that deploying LLMs in formative assessment requires context-specific prompt engineering and human-in-the-loop oversight to manage cognitive load, particularly for novice learners.

Implications for practice or policy:

• Educators can enhance artificial intelligence feedback quality by embedding explicit rubric criteria and concrete input-output exemplars into prompts.
• Instructors should mediate LLM outputs for novice programmers to prevent automation bias and manage high cognitive load.
• Institutions should mandate human-in-the-loop oversight protocols for automated grading systems to ensure evaluative fairness and accuracy.

Assessors must demonstrate the instrumental value of artificial intelligence feedback to students to drive substantive code revision behaviours.