An Automated Exam Marking Framework using an Artificial Intelligence Tool for English Writing in General Foundation Program

Abstract

This study proposes a fully automated exam marking framework for English writing in the General Foundation Program (GFP) through the use of prompt engineering with artificial intelligence (AI) tool ChatGPT and computer-based analysis. The AI tool generated 120 GFP B2 level writing responses (120 to 150 words each) with intentional grammatical errors. Four prompts were applied for automated marking by the AI tool as an examiner, based on a provided rubric, to detect errors, and awarding of a score [/10] (out of 10) to each response. A combination of different prompting techniques, including instruction-based, role-based, and formatting prompting were collectively used to design each prompt. All of the 120 responses were marked by each prompt. The first and second prompts were similar, just with a difference of paraphrasing. Prompts 3 and 4 were based on Prompt 2, containing only an addition of a single word for Prompt 3, and replacement of the added word with another word for Prompt 4. Statistical comparisons were done in SPSS. The Friedman test showed the similarities (Prompts 1 and 2 with paraphrased versions) and significant differences (Prompts 2, 3 and 4; p < .001, just due to an addition, and replacement of a single word) among the prompts outcomes based on the markings. Pairwise comparisons were performed using the Bonferroni correction to check for significant differences, while Kendall’s W test was used to measure the strength of agreement among the rankings. Human involvement in the study was limited to author-led validation of all the 120 responses, toward the accuracy of intentional added errors. The framework demonstrated how different prompts impacted the marking outcomes, which highlighted the sensitivity of the prompts to the addition of a single determinative word and the replacement of that word with another. Moreover, statistical comparisons showed how scalable assessment design and quality checks can be consistently applied for GFP English writing. This study showed that the proposed automated marking framework can support consistent, scalable, and prompt-sensitive assessment of GFP English writing.