Scalable Oversight for Advanced AI: Recursive Reward Modeling and Debate

Authors

  • Raji N YuvakshatraInstitute of Management Studies (YIMS), Mundur, India Author

DOI:

https://doi.org/10.63090/EJAIR/3139.7336.0012

Keywords:

Scalable Oversight, Reinforcement Learning from Human Feedback, Recursive Reward Modelling, Iterated Amplification, AI Safety Via Debate, Weak-To-Strong Generalization, Sandwiching

Abstract

Reinforcement learning from human feedback trains language models against a reward model fitted to human comparisons, and it works only as long as the people giving feedback can tell a good output from a bad one. Scalable oversight is the problem of supervising systems whose outputs exceed that competence. This article reviews the two oldest proposals for the problem, recursive reward modelling with its close relative iterated amplification, and debate between competing models before a human judge, and places them beside two newer lines of work, weak-to-strong generalization and model-written critique. The review first sets out why direct feedback fails under optimization pressure, drawing on published evidence of reward hacking, reward model overoptimization, sycophancy and unintended persuasion. It then examines the assumptions each oversight method needs, using simple cost and error formulas to show what decomposition and adversarial argument gain and where their guarantees stop. The empirical record on debate is reported as it stands, with clear gains for multi-turn debate in reading comprehension tasks where the judge cannot see the source passage and small or mixed effects elsewhere. A comparative framework is proposed that links each method to its required assumption, its characteristic failure mode and a diagnostic test, together with an evaluation protocol built on the sandwiching design. The framework is analytical and has not yet been tested empirically.

Author Biography

  • Raji N, YuvakshatraInstitute of Management Studies (YIMS), Mundur, India

    Assistant Professor, Department of Computer Science

Downloads

Published

2026-11-20