Human Oversight for AI-assisted Research and Coding
Two related papers will be presented. The first with Rawan Qadri and Iddo Drori on Human Oversight for AI-assisted Research, and the second with Adi Katz on Oversight in Vibe Coding, which was done for the former.
Autonomous scientific discovery (ASD) is increasingly gaining the ability to produce conference-quality manuscripts. However, concerns remain about safeguarding researchers’ intent and ensuring research trustworthiness. This paper presents a design theory for human oversight in ASD, and focuses particularly on achieving high output quality and specification alignment as a means to enhance users’ perceptions of trustworthiness, and more specifically, perceived trusted ability and perceived goal alignment. We implemented these principles in IScopilot, a system that executes users’ research plans and generates manuscript drafts through cycles of human-supervised review. IScopilot incorporates supervisory features for external validation, oversight, and continuous monitoring of quality and alignment.
We investigate communication breakdowns and functional errors in “vibe coding” that I did in building IScopilot. We develop an integrated theoretical model drawing on computer-mediated communication theory, Reciprocal Human-Machine Learning (RHML), and Cognitive Load Theory to explain how cognitive overload degrades context maintenance, producing communication breakdowns that cascade into functional errors. Thematic analysis identified ten recurring breakdown patterns, with verification burden and trust erosion most prevalent. A critical temporal finding revealed that functional errors surged from 20.6% to 64.3% of episodes as project complexity grew, while communication breakdowns were partially mitigated through learned interaction protocols. We discuss implications for developers: mainly, that developers using AI may build systems more efficiently but must invest more effort in oversight and verification and AI system designers must support oversight with contingent contextualization.
