Backtestsopenai-agents-pythonrun-20261024-184640#4468

Issue #4468 Merged fix PR #4470 Run 24 Oct 2026 · run-20261024-184640 openai/openai-agents-python

apply_patch grammar accepts empty updates that the runtime parser rejects

Better

Our patch was judged better than the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

Both patches close the grammar/runtime mismatch, but they resolve it in opposite directions. Patch A tightens the lark grammar and the human-readable description so a move-only Update File is no longer expressible, leaving the runtime unchanged. That eliminates the contract gap, but the issue explicitly calls a rename-without-content-changes 'a legitimate file operation'; Patch A simply makes that operation impossible rather than supporting it, and the independent test test_move_only_update_renames_file still fails on A with the same 'must include a hunk' error as base. Patch B fixes the root cause on both sides: the grammar becomes '(change_move change? | change)' so a bare Update File is rejected while move-only remains valid, the EBNF in the description is updated to match, a sentence is added explaining that hunks may be omitted only with Move to, and _parse_update_file now accepts a move without diff lines (empty diff string) while still rejecting an update with neither. All three independent tests pass on B, including the actual rename producing 'Updated foo.py\nMoved foo.py to bar.py' with contents preserved. On tests, Patch A's additions are largely tautological string matches against the grammar text plus a rejection check that already passed on base for the empty case; Patch B adds a behavioural test that performs the move-only rename through the real tool path, a rejection test for the hunk-less, move-less case verifying the file is untouched, and a grammar assertion. Scope is comparable and B's runtime change is minimal and guarded. B is the better fix.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

4m 37s

Wall-clock time in the container

Full test suite

No new failures

36 passed · 0 skipped · 36 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

57vs 42 merged

Changed lines, ours vs merged

The custom apply_patch grammar allows an Update File operation containing only a rename:

# src/agents/sandbox/capabilities/tools/apply_patch_tool.py
update_hunk: "*** Update File: " filename LF change_move? change?

Because change is optional, the tool grammar tells the LLM that a move-only update is valid:

*** Begin Patch
*** Update File: foo.py
*** Move to: bar.py
*** End Patch

However, _parse_update_file() rejects it because no diff lines are present:

# src/agents/sandbox/capabilities/tools/apply_patch_tool.py
if not diff_lines:
    raise ValueError(f"Update File patch for {path} must include a hunk")

This creates a grammar/runtime contract mismatch for a legitimate file operation and wastes a tool call.

Note: This issue was identified by an automated testing tool for academic research and manually verified. If you have any concerns about this type of reporting, please let me know, and I will adjust my workflow accordingly.

The two patches

Our patch+52 −5
src/agents/sandbox/capabilities/tools/apply_patch_tool.pyindex 6bfeeb41..826180e4 100644@@ -28,7 +28,7 @@ end_patch: "*** End Patch" LF? hunk: add_hunk | delete_hunk | update_hunk add_hunk: "*** Add File: " filename LF add_line+ delete_hunk: "*** Delete File: " filename LF-update_hunk: "*** Update File: " filename LF change_move? change?+update_hunk: "*** Update File: " filename LF (change_move change? | change)  filename: /(.+)/ add_line: "+" /(.*)/ LF -> line@@ -61,6 +61,7 @@ Each operation starts with one of three headers:  May be immediately followed by *** Move to: <new path> if you want to rename the file. Then one or more hunks, each introduced by @@ (optionally followed by a hunk header).+Hunks may be omitted only when *** Move to: is present (a rename without content changes). Within a hunk, each line starts with a space, -, or +.  For context lines:@@ -94,7 +95,7 @@ End := "*** End Patch" NEWLINE FileOp := AddFile | DeleteFile | UpdateFile AddFile := "*** Add File: " path NEWLINE { "+" line NEWLINE } DeleteFile := "*** Delete File: " path NEWLINE-UpdateFile := "*** Update File: " path NEWLINE [ MoveTo ] { Hunk }+UpdateFile := "*** Update File: " path NEWLINE ( MoveTo { Hunk } | Hunk { Hunk } ) MoveTo := "*** Move to: " newPath NEWLINE Hunk := "@@" [ header ] NEWLINE { HunkLine } [ "*** End of File" NEWLINE ] HunkLine := (" " | "-" | "+") text NEWLINE@@ -389,13 +390,13 @@ def _parse_update_file(lines: list[str], index: int) -> tuple[ApplyPatchOperatio     while index < len(lines) - 1 and not _is_file_operation_header(lines[index]):         diff_lines.append(lines[index])         index += 1-    if not diff_lines:-        raise ValueError(f"Update File patch for {path} must include a hunk")+    if not diff_lines and move_to is None:+        raise ValueError(f"Update File patch for {path} must include a hunk or a Move to")     return (         ApplyPatchOperation(             type="update_file",             path=path,-            diff=_join_diff(diff_lines),+            diff=_join_diff(diff_lines) if diff_lines else "",             move_to=move_to,         ),         index,tests/sandbox/capabilities/test_apply_patch_tool.pyindex db877dea..1765c856 100644@@ -422,6 +422,52 @@ class TestSandboxApplyPatchTool:         assert result.output == "Updated notes.txt\nMoved notes.txt to moved.txt"         assert session.files[Path("/workspace/tasks/a/moved.txt")] == b"new\n" +    @pytest.mark.asyncio+    async def test_custom_tool_applies_move_only_update(self) -> None:+        session = ApplyPatchSession()+        session.files[Path("/workspace/foo.py")] = b"print('hi')\n"+        tool = SandboxApplyPatchTool(session=session)++        result = await _execute_custom_tool_call(+            tool,+            context_wrapper=make_context_wrapper(),+            raw_input=(+                "*** Begin Patch\n"+                "*** Update File: foo.py\n"+                "*** Move to: bar.py\n"+                "*** End Patch\n"+            ),+        )++        assert isinstance(result, ToolCallOutputItem)+        assert result.output == "Updated foo.py\nMoved foo.py to bar.py"+        assert session.files == {Path("/workspace/bar.py"): b"print('hi')\n"}++    @pytest.mark.asyncio+    async def test_custom_tool_rejects_update_without_hunk_or_move(self) -> None:+        session = ApplyPatchSession()+        session.files[Path("/workspace/foo.py")] = b"print('hi')\n"+        tool = SandboxApplyPatchTool(session=session)++        result = await _execute_custom_tool_call(+            tool,+            context_wrapper=make_context_wrapper(),+            raw_input="*** Begin Patch\n*** Update File: foo.py\n*** End Patch\n",+        )++        assert isinstance(result, ToolCallOutputItem)+        assert "Update File patch for foo.py must include a hunk or a Move to" in result.output+        assert session.files == {Path("/workspace/foo.py"): b"print('hi')\n"}++    def test_grammar_requires_move_or_hunk_for_update(self) -> None:+        tool = SandboxApplyPatchTool(session=scripted_sandbox_session())+        definition = tool.tool_config["format"]["definition"]++        assert (+            'update_hunk: "*** Update File: " filename LF (change_move change? | change)'+            in definition+        )+     @pytest.mark.asyncio     async def test_scoped_custom_tool_keeps_absolute_path_at_workspace_root(self) -> None:         session = ApplyPatchSession() 
Merged fix · PR #4470+40 −2
src/agents/sandbox/capabilities/tools/apply_patch_tool.pyindex 6bfeeb416..bcf4462c1 100644@@ -28,7 +28,7 @@ end_patch: "*** End Patch" LF? hunk: add_hunk | delete_hunk | update_hunk add_hunk: "*** Add File: " filename LF add_line+ delete_hunk: "*** Delete File: " filename LF-update_hunk: "*** Update File: " filename LF change_move? change?+update_hunk: "*** Update File: " filename LF change_move? change  filename: /(.+)/ add_line: "+" /(.*)/ LF -> line@@ -94,7 +94,7 @@ End := "*** End Patch" NEWLINE FileOp := AddFile | DeleteFile | UpdateFile AddFile := "*** Add File: " path NEWLINE { "+" line NEWLINE } DeleteFile := "*** Delete File: " path NEWLINE-UpdateFile := "*** Update File: " path NEWLINE [ MoveTo ] { Hunk }+UpdateFile := "*** Update File: " path NEWLINE [ MoveTo ] Hunk { Hunk } MoveTo := "*** Move to: " newPath NEWLINE Hunk := "@@" [ header ] NEWLINE { HunkLine } [ "*** End of File" NEWLINE ] HunkLine := (" " | "-" | "+") text NEWLINEtests/sandbox/capabilities/test_apply_patch_tool.pyindex db877dea7..c0f5f46ad 100644@@ -41,6 +41,20 @@ class TestSandboxApplyPatchTool:         assert tool.tool_config["format"]["type"] == "grammar"         assert tool.tool_config["format"]["syntax"] == "lark" +    def test_grammar_requires_update_diff_after_optional_move(self) -> None:+        tool = SandboxApplyPatchTool(session=scripted_sandbox_session())++        grammar = cast(dict[str, Any], tool.tool_config["format"])["definition"]+        assert isinstance(grammar, str)+        update_rule = next(line for line in grammar.splitlines() if line.startswith("update_hunk:"))+        assert update_rule == 'update_hunk: "*** Update File: " filename LF change_move? change'++        description = tool.tool_config["description"]+        assert isinstance(description, str)+        assert (+            'UpdateFile := "*** Update File: " path NEWLINE [ MoveTo ] Hunk { Hunk }' in description+        )+     def test_converter_uses_sandbox_custom_apply_patch_tool_config(self) -> None:         tool = SandboxApplyPatchTool(session=scripted_sandbox_session()) @@ -54,6 +68,30 @@ class TestSandboxApplyPatchTool:         assert "A full patch can combine several operations" in description         tool_format = cast(dict[str, Any], converted.tools[0]["format"])         assert tool_format["syntax"] == "lark"+        assert tool_format["definition"] == tool.tool_config["format"]["definition"]++    @pytest.mark.parametrize(+        "update_body",+        [+            "",+            "*** Move to: moved.txt\n",+        ],+        ids=["empty", "move-only"],+    )+    @pytest.mark.asyncio+    async def test_runtime_rejects_updates_without_diff(self, update_body: str) -> None:+        tool = SandboxApplyPatchTool(session=scripted_sandbox_session())++        result = await _execute_custom_tool_call(+            tool,+            context_wrapper=make_context_wrapper(),+            raw_input=(+                f"*** Begin Patch\n*** Update File: notes.txt\n{update_body}*** End Patch\n"+            ),+        )++        assert isinstance(result, ToolCallOutputItem)+        assert "Update File patch for notes.txt must include a hunk" in result.output      def test_needs_approval_exposes_operation_typed_setting(self) -> None:         async def needs_approval(