Backtestsopenai-agents-pythonrun-20261024-184640#4757
OpenAIConversationsSession get_items(limit>provider page size) forwards the session limit as provider page size
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches make the identical root-cause change: drop `limit=session_limit` from `conversations.items.list()` so the client paginates with the provider's own page size, while keeping the existing local `break` once N items are collected and the final reverse into chronological order. Independent tests confirm both fix every scenario (large explicit limit, limit via session_settings, limit exceeding history, small limit still cut off locally). Patch B additionally drops the now-dead `session_limit is not None` guard in the break condition, which is safe because that branch is only reached when the limit is finite and it makes the intent of the local cutoff clearer; it adds no risk. On tests, B is meaningfully stronger: its mock actively rejects page sizes over 100 (so it would fail before the fix for the exact reason the issue describes), and it covers three scenarios the issue implies, including a large limit supplied via session_settings and a limit larger than the stored history, both asserting the list call carries no `limit` and the newest N items come back in order. Patch A's single test is valid and verifies the stop-at-N behaviour via recorded yields, but it only exercises one path and does not simulate the provider rejection that motivates the issue. Craft is comparable; B's shared helper keeps the three tests compact. B wins on test coverage and the small clarity improvement, with no correctness or scope downside.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
5m 54s
Wall-clock time in the container
Full test suite
No new failures
249 passed · 0 skipped · 249 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
81vs 37 merged
Changed lines, ours vs merged
The issue
View on GitHubOpenAIConversationsSession.get_items(limit=N) currently forwards N directly to conversations.items.list(limit=N). For large session limits this couples the SDK's local history cutoff to the provider's page-size constraints and can make an otherwise valid Session.get_items(limit=N) call fail before client pagination occurs.
A previous implementation attempt in #4721 confirmed the underlying bug. Maintainer guidance there was to avoid copying provider page-size constants into the SDK and instead omit limit from conversations.items.list(), letting the OpenAI client handle provider pagination while preserving the SDK's existing local cutoff.
Expected behavior: for a finite session limit, request items in descending order without forwarding the SDK limit as a provider page size, stop locally once N items are collected, then reverse them into chronological order.
The two patches
src/agents/memory/openai_conversations_session.pyindex 47c445d3..cb8fd0de 100644@@ -98,14 +98,15 @@ class OpenAIConversationsSession(SessionABC): # calling model_dump() to make this serializable all_items.append(item.model_dump(exclude_unset=True)) else:+ # Do not forward the session limit as the provider page size; let the client+ # paginate and stop locally once enough items have been collected. async for item in self._openai_client.conversations.items.list( conversation_id=session_id,- limit=session_limit, order="desc", ): # calling model_dump() to make this serializable all_items.append(item.model_dump(exclude_unset=True))- if session_limit is not None and len(all_items) >= session_limit:+ if len(all_items) >= session_limit: break all_items.reverse() tests/memory/test_openai_conversations_session.pyindex 0357ddf4..22b3d7a2 100644@@ -232,6 +232,82 @@ class TestOpenAIConversationsSessionBasicOperations: assert session.session_id == "test_conversation_id" mock_openai_client.conversations.items.list.assert_not_called() + @staticmethod+ def _install_desc_items_list(mock_openai_client, total: int) -> list[int]:+ """Mock items.list to yield `total` items newest-first, recording consumption."""+ consumed: list[int] = []++ def list_items(**kwargs: Any) -> Any:+ if "limit" in kwargs and kwargs["limit"] > 100:+ raise ValueError("provider rejects page sizes above 100")++ async def iterate():+ for index in reversed(range(total)):+ consumed.append(index)+ item = MagicMock()+ item.model_dump.return_value = {"id": f"item_{index}"}+ yield item++ return iterate()++ mock_openai_client.conversations.items.list = MagicMock(side_effect=list_items)+ return consumed++ @pytest.mark.asyncio+ async def test_get_items_large_limit_not_forwarded_as_page_size(self, mock_openai_client):+ """A session limit above the provider page size must still return the newest N items."""+ consumed = self._install_desc_items_list(mock_openai_client, total=250)+ session = OpenAIConversationsSession(+ conversation_id="test_id", openai_client=mock_openai_client+ )++ items = await session.get_items(limit=150)++ assert [item["id"] for item in items] == [ # type: ignore[typeddict-item]+ f"item_{index}" for index in range(100, 250)+ ]+ mock_openai_client.conversations.items.list.assert_called_once_with(+ conversation_id="test_id", order="desc"+ )+ assert len(consumed) == 150++ @pytest.mark.asyncio+ async def test_get_items_large_settings_limit_not_forwarded_as_page_size(+ self, mock_openai_client+ ):+ """A large limit from session_settings must also not be forwarded as the page size."""+ self._install_desc_items_list(mock_openai_client, total=250)+ session = OpenAIConversationsSession(+ conversation_id="test_id",+ openai_client=mock_openai_client,+ session_settings={"limit": 200},+ )++ items = await session.get_items()++ assert [item["id"] for item in items] == [ # type: ignore[typeddict-item]+ f"item_{index}" for index in range(50, 250)+ ]+ mock_openai_client.conversations.items.list.assert_called_once_with(+ conversation_id="test_id", order="desc"+ )++ @pytest.mark.asyncio+ async def test_get_items_limit_larger_than_history_returns_all(self, mock_openai_client):+ """A limit larger than the stored history returns all items chronologically."""+ self._install_desc_items_list(mock_openai_client, total=3)+ session = OpenAIConversationsSession(+ conversation_id="test_id", openai_client=mock_openai_client+ )++ items = await session.get_items(limit=500)++ assert [item["id"] for item in items] == [ # type: ignore[typeddict-item]+ "item_0",+ "item_1",+ "item_2",+ ]+ @pytest.mark.asyncio async def test_add_items_simple(self, mock_openai_client): """Test adding items to the conversation."""
src/agents/memory/openai_conversations_session.pyindex 47c445d38..d9fb4f1c1 100644@@ -98,9 +98,9 @@ class OpenAIConversationsSession(SessionABC): # calling model_dump() to make this serializable all_items.append(item.model_dump(exclude_unset=True)) else:+ # Omit provider page size. Apply the session cutoff locally after pagination. async for item in self._openai_client.conversations.items.list( conversation_id=session_id,- limit=session_limit, order="desc", ): # calling model_dump() to make this serializabletests/memory/test_openai_conversations_session.pyindex 0357ddf41..5d5868f7b 100644@@ -232,6 +232,41 @@ class TestOpenAIConversationsSessionBasicOperations: assert session.session_id == "test_conversation_id" mock_openai_client.conversations.items.list.assert_not_called() + @pytest.mark.asyncio+ async def test_get_items_applies_large_limit_after_provider_pagination(+ self, mock_openai_client+ ):+ """A session limit must not be forwarded as the Conversations API page size."""++ class ConversationItem:+ def __init__(self, item_id: int) -> None:+ self.item_id = item_id++ def model_dump(self, *, exclude_unset: bool) -> dict[str, int]:+ assert exclude_unset is True+ return {"item_id": self.item_id}++ yielded_item_ids: list[int] = []++ async def descending_items():+ # Yield one extra newest-first item so the local cutoff can prove it stops at N.+ for item_id in range(101, -1, -1):+ yielded_item_ids.append(item_id)+ yield ConversationItem(item_id)++ mock_openai_client.conversations.items.list = MagicMock(return_value=descending_items())+ session = OpenAIConversationsSession(+ conversation_id="test_id", openai_client=mock_openai_client+ )++ items = await session.get_items(limit=101)++ assert [cast(dict[str, int], item)["item_id"] for item in items] == list(range(1, 102))+ assert yielded_item_ids == list(range(101, 0, -1))+ mock_openai_client.conversations.items.list.assert_called_once_with(+ conversation_id="test_id", order="desc"+ )+ @pytest.mark.asyncio async def test_add_items_simple(self, mock_openai_client): """Test adding items to the conversation."""