Backtestspydantic-airun-20260918-215936#4448
Evals: SpanQuery format mismatch
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches fix the reported case (dict query matching a logfire-serialized JSON attribute), so the decision turns on fidelity to the root cause and test quality. B addresses the root cause directly: logfire stored the value via its own encoder (logfire_json_dumps), and B compares the stored string against logfire_json_dumps(query_value) — the exact inverse of what actually happened. This matches the issue author's proposed solution and correctly handles values that logfire serializes non-standardly (datetimes, sets, etc.), where plain json can diverge. A instead calls stdlib json.loads(stored) and compares as Python objects; this only works for values whose logfire encoding happens to be standard JSON, and it introduces a type-coercion false positive (a genuine string attribute '123' would match an int query 123). A does gain order-independence (dict equality ignores key order) whereas B is key-order sensitive, which is a real point in A's favor, but for the reported/typical usage the query reconstructs the same dict and both succeed. The decisive factor is verification: B's test drives real logfire (context_subtree + logfire.span/info) to reproduce the reported scenario end-to-end and also asserts backward compatibility with the pre-existing JSON-string form and a non-match case — so it proves the fix works against logfire's actual output. A's tests are more numerous but synthetic: they hardcode the assumed serialized string ('{"foo":1,"bar":true}') into a manually-built SpanNode, so they are self-consistent with A's implementation but never verify that assumption against real logfire; a wrong assumption about logfire's format would leave the real bug unfixed while A's tests still pass. Both patches are minimal and comparably scoped; A's stdlib import is marginally cleaner than B's lazy logfire import with ImportError fallback, but that fallback is justified and guarded. On balance, B's faithfulness to the actual serialization mechanism plus a realistic reproduction test outweigh A's order-independence and cleaner import.
The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.
Solve
11m1 turn
Wall-clock time in the container
Full test suite
No new failures
3,998 passed · 181 skipped · 4,278 total
Contamination audit
Unknown
See the audit notes for this run
Patch size
40vs 96 merged
Changed lines, ours vs merged
The issue
View on GitHubInitial Checks
- I'm using the latest version of Pydantic AI
- I've searched for my issue in the issue tracker before opening this issue
Description
Problem
When trying to match dict like attributes in Span, the original dict yields no match. This is due to dumping json attributes as string, which makes matching these less intuitive.
Desired State
the original dict matches the span where the dict was used
Solution
on pydantic_evals/otel/span_tree.py#L270 replace
self.attributes.get(key) == value for key, value in has_attributes.items()
with
self.attributes.get(key) == logfire_json_dumps(value) for key, value in has_attributes.items()
Minimal, Reproducible Example
import os
import dotenv
import logfire
from pydantic_evals.otel._context_in_memory_span_exporter import (
context_subtree,
)
from pydantic_evals.otel.span_tree import SpanQuery
dotenv.load_dotenv()
logfire.configure(token=os.getenv("LOGFIRE_TOKEN"))
DATA = {"foo": 1, "bar": True}
with context_subtree() as tree:
with logfire.span("task"):
with logfire.span("subtask"):
data = DATA
logfire.info("some data", data=data)
assert tree.first(SpanQuery(has_attribute_keys=["data"])) is not None
# Works but requires JSON parsing
assert (
tree.first(SpanQuery(has_attributes={"data": '{"foo":1,"bar":true}'}))
is not None
)
# Intuitive Query but yields None
assert (
tree.first(SpanQuery(has_attributes={"data": DATA}))
is None
)
Logfire Trace
_No response_
Python, Pydantic AI & LLM client version
- Python: 3.12
- Pydantic AI: 1.63.0
- LLM provider SDK:
The two patches
pydantic_evals/pydantic_evals/otel/span_tree.pyindex 34887064..2cf27db9 100644@@ -22,6 +22,21 @@ AttributeValue = str | bool | int | float | Sequence[str] | Sequence[bool] | Seq __all__ = 'SpanNode', 'SpanTree', 'SpanQuery' +def _matches_serialized_attribute(attribute_value: AttributeValue | None, query_value: Any) -> bool:+ """Check whether a span attribute matches a query value.++ logfire serializes non-primitive attribute values (e.g. dicts and lists) to JSON+ strings, so fall back to comparing against the JSON-serialized form of the query value.+ """+ if attribute_value == query_value:+ return True+ try:+ from logfire._internal.json_encoder import logfire_json_dumps+ except ImportError: # pragma: no cover+ return False+ return attribute_value == logfire_json_dumps(query_value)++ class SpanQuery(TypedDict, total=False): """A serializable query for filtering SpanNodes based on various conditions. @@ -267,7 +282,7 @@ class SpanNode: # Attribute conditions if (has_attributes := query.get('has_attributes')) and not all(- self.attributes.get(key) == value for key, value in has_attributes.items()+ _matches_serialized_attribute(self.attributes.get(key), value) for key, value in has_attributes.items() ): return False if (has_attributes_keys := query.get('has_attribute_keys')) and not all(tests/evals/test_otel.pyindex 5c6bfacd..20ad9af4 100644@@ -223,6 +223,29 @@ async def test_span_node_matches(span_tree: SpanTree): assert not child1_node.matches(SpanQuery(name_equals='child1', has_attributes={'type': 'normal'})) +async def test_span_query_has_attributes_with_dict_and_list():+ """Test that has_attributes matches dict/list attributes stored by logfire as JSON strings."""+ with context_subtree() as tree:+ with logfire.span('task'):+ logfire.info('some data', data={'foo': 1, 'bar': True}, items=[1, 2, 3])+ assert isinstance(tree, SpanTree)++ # Key-only query still works+ assert tree.first(SpanQuery(has_attribute_keys=['data'])) is not None++ # Intuitive dict query matches (the reported bug)+ assert tree.first(SpanQuery(has_attributes={'data': {'foo': 1, 'bar': True}})) is not None++ # Intuitive list query matches+ assert tree.first(SpanQuery(has_attributes={'items': [1, 2, 3]})) is not None++ # Pre-existing JSON-string form also still works (backward compat)+ assert tree.first(SpanQuery(has_attributes={'data': '{"foo":1,"bar":true}'})) is not None++ # A non-matching dict does not match+ assert tree.first(SpanQuery(has_attributes={'data': {'foo': 2}})) is None++ async def test_span_tree_repr(span_tree: SpanTree): assert repr(SpanTree()) == snapshot('<SpanTree />') assert str(span_tree) == snapshot('<SpanTree num_roots=1 total_spans=6 />')
pydantic_evals/pydantic_evals/otel/span_tree.pyindex 348870645f..a4893ca71a 100644@@ -1,5 +1,6 @@ from __future__ import annotations +import json import re from collections.abc import Callable, Iterator, Sequence from dataclasses import dataclass, field@@ -78,6 +79,24 @@ class SpanQuery(TypedDict, total=False): no_ancestor_has: SpanQuery +def _attribute_value_matches(stored: AttributeValue | None, expected: Any) -> bool:+ """Compare a stored span attribute value against an expected query value.++ OpenTelemetry attribute values only support primitive types and sequences of primitives.+ When complex values like dicts are set as attributes, instrumentation libraries (e.g. logfire)+ serialize them to JSON strings. This function handles that case by attempting to deserialize+ stored JSON strings and comparing as Python objects when a direct comparison fails.+ """+ if stored == expected:+ return True+ if isinstance(stored, str) and not isinstance(expected, str):+ try:+ return json.loads(stored) == expected+ except (json.JSONDecodeError, ValueError):+ return False+ return False++ @dataclass(repr=False, kw_only=True) class SpanNode: """A node in the span tree; provides references to parents/children for easy traversal and queries."""@@ -267,7 +286,7 @@ class SpanNode: # Attribute conditions if (has_attributes := query.get('has_attributes')) and not all(- self.attributes.get(key) == value for key, value in has_attributes.items()+ _attribute_value_matches(self.attributes.get(key), value) for key, value in has_attributes.items() ): return False if (has_attributes_keys := query.get('has_attribute_keys')) and not all(tests/evals/test_otel.pyindex 5c6bfacde0..e57c12d5ea 100644@@ -15,7 +15,7 @@ with try_import() as imports_successful: from pydantic_evals.otel._context_subtree import ( context_subtree, )- from pydantic_evals.otel.span_tree import SpanQuery, SpanTree+ from pydantic_evals.otel.span_tree import SpanNode, SpanQuery, SpanTree, _attribute_value_matches pytestmark = [pytest.mark.skipif(not imports_successful(), reason='pydantic-evals not installed'), pytest.mark.anyio] @@ -223,6 +223,79 @@ async def test_span_node_matches(span_tree: SpanTree): assert not child1_node.matches(SpanQuery(name_equals='child1', has_attributes={'type': 'normal'})) +async def test_attribute_value_matches_json_serialized_dicts():+ """Test that has_attributes can match dict values stored as JSON strings.++ When instrumentation libraries like logfire store complex attribute values (dicts, lists),+ they serialize them to JSON strings since OpenTelemetry only supports primitive attribute types.+ SpanQuery should still match these by deserializing the stored string before comparison.+ """+ from datetime import datetime, timezone++ # Build a SpanNode with a JSON-serialized dict attribute (simulating what logfire does)+ node = SpanNode(+ name='test_span',+ trace_id=1,+ span_id=1,+ parent_span_id=None,+ start_timestamp=datetime(2024, 1, 1, tzinfo=timezone.utc),+ end_timestamp=datetime(2024, 1, 1, 0, 0, 1, tzinfo=timezone.utc),+ attributes={+ 'simple': 'value',+ 'data': '{"foo":1,"bar":true}',+ 'nested': '{"outer":{"inner":"deep"}}',+ 'list_attr': '[1,2,3]',+ 'not_json': 'just a plain string',+ },+ )++ # Direct string match still works+ assert node.matches(SpanQuery(has_attributes={'simple': 'value'}))++ # Dict value matches against its JSON-serialized form+ assert node.matches(SpanQuery(has_attributes={'data': {'foo': 1, 'bar': True}}))++ # Nested dict value matches+ assert node.matches(SpanQuery(has_attributes={'nested': {'outer': {'inner': 'deep'}}}))++ # List value matches against its JSON-serialized form+ assert node.matches(SpanQuery(has_attributes={'list_attr': [1, 2, 3]}))++ # Wrong value still fails+ assert not node.matches(SpanQuery(has_attributes={'data': {'foo': 2, 'bar': True}}))++ # Non-JSON string does not match a dict query+ assert not node.matches(SpanQuery(has_attributes={'not_json': {'key': 'value'}}))++ # Missing key fails+ assert not node.matches(SpanQuery(has_attributes={'missing': 'anything'}))+++def test_attribute_value_matches_helper():+ """Unit tests for the _attribute_value_matches helper function."""+ # Exact match+ assert _attribute_value_matches('hello', 'hello')+ assert _attribute_value_matches(42, 42)+ assert _attribute_value_matches(True, True)++ # JSON string vs dict+ assert _attribute_value_matches('{"a":1}', {'a': 1})+ assert not _attribute_value_matches('{"a":1}', {'a': 2})++ # JSON string vs list+ assert _attribute_value_matches('[1,2,3]', [1, 2, 3])++ # Non-JSON string vs dict+ assert not _attribute_value_matches('not json', {'a': 1})++ # None stored value+ assert not _attribute_value_matches(None, 'anything')++ # Both strings, no JSON fallback needed+ assert _attribute_value_matches('same', 'same')+ assert not _attribute_value_matches('different', 'other')++ async def test_span_tree_repr(span_tree: SpanTree): assert repr(SpanTree()) == snapshot('<SpanTree />') assert str(span_tree) == snapshot('<SpanTree num_roots=1 total_spans=6 />')
What the agent reported
The reviewer confirms the fix is correct, minimal, and complete. The work is done.
Summary
What was wrong
SpanQuery(has_attributes={...}) in pydantic_evals/otel/span_tree.py compared the query value directly (self.attributes.get(key) == value) against the stored span attribute. logfire serializes non-primitive attribute values (dicts, lists, enums, large ints) to JSON strings before they reach the span, so a query like has_attributes={'data': {'foo': 1, 'bar': True}} was comparing a JSON string against a Python dict and never matched — even though the span carried that dict.
What I changed
pydantic_evals/pydantic_evals/otel/span_tree.py: added a module-level helper_matches_serialized_attribute(...)and routed thehas_attributesmatching through it. It does the raw==comparison first (short-circuitingTrue), and only on a miss falls back to comparing againstlogfire_json_dumps(query_value).logfire_json_dumpsis imported lazily inside atry/except ImportErrorso the module still imports when logfire (an optional dependency) is absent.tests/evals/test_otel.py: addedtest_span_query_has_attributes_with_dict_and_list, a regression test mirroring the reported reproduction.
The report literally suggested replacing the comparison with logfire_json_dumps(value). I deliberately made it a fallback rather than a blanket replacement, because a blanket replacement would break every existing scalar match (e.g. has_attributes={'depth': 3} — logfire_json_dumps(3) is the string '3', which does not equal the natively-stored int 3). The fallback fixes the dict/list case while preserving all scalar/str/bool matching.
Test evidence
./bt-test tests/evals/test_otel.py→ 26 passed (with the fix), including the new test.- Confirmed the new test genuinely captures the bug: with the production change temporarily reverted,
test_span_query_has_attributes_with_dict_and_listfails at the exact reported assertion (SpanQuery(has_attributes={'data': {'foo': 1, 'bar': True}})returnsNone), then passes once restored. - Reviewer confirmed: correct, minimal, no existing tests weakened, no unrelated changes, no scratch files.
Requirements disposition
- Original dict matches the span where the dict was used — Implemented & tested (test_otel.py:237, dict query now matches).
- Also list values / intuitive queries — Implemented & tested (test_otel.py:240).
- Backward compatibility with the existing JSON-string form — Implemented & tested (test_otel.py:243 still matches).
- Suggested edit at span_tree.py#L270 using
logfire_json_dumps— Implemented as a fallback (not a blanket replacement, with evidence above that a literal replacement would break existing int/scalar tests such as{'depth': 3}). The suggested serialization function is used, guarded by a lazy import since logfire is optional.
Changes are left uncommitted in the working tree. (The three deleted files — .claude/settings.json, AGENTS.md, CLAUDE.md — were pre-existing deletions in the starting working tree, untouched by me.)