Backtestspydantic-airun-20260918-215936#4448

Issue #4448 Merged fix PR # Run 18 Sep 2026 · run-20260918-215936 pydantic/pydantic-ai

Evals: SpanQuery format mismatch

Better

Our patch was judged better than the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

Both patches fix the reported case (dict query matching a logfire-serialized JSON attribute), so the decision turns on fidelity to the root cause and test quality. B addresses the root cause directly: logfire stored the value via its own encoder (logfire_json_dumps), and B compares the stored string against logfire_json_dumps(query_value) — the exact inverse of what actually happened. This matches the issue author's proposed solution and correctly handles values that logfire serializes non-standardly (datetimes, sets, etc.), where plain json can diverge. A instead calls stdlib json.loads(stored) and compares as Python objects; this only works for values whose logfire encoding happens to be standard JSON, and it introduces a type-coercion false positive (a genuine string attribute '123' would match an int query 123). A does gain order-independence (dict equality ignores key order) whereas B is key-order sensitive, which is a real point in A's favor, but for the reported/typical usage the query reconstructs the same dict and both succeed. The decisive factor is verification: B's test drives real logfire (context_subtree + logfire.span/info) to reproduce the reported scenario end-to-end and also asserts backward compatibility with the pre-existing JSON-string form and a non-match case — so it proves the fix works against logfire's actual output. A's tests are more numerous but synthetic: they hardcode the assumed serialized string ('{"foo":1,"bar":true}') into a manually-built SpanNode, so they are self-consistent with A's implementation but never verify that assumption against real logfire; a wrong assumption about logfire's format would leave the real bug unfixed while A's tests still pass. Both patches are minimal and comparably scoped; A's stdlib import is marginally cleaner than B's lazy logfire import with ImportError fallback, but that fallback is justified and guarded. On balance, B's faithfulness to the actual serialization mechanism plus a realistic reproduction test outweigh A's order-independence and cleaner import.

The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.

Solve

11m1 turn

Wall-clock time in the container

Full test suite

No new failures

3,998 passed · 181 skipped · 4,278 total

Contamination audit

Unknown

See the audit notes for this run

Patch size

40vs 96 merged

Changed lines, ours vs merged

Initial Checks

Description

Problem

When trying to match dict like attributes in Span, the original dict yields no match. This is due to dumping json attributes as string, which makes matching these less intuitive.

Desired State

the original dict matches the span where the dict was used

Solution

on pydantic_evals/otel/span_tree.py#L270 replace

self.attributes.get(key) == value for key, value in has_attributes.items()

with

self.attributes.get(key) == logfire_json_dumps(value) for key, value in has_attributes.items()

Minimal, Reproducible Example

import os

import dotenv
import logfire
from pydantic_evals.otel._context_in_memory_span_exporter import (
    context_subtree,
)
from pydantic_evals.otel.span_tree import SpanQuery

dotenv.load_dotenv()
logfire.configure(token=os.getenv("LOGFIRE_TOKEN"))

DATA = {"foo": 1, "bar": True}

with context_subtree() as tree:
    with logfire.span("task"):
        with logfire.span("subtask"):
            data = DATA
            logfire.info("some data", data=data)


assert tree.first(SpanQuery(has_attribute_keys=["data"])) is not None

# Works but requires JSON parsing
assert (
    tree.first(SpanQuery(has_attributes={"data": '{"foo":1,"bar":true}'}))
    is not None
)

# Intuitive Query but yields None
assert (
    tree.first(SpanQuery(has_attributes={"data": DATA}))
    is None
)

Logfire Trace

_No response_

Python, Pydantic AI & LLM client version

  • Python: 3.12
  • Pydantic AI: 1.63.0
  • LLM provider SDK:

The two patches

Our patch+39 −1
pydantic_evals/pydantic_evals/otel/span_tree.pyindex 34887064..2cf27db9 100644@@ -22,6 +22,21 @@ AttributeValue = str | bool | int | float | Sequence[str] | Sequence[bool] | Seq __all__ = 'SpanNode', 'SpanTree', 'SpanQuery'  +def _matches_serialized_attribute(attribute_value: AttributeValue | None, query_value: Any) -> bool:+    """Check whether a span attribute matches a query value.++    logfire serializes non-primitive attribute values (e.g. dicts and lists) to JSON+    strings, so fall back to comparing against the JSON-serialized form of the query value.+    """+    if attribute_value == query_value:+        return True+    try:+        from logfire._internal.json_encoder import logfire_json_dumps+    except ImportError:  # pragma: no cover+        return False+    return attribute_value == logfire_json_dumps(query_value)++ class SpanQuery(TypedDict, total=False):     """A serializable query for filtering SpanNodes based on various conditions. @@ -267,7 +282,7 @@ class SpanNode:          # Attribute conditions         if (has_attributes := query.get('has_attributes')) and not all(-            self.attributes.get(key) == value for key, value in has_attributes.items()+            _matches_serialized_attribute(self.attributes.get(key), value) for key, value in has_attributes.items()         ):             return False         if (has_attributes_keys := query.get('has_attribute_keys')) and not all(tests/evals/test_otel.pyindex 5c6bfacd..20ad9af4 100644@@ -223,6 +223,29 @@ async def test_span_node_matches(span_tree: SpanTree):     assert not child1_node.matches(SpanQuery(name_equals='child1', has_attributes={'type': 'normal'}))  +async def test_span_query_has_attributes_with_dict_and_list():+    """Test that has_attributes matches dict/list attributes stored by logfire as JSON strings."""+    with context_subtree() as tree:+        with logfire.span('task'):+            logfire.info('some data', data={'foo': 1, 'bar': True}, items=[1, 2, 3])+    assert isinstance(tree, SpanTree)++    # Key-only query still works+    assert tree.first(SpanQuery(has_attribute_keys=['data'])) is not None++    # Intuitive dict query matches (the reported bug)+    assert tree.first(SpanQuery(has_attributes={'data': {'foo': 1, 'bar': True}})) is not None++    # Intuitive list query matches+    assert tree.first(SpanQuery(has_attributes={'items': [1, 2, 3]})) is not None++    # Pre-existing JSON-string form also still works (backward compat)+    assert tree.first(SpanQuery(has_attributes={'data': '{"foo":1,"bar":true}'})) is not None++    # A non-matching dict does not match+    assert tree.first(SpanQuery(has_attributes={'data': {'foo': 2}})) is None++ async def test_span_tree_repr(span_tree: SpanTree):     assert repr(SpanTree()) == snapshot('<SpanTree />')     assert str(span_tree) == snapshot('<SpanTree num_roots=1 total_spans=6 />') 
Merged fix · PR #+94 −2
pydantic_evals/pydantic_evals/otel/span_tree.pyindex 348870645f..a4893ca71a 100644@@ -1,5 +1,6 @@ from __future__ import annotations +import json import re from collections.abc import Callable, Iterator, Sequence from dataclasses import dataclass, field@@ -78,6 +79,24 @@ class SpanQuery(TypedDict, total=False):     no_ancestor_has: SpanQuery  +def _attribute_value_matches(stored: AttributeValue | None, expected: Any) -> bool:+    """Compare a stored span attribute value against an expected query value.++    OpenTelemetry attribute values only support primitive types and sequences of primitives.+    When complex values like dicts are set as attributes, instrumentation libraries (e.g. logfire)+    serialize them to JSON strings. This function handles that case by attempting to deserialize+    stored JSON strings and comparing as Python objects when a direct comparison fails.+    """+    if stored == expected:+        return True+    if isinstance(stored, str) and not isinstance(expected, str):+        try:+            return json.loads(stored) == expected+        except (json.JSONDecodeError, ValueError):+            return False+    return False++ @dataclass(repr=False, kw_only=True) class SpanNode:     """A node in the span tree; provides references to parents/children for easy traversal and queries."""@@ -267,7 +286,7 @@ class SpanNode:          # Attribute conditions         if (has_attributes := query.get('has_attributes')) and not all(-            self.attributes.get(key) == value for key, value in has_attributes.items()+            _attribute_value_matches(self.attributes.get(key), value) for key, value in has_attributes.items()         ):             return False         if (has_attributes_keys := query.get('has_attribute_keys')) and not all(tests/evals/test_otel.pyindex 5c6bfacde0..e57c12d5ea 100644@@ -15,7 +15,7 @@ with try_import() as imports_successful:     from pydantic_evals.otel._context_subtree import (         context_subtree,     )-    from pydantic_evals.otel.span_tree import SpanQuery, SpanTree+    from pydantic_evals.otel.span_tree import SpanNode, SpanQuery, SpanTree, _attribute_value_matches  pytestmark = [pytest.mark.skipif(not imports_successful(), reason='pydantic-evals not installed'), pytest.mark.anyio] @@ -223,6 +223,79 @@ async def test_span_node_matches(span_tree: SpanTree):     assert not child1_node.matches(SpanQuery(name_equals='child1', has_attributes={'type': 'normal'}))  +async def test_attribute_value_matches_json_serialized_dicts():+    """Test that has_attributes can match dict values stored as JSON strings.++    When instrumentation libraries like logfire store complex attribute values (dicts, lists),+    they serialize them to JSON strings since OpenTelemetry only supports primitive attribute types.+    SpanQuery should still match these by deserializing the stored string before comparison.+    """+    from datetime import datetime, timezone++    # Build a SpanNode with a JSON-serialized dict attribute (simulating what logfire does)+    node = SpanNode(+        name='test_span',+        trace_id=1,+        span_id=1,+        parent_span_id=None,+        start_timestamp=datetime(2024, 1, 1, tzinfo=timezone.utc),+        end_timestamp=datetime(2024, 1, 1, 0, 0, 1, tzinfo=timezone.utc),+        attributes={+            'simple': 'value',+            'data': '{"foo":1,"bar":true}',+            'nested': '{"outer":{"inner":"deep"}}',+            'list_attr': '[1,2,3]',+            'not_json': 'just a plain string',+        },+    )++    # Direct string match still works+    assert node.matches(SpanQuery(has_attributes={'simple': 'value'}))++    # Dict value matches against its JSON-serialized form+    assert node.matches(SpanQuery(has_attributes={'data': {'foo': 1, 'bar': True}}))++    # Nested dict value matches+    assert node.matches(SpanQuery(has_attributes={'nested': {'outer': {'inner': 'deep'}}}))++    # List value matches against its JSON-serialized form+    assert node.matches(SpanQuery(has_attributes={'list_attr': [1, 2, 3]}))++    # Wrong value still fails+    assert not node.matches(SpanQuery(has_attributes={'data': {'foo': 2, 'bar': True}}))++    # Non-JSON string does not match a dict query+    assert not node.matches(SpanQuery(has_attributes={'not_json': {'key': 'value'}}))++    # Missing key fails+    assert not node.matches(SpanQuery(has_attributes={'missing': 'anything'}))+++def test_attribute_value_matches_helper():+    """Unit tests for the _attribute_value_matches helper function."""+    # Exact match+    assert _attribute_value_matches('hello', 'hello')+    assert _attribute_value_matches(42, 42)+    assert _attribute_value_matches(True, True)++    # JSON string vs dict+    assert _attribute_value_matches('{"a":1}', {'a': 1})+    assert not _attribute_value_matches('{"a":1}', {'a': 2})++    # JSON string vs list+    assert _attribute_value_matches('[1,2,3]', [1, 2, 3])++    # Non-JSON string vs dict+    assert not _attribute_value_matches('not json', {'a': 1})++    # None stored value+    assert not _attribute_value_matches(None, 'anything')++    # Both strings, no JSON fallback needed+    assert _attribute_value_matches('same', 'same')+    assert not _attribute_value_matches('different', 'other')++ async def test_span_tree_repr(span_tree: SpanTree):     assert repr(SpanTree()) == snapshot('<SpanTree />')     assert str(span_tree) == snapshot('<SpanTree num_roots=1 total_spans=6 />') 

What the agent reported

The reviewer confirms the fix is correct, minimal, and complete. The work is done.

Summary

What was wrong

SpanQuery(has_attributes={...}) in pydantic_evals/otel/span_tree.py compared the query value directly (self.attributes.get(key) == value) against the stored span attribute. logfire serializes non-primitive attribute values (dicts, lists, enums, large ints) to JSON strings before they reach the span, so a query like has_attributes={'data': {'foo': 1, 'bar': True}} was comparing a JSON string against a Python dict and never matched — even though the span carried that dict.

What I changed

  • pydantic_evals/pydantic_evals/otel/span_tree.py: added a module-level helper _matches_serialized_attribute(...) and routed the has_attributes matching through it. It does the raw == comparison first (short-circuiting True), and only on a miss falls back to comparing against logfire_json_dumps(query_value). logfire_json_dumps is imported lazily inside a try/except ImportError so the module still imports when logfire (an optional dependency) is absent.
  • tests/evals/test_otel.py: added test_span_query_has_attributes_with_dict_and_list, a regression test mirroring the reported reproduction.

The report literally suggested replacing the comparison with logfire_json_dumps(value). I deliberately made it a fallback rather than a blanket replacement, because a blanket replacement would break every existing scalar match (e.g. has_attributes={'depth': 3} — logfire_json_dumps(3) is the string '3', which does not equal the natively-stored int 3). The fallback fixes the dict/list case while preserving all scalar/str/bool matching.

Test evidence

  • ./bt-test tests/evals/test_otel.py → 26 passed (with the fix), including the new test.
  • Confirmed the new test genuinely captures the bug: with the production change temporarily reverted, test_span_query_has_attributes_with_dict_and_list fails at the exact reported assertion (SpanQuery(has_attributes={'data': {'foo': 1, 'bar': True}}) returns None), then passes once restored.
  • Reviewer confirmed: correct, minimal, no existing tests weakened, no unrelated changes, no scratch files.

Requirements disposition

  1. Original dict matches the span where the dict was used — Implemented & tested (test_otel.py:237, dict query now matches).
  2. Also list values / intuitive queries — Implemented & tested (test_otel.py:240).
  3. Backward compatibility with the existing JSON-string form — Implemented & tested (test_otel.py:243 still matches).
  4. Suggested edit at span_tree.py#L270 using logfire_json_dumps — Implemented as a fallback (not a blanket replacement, with evidence above that a literal replacement would break existing int/scalar tests such as {'depth': 3}). The suggested serialization function is used, guarded by a lazy import since logfire is optional.

Changes are left uncommitted in the working tree. (The three deleted files — .claude/settings.json, AGENTS.md, CLAUDE.md — were pre-existing deletions in the starting working tree, untouched by me.)