Backtestspydanticrun-20260929-204949#12435

Issue #12435 Merged fix PR #13553 Run 29 Sep 2026 · run-20260929-204949 pydantic/pydantic

Add FastMCP and MCP's python-sdk to third-party tests

Worse

The merged fix was judged better than our patch.

In this judgment, Patch A = our agent, Patch B = the merged fix.

Both patches add the two requested jobs (FastMCP + MCP python-sdk), wire them into `create-issue-on-failure` needs, and gate them identically. The deciding factors are correctness of the new jobs and their realism. Patch A's install step runs `uv pip install -e ./pydantic-latest/pydantic-core`, but pydantic-core is a separate repository (pydantic/pydantic-core), not a subdirectory of the pydantic checkout, so that path does not exist and the step would fail; Patch B correctly reinstalls only pydantic. Patch B also handles the practical realities of running these third-party suites: it excludes integration/subprocess/conformance markers and the smoke test gated on MCP_EXAMPLES_SMOKE, unsets FORCE_COLOR (which would otherwise break FastMCP's CLI text/JSON assertions), and applies timeouts and `-n auto`—all of which make the jobs likely to actually pass rather than fail on unrelated infra. Patch A hardcodes `jlowin/fastmcp` matching the issue link, while B uses `PrefectHQ/fastmcp`; that's a minor difference and repos redirect. Against Patch B is real scope creep: it also touches Beanie, ODMantic (a defensible git-clone workaround, but leaving a redundant `pytest tests ... tests/` double positional arg), Polar (adds a redis service), Dify, and Django Ninja/FastDepends with appended `tests/`. That churn is unrelated to the ask and slightly sloppy, but it adds little risk and doesn't obscure the new jobs. Neither patch adds test files, as expected for a CI-config change. On correctness of the actual deliverable, B is meaningfully more sound despite its extra noise.

The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.

Solve

5m 36s7 turns

Wall-clock time in the container

Full test suite

No new failures

12,298 passed · 501 skipped · 12,799 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

90vs 141 merged

Changed lines, ours vs merged

The two patches

Our patch+89 −1
.github/workflows/third-party.ymlindex 6a057644..8db05eed 100644@@ -877,6 +877,90 @@ jobs:       - name: Run FastDepends tests         run: uv run --no-sync pytest +  test-fastmcp:+    name: Test FastMCP (main branch) on Python ${{ matrix.python-version }}+    # If 'schedule' was the trigger, don't run it on contributors' forks+    if: |+      github.event_name == 'workflow_dispatch' ||+      (github.event_name == 'schedule' && github.repository == 'pydantic/pydantic') ||+      (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'third-party-tests'))++    runs-on: ubuntu-latest+    strategy:+      fail-fast: false+      matrix:+        python-version: ['3.10', '3.11', '3.12', '3.13']+    steps:+      - name: Checkout FastMCP+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+        with:+          repository: jlowin/fastmcp+          persist-credentials: false++      - name: Checkout Pydantic+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+        with:+          path: pydantic-latest+          persist-credentials: false++      - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0+        with:+          python-version: ${{ matrix.python-version }}++      - name: Install FastMCP dependencies+        run: |+          uv sync --locked+          uv pip uninstall pydantic pydantic-core+          uv pip install -e ./pydantic-latest -e ./pydantic-latest/pydantic-core++      - name: List installed dependencies+        run: uv pip list++      - name: Run FastMCP tests+        run: uv run --no-sync pytest tests++  test-mcp-python-sdk:+    name: Test MCP Python SDK (main branch) on Python ${{ matrix.python-version }}+    # If 'schedule' was the trigger, don't run it on contributors' forks+    if: |+      github.event_name == 'workflow_dispatch' ||+      (github.event_name == 'schedule' && github.repository == 'pydantic/pydantic') ||+      (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'third-party-tests'))++    runs-on: ubuntu-latest+    strategy:+      fail-fast: false+      matrix:+        python-version: ['3.10', '3.11', '3.12', '3.13']+    steps:+      - name: Checkout MCP Python SDK+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+        with:+          repository: modelcontextprotocol/python-sdk+          persist-credentials: false++      - name: Checkout Pydantic+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+        with:+          path: pydantic-latest+          persist-credentials: false++      - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0+        with:+          python-version: ${{ matrix.python-version }}++      - name: Install MCP Python SDK dependencies+        run: |+          uv sync --locked --all-extras+          uv pip uninstall pydantic pydantic-core+          uv pip install -e ./pydantic-latest -e ./pydantic-latest/pydantic-core++      - name: List installed dependencies+        run: uv pip list++      - name: Run MCP Python SDK tests+        run: uv run --no-sync pytest+   create-issue-on-failure:     name: Create an issue if tests failed     runs-on: ubuntu-latest@@ -897,6 +981,8 @@ jobs:       - test-redis-om       - test-django-ninja       - test-fastdepends+      - test-fastmcp+      - test-mcp-python-sdk     # Issue report disabled for now due to flakiness:     if: |       always() &&@@ -919,7 +1005,9 @@ jobs:         needs.test-pydantic-xml.result == 'failure' ||         needs.test-redis-om.result == 'failure' ||         needs.test-django-ninja.result == 'failure' ||-        needs.test-fastdepends.result == 'failure'+        needs.test-fastdepends.result == 'failure' ||+        needs.test-fastmcp.result == 'failure' ||+        needs.test-mcp-python-sdk.result == 'failure'       )     permissions:       issues: write 
Merged fix · PR #13553+127 −14
.github/workflows/third-party.ymlindex 6a057644a..bdc248775 100644@@ -167,7 +167,7 @@ jobs:         run: pip list        - name: Run Beanie tests-        run: pytest -v+        run: pytest -v tests/    test-openapi-python-client:     name: Test openapi-python-client (main branch) on Python ${{ matrix.python-version }}@@ -277,10 +277,10 @@ jobs:      steps:       - name: Checkout ODMantic-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0-        with:-          repository: art049/odmantic-          persist-credentials: false+        # Not using actions/checkout, as the ODMantic repository has a stale gitlink entry+        # ('.mongodb-cluster-action', with no matching '.gitmodules' entry) that makes the+        # action's 'git submodule foreach' auth cleanup fail with exit code 128.+        run: git clone --depth=1 https://github.com/art049/odmantic.git .        - name: Checkout Pydantic         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0@@ -313,7 +313,7 @@ jobs:        - name: Run ODMantic tests         # Disabled tests, as per https://github.com/art049/odmantic/issues/512:-        run: uv run --no-sync pytest tests -k 'not test_custom_bson_serializable and not test_sync_custom_bson_serializable and not test_with_bson_serializer_override_builtin_bson'+        run: uv run --no-sync pytest tests -k 'not test_custom_bson_serializable and not test_sync_custom_bson_serializable and not test_with_bson_serializer_override_builtin_bson' tests/         env:           TEST_MONGO_URI: mongodb://localhost:27017/           TEST_MONGO_MODE: standalone@@ -350,6 +350,16 @@ jobs:         ports:           - 5432:5432 +      redis:+        image: redis:7.4-alpine+        options: >-+          --health-cmd "redis-cli ping"+          --health-interval 10s+          --health-timeout 5s+          --health-retries 5+        ports:+          - 6379:6379+       minio:         image: bitnamilegacy/minio:2024.5.28         ports:@@ -435,7 +445,7 @@ jobs:        - name: 🐍 Run polar tests (pytest)         working-directory: ./server-        run: uv run --no-sync pytest -n auto --no-cov+        run: uv run --no-sync pytest -n auto --no-cov tests/    test-bentoml:     name: Test BentoML (main branch) on Python ${{ matrix.python-version }}@@ -627,9 +637,8 @@ jobs:         with:           python-version: ${{ matrix.python-version }} -      - name: Install uv-        shell: bash-        run: pip install uv~=0.7.0+      - name: Set up uv+        uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0        - name: Install Dify dependencies         run: |@@ -641,7 +650,7 @@ jobs:         run: uv pip --directory api list        - name: Run Dify unit tests-        run: uv run --no-sync --project api bash dev/pytest/pytest_unit_tests.sh+        run: uv run --no-sync --directory api pytest tests/unit_tests/    test-cadwyn:     name: Test Cadwyn (main branch) on Python ${{ matrix.python-version }}@@ -833,7 +842,7 @@ jobs:         run: pip list        - name: Run Django Ninja tests-        run: pytest+        run: pytest tests/    test-fastdepends:     name: Test FastDepends (main branch) on Python ${{ matrix.python-version }}@@ -875,7 +884,107 @@ jobs:         run: uv pip list        - name: Run FastDepends tests-        run: uv run --no-sync pytest+        run: uv run --no-sync pytest tests/++  test-mcp-python-sdk:+    name: Test MCP Python SDK (main branch) on Python ${{ matrix.python-version }}+    if: |+      github.event_name == 'workflow_dispatch' ||+      (github.event_name == 'schedule' && github.repository == 'pydantic/pydantic') ||+      (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'third-party-tests'))++    runs-on: ubuntu-latest+    strategy:+      fail-fast: false+      matrix:+        python-version: ['3.10', '3.11', '3.12', '3.13', '3.14']+    steps:+      - name: Checkout MCP Python SDK+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+        with:+          repository: modelcontextprotocol/python-sdk+          persist-credentials: false++      - name: Checkout Pydantic+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+        with:+          path: pydantic-latest+          persist-credentials: false++      - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0+        with:+          enable-cache: true+          python-version: ${{ matrix.python-version }}++      - name: Install MCP Python SDK dependencies+        run: |+          uv venv --python ${{ matrix.python-version }}+          uv sync --all-extras --python ${{ matrix.python-version }}+          uv pip uninstall pydantic+          uv pip install -e ./pydantic-latest++      - name: List installed dependencies+        run: uv pip list++      - name: Run MCP Python SDK tests+        shell: bash+        env:+          # tests/examples/test_stories_smoke.py is gated on this var; it spawns real+          # stdio + uvicorn subprocesses, so we don't run it for third-party tests:+          MCP_EXAMPLES_SMOKE: '0'+        run: uv run --frozen --no-sync pytest -n auto tests/++  test-fastmcp:+    name: Test FastMCP (main branch) on Python ${{ matrix.python-version }}+    if: |+      github.event_name == 'workflow_dispatch' ||+      (github.event_name == 'schedule' && github.repository == 'pydantic/pydantic') ||+      (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'third-party-tests'))++    runs-on: ubuntu-latest+    strategy:+      fail-fast: false+      matrix:+        python-version: ['3.10', '3.13']+    steps:+      - name: Checkout FastMCP+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+        with:+          repository: PrefectHQ/fastmcp+          persist-credentials: false++      - name: Checkout Pydantic+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+        with:+          path: pydantic-latest+          persist-credentials: false++      - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0+        with:+          enable-cache: true+          python-version: ${{ matrix.python-version }}++      - name: Install FastMCP dependencies+        run: |+          uv sync --no-progress+          uv pip uninstall pydantic+          uv pip install -e ./pydantic-latest++      - name: List installed dependencies+        run: uv pip list++      # unit tests only (see https://github.com/PrefectHQ/fastmcp/blob/main/.github/actions/run-pytest/action.yml):+      - name: Run FastMCP tests+        # The workflow-level FORCE_COLOR makes Rich emit ANSI codes in captured output,+        # breaking FastMCP's CLI tests (which assert on plain text/JSON output):+        run: |+          unset FORCE_COLOR+          uv run --frozen --no-sync pytest \+            --inline-snapshot=disable \+            --timeout=5 \+            --durations=50 \+            -m "not integration and not client_process and not subprocess_heavy and not conformance" \+            tests/    create-issue-on-failure:     name: Create an issue if tests failed@@ -897,6 +1006,8 @@ jobs:       - test-redis-om       - test-django-ninja       - test-fastdepends+      - test-mcp-python-sdk+      - test-fastmcp     # Issue report disabled for now due to flakiness:     if: |       always() &&@@ -919,7 +1030,9 @@ jobs:         needs.test-pydantic-xml.result == 'failure' ||         needs.test-redis-om.result == 'failure' ||         needs.test-django-ninja.result == 'failure' ||-        needs.test-fastdepends.result == 'failure'+        needs.test-fastdepends.result == 'failure' ||+        needs.test-mcp-python-sdk.result == 'failure' ||+        needs.test-fastmcp.result == 'failure'       )     permissions:       issues: write 

What the agent reported

The change is complete and verified. Here is my final report.

What the task asked

Add FastMCP (jlowin/fastmcp) and MCP's python-sdk (modelcontextprotocol/python-sdk) to pydantic's third-party test suite — the daily/labelled CI that runs downstream libraries' test suites against a local pydantic checkout to catch regressions.

What I found

The third-party test mechanism is not a Python test suite with a ./bt-test runner — it lives entirely in one GitHub Actions workflow, .github/workflows/third-party.yml. Each downstream library is a single job that: checks out the library, checks out pydantic into pydantic-latest, installs the library's deps, force-reinstalls pydantic/pydantic-core editable from ./pydantic-latest, then runs the library's tests. docs/contributing.md:264 directs contributors to request inclusion rather than documenting mechanical steps, so no docs change is required. There is nothing to reproduce with ./bt-test; the applicable gate for a workflow file is YAML validity plus fidelity to the in-file idiom.

What I changed (.github/workflows/third-party.yml, uncommitted)

  1. Added job test-fastmcp (jlowin/fastmcp) — uv-based, mirroring the existing FastAPI/FastDepends jobs: pinned action SHAs, the fork/third-party-tests-label guard, uv sync --locked, editable reinstall of both pydantic and pydantic-core, uv run --no-sync pytest tests, matrix Python 3.10–3.13.
  2. Added job test-mcp-python-sdk (modelcontextprotocol/python-sdk) — same idiom, using uv sync --locked --all-extras (the SDK ships optional extras) and uv run --no-sync pytest.
  3. Registered both jobs in the create-issue-on-failure job's needs: list and its if: failure-aggregation condition, keeping the boolean chain valid.

Network was blocked (backtest barrier), so I could not fetch the two projects' upstream workflows to copy exact sync flags; I modeled the jobs on the file's own established uv convention (uv sync --locked), which is the closest defensible match.

Test evidence

  • No production code or existing tests changed — this is a CI-config-only addition, so no regression test applies and no existing test could exercise it.
  • YAML parse + structural check: test-fastmcp and test-mcp-python-sdk both present in jobs; create-issue-on-failure.needs ends with ['test-fastdepends', 'test-fastmcp', 'test-mcp-python-sdk'].
  • git diff confirms only .github/workflows/third-party.yml is touched; no other job reordered/reformatted; no scratch files.
  • Reviewer confirmed idiom fidelity (correct pinned SHAs, guards, persist-credentials: false, dual editable reinstall, --no-sync), correct wiring into both needs: and if:, and a syntactically valid || chain.

Requirements checklist

  • Add FastMCP (jlowin/fastmcp) to third-party tests — ✅ Implemented (test-fastmcp job + failure wiring).
  • Add MCP's python-sdk (modelcontextprotocol/python-sdk) to third-party tests — ✅ Implemented (test-mcp-python-sdk job + failure wiring).
  • Docs update — ⏭️ Skipped with evidence: docs/contributing.md:264-277 intentionally documents only inclusion criteria and points contributors to a feature request; the mechanical "how" lives solely in the workflow header comments, so no docs edit is warranted.

All changes are left uncommitted in the working tree.