Backtestspydanticrun-20260923-170312#12435

Issue #12435 Merged fix PR # Run 23 Sep 2026 · run-20260923-170312 pydantic/pydantic

Add FastMCP and MCP's python-sdk to third-party tests

Worse

The merged fix was judged better than our patch.

In this judgment, Patch B = our agent, Patch A = the merged fix.

Both patches add the two required jobs (MCP python-sdk + FastMCP) with correct `if:` gating, the pydantic-latest checkout/uninstall/reinstall dance, and wiring into `create-issue-on-failure`'s `needs` and failure condition. The decisive difference is correctness of the actual test execution. Third-party CI here runs on a schedule and files an issue on any failure, so a job that naively runs the upstream project's *entire* suite will spuriously fail. Patch A shows real understanding of the operational reality of these two projects: it sets `MCP_EXAMPLES_SMOKE=0` to skip the smoke test that spawns real stdio/uvicorn subprocesses, deselects FastMCP's `integration`/`client_process`/`subprocess_heavy`/`conformance` markers, unsets the workflow-level `FORCE_COLOR` that would inject ANSI codes and break FastMCP's CLI text/JSON assertions, and uses `--frozen`/`-n auto`. Patch B just runs `uv run --no-sync pytest` for both — no marker filtering, no smoke gating, no FORCE_COLOR handling — which would collect integration and subprocess-heavy tests and almost certainly produce failing/flaky scheduled runs, defeating the purpose. Patch B's advantages are narrower: it uses `jlowin/fastmcp` (matching the issue text exactly) vs A's `PrefectHQ/fastmcp` (the org fastmcp now lives under; plausible but a slight risk if it doesn't redirect), and it is tightly scoped. Patch A's real weakness is scope creep: it bundles many unrelated edits (Beanie/Django-Ninja/FastDepends/polar `tests/` scoping, an ODMantic `git clone` gitlink workaround, a new redis service, a Dify uv/setup change). That churn counts against A on the scope criterion. But correctness is weighted first, and A's jobs are far more likely to run green and stay meaningful, whereas B's would generate false-alarm failures. The correctness edge on the primary change outweighs A's scope noise.

The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.

Solve

4m 19s15 turns

Wall-clock time in the container

Full test suite

No new failures

6,452 passed · 368 skipped · 6,820 total

Contamination audit

Unknown

See the audit notes for this run

Patch size

88vs 141 merged

Changed lines, ours vs merged

The two patches

Our patch+87 −1
.github/workflows/third-party.ymlindex 6a057644..d1017080 100644@@ -877,6 +877,88 @@ jobs:       - name: Run FastDepends tests         run: uv run --no-sync pytest +  test-fastmcp:+    name: Test FastMCP (main branch) on Python ${{ matrix.python-version }}+    if: |+      github.event_name == 'workflow_dispatch' ||+      (github.event_name == 'schedule' && github.repository == 'pydantic/pydantic') ||+      (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'third-party-tests'))++    runs-on: ubuntu-latest+    strategy:+      fail-fast: false+      matrix:+        python-version: ['3.10', '3.11', '3.12', '3.13']+    steps:+      - name: Checkout FastMCP+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+        with:+          repository: jlowin/fastmcp+          persist-credentials: false++      - name: Checkout Pydantic+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+        with:+          path: pydantic-latest+          persist-credentials: false++      - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0+        with:+          python-version: ${{ matrix.python-version }}++      - name: Install FastMCP dependencies+        run: |+          uv sync+          uv pip uninstall pydantic+          uv pip install -e ./pydantic-latest++      - name: List installed dependencies+        run: uv pip list++      - name: Run FastMCP tests+        run: uv run --no-sync pytest++  test-mcp:+    name: Test MCP Python SDK (main branch) on Python ${{ matrix.python-version }}+    if: |+      github.event_name == 'workflow_dispatch' ||+      (github.event_name == 'schedule' && github.repository == 'pydantic/pydantic') ||+      (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'third-party-tests'))++    runs-on: ubuntu-latest+    strategy:+      fail-fast: false+      matrix:+        python-version: ['3.10', '3.11', '3.12', '3.13']+    steps:+      - name: Checkout MCP Python SDK+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+        with:+          repository: modelcontextprotocol/python-sdk+          persist-credentials: false++      - name: Checkout Pydantic+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+        with:+          path: pydantic-latest+          persist-credentials: false++      - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0+        with:+          python-version: ${{ matrix.python-version }}++      - name: Install MCP Python SDK dependencies+        run: |+          uv sync --all-extras+          uv pip uninstall pydantic+          uv pip install -e ./pydantic-latest++      - name: List installed dependencies+        run: uv pip list++      - name: Run MCP Python SDK tests+        run: uv run --no-sync pytest+   create-issue-on-failure:     name: Create an issue if tests failed     runs-on: ubuntu-latest@@ -897,6 +979,8 @@ jobs:       - test-redis-om       - test-django-ninja       - test-fastdepends+      - test-fastmcp+      - test-mcp     # Issue report disabled for now due to flakiness:     if: |       always() &&@@ -919,7 +1003,9 @@ jobs:         needs.test-pydantic-xml.result == 'failure' ||         needs.test-redis-om.result == 'failure' ||         needs.test-django-ninja.result == 'failure' ||-        needs.test-fastdepends.result == 'failure'+        needs.test-fastdepends.result == 'failure' ||+        needs.test-fastmcp.result == 'failure' ||+        needs.test-mcp.result == 'failure'       )     permissions:       issues: write 
Merged fix · PR #+127 −14
.github/workflows/third-party.ymlindex 6a057644a..bdc248775 100644@@ -167,7 +167,7 @@ jobs:         run: pip list        - name: Run Beanie tests-        run: pytest -v+        run: pytest -v tests/    test-openapi-python-client:     name: Test openapi-python-client (main branch) on Python ${{ matrix.python-version }}@@ -277,10 +277,10 @@ jobs:      steps:       - name: Checkout ODMantic-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0-        with:-          repository: art049/odmantic-          persist-credentials: false+        # Not using actions/checkout, as the ODMantic repository has a stale gitlink entry+        # ('.mongodb-cluster-action', with no matching '.gitmodules' entry) that makes the+        # action's 'git submodule foreach' auth cleanup fail with exit code 128.+        run: git clone --depth=1 https://github.com/art049/odmantic.git .        - name: Checkout Pydantic         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0@@ -313,7 +313,7 @@ jobs:        - name: Run ODMantic tests         # Disabled tests, as per https://github.com/art049/odmantic/issues/512:-        run: uv run --no-sync pytest tests -k 'not test_custom_bson_serializable and not test_sync_custom_bson_serializable and not test_with_bson_serializer_override_builtin_bson'+        run: uv run --no-sync pytest tests -k 'not test_custom_bson_serializable and not test_sync_custom_bson_serializable and not test_with_bson_serializer_override_builtin_bson' tests/         env:           TEST_MONGO_URI: mongodb://localhost:27017/           TEST_MONGO_MODE: standalone@@ -350,6 +350,16 @@ jobs:         ports:           - 5432:5432 +      redis:+        image: redis:7.4-alpine+        options: >-+          --health-cmd "redis-cli ping"+          --health-interval 10s+          --health-timeout 5s+          --health-retries 5+        ports:+          - 6379:6379+       minio:         image: bitnamilegacy/minio:2024.5.28         ports:@@ -435,7 +445,7 @@ jobs:        - name: 🐍 Run polar tests (pytest)         working-directory: ./server-        run: uv run --no-sync pytest -n auto --no-cov+        run: uv run --no-sync pytest -n auto --no-cov tests/    test-bentoml:     name: Test BentoML (main branch) on Python ${{ matrix.python-version }}@@ -627,9 +637,8 @@ jobs:         with:           python-version: ${{ matrix.python-version }} -      - name: Install uv-        shell: bash-        run: pip install uv~=0.7.0+      - name: Set up uv+        uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0        - name: Install Dify dependencies         run: |@@ -641,7 +650,7 @@ jobs:         run: uv pip --directory api list        - name: Run Dify unit tests-        run: uv run --no-sync --project api bash dev/pytest/pytest_unit_tests.sh+        run: uv run --no-sync --directory api pytest tests/unit_tests/    test-cadwyn:     name: Test Cadwyn (main branch) on Python ${{ matrix.python-version }}@@ -833,7 +842,7 @@ jobs:         run: pip list        - name: Run Django Ninja tests-        run: pytest+        run: pytest tests/    test-fastdepends:     name: Test FastDepends (main branch) on Python ${{ matrix.python-version }}@@ -875,7 +884,107 @@ jobs:         run: uv pip list        - name: Run FastDepends tests-        run: uv run --no-sync pytest+        run: uv run --no-sync pytest tests/++  test-mcp-python-sdk:+    name: Test MCP Python SDK (main branch) on Python ${{ matrix.python-version }}+    if: |+      github.event_name == 'workflow_dispatch' ||+      (github.event_name == 'schedule' && github.repository == 'pydantic/pydantic') ||+      (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'third-party-tests'))++    runs-on: ubuntu-latest+    strategy:+      fail-fast: false+      matrix:+        python-version: ['3.10', '3.11', '3.12', '3.13', '3.14']+    steps:+      - name: Checkout MCP Python SDK+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+        with:+          repository: modelcontextprotocol/python-sdk+          persist-credentials: false++      - name: Checkout Pydantic+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+        with:+          path: pydantic-latest+          persist-credentials: false++      - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0+        with:+          enable-cache: true+          python-version: ${{ matrix.python-version }}++      - name: Install MCP Python SDK dependencies+        run: |+          uv venv --python ${{ matrix.python-version }}+          uv sync --all-extras --python ${{ matrix.python-version }}+          uv pip uninstall pydantic+          uv pip install -e ./pydantic-latest++      - name: List installed dependencies+        run: uv pip list++      - name: Run MCP Python SDK tests+        shell: bash+        env:+          # tests/examples/test_stories_smoke.py is gated on this var; it spawns real+          # stdio + uvicorn subprocesses, so we don't run it for third-party tests:+          MCP_EXAMPLES_SMOKE: '0'+        run: uv run --frozen --no-sync pytest -n auto tests/++  test-fastmcp:+    name: Test FastMCP (main branch) on Python ${{ matrix.python-version }}+    if: |+      github.event_name == 'workflow_dispatch' ||+      (github.event_name == 'schedule' && github.repository == 'pydantic/pydantic') ||+      (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'third-party-tests'))++    runs-on: ubuntu-latest+    strategy:+      fail-fast: false+      matrix:+        python-version: ['3.10', '3.13']+    steps:+      - name: Checkout FastMCP+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+        with:+          repository: PrefectHQ/fastmcp+          persist-credentials: false++      - name: Checkout Pydantic+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+        with:+          path: pydantic-latest+          persist-credentials: false++      - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0+        with:+          enable-cache: true+          python-version: ${{ matrix.python-version }}++      - name: Install FastMCP dependencies+        run: |+          uv sync --no-progress+          uv pip uninstall pydantic+          uv pip install -e ./pydantic-latest++      - name: List installed dependencies+        run: uv pip list++      # unit tests only (see https://github.com/PrefectHQ/fastmcp/blob/main/.github/actions/run-pytest/action.yml):+      - name: Run FastMCP tests+        # The workflow-level FORCE_COLOR makes Rich emit ANSI codes in captured output,+        # breaking FastMCP's CLI tests (which assert on plain text/JSON output):+        run: |+          unset FORCE_COLOR+          uv run --frozen --no-sync pytest \+            --inline-snapshot=disable \+            --timeout=5 \+            --durations=50 \+            -m "not integration and not client_process and not subprocess_heavy and not conformance" \+            tests/    create-issue-on-failure:     name: Create an issue if tests failed@@ -897,6 +1006,8 @@ jobs:       - test-redis-om       - test-django-ninja       - test-fastdepends+      - test-mcp-python-sdk+      - test-fastmcp     # Issue report disabled for now due to flakiness:     if: |       always() &&@@ -919,7 +1030,9 @@ jobs:         needs.test-pydantic-xml.result == 'failure' ||         needs.test-redis-om.result == 'failure' ||         needs.test-django-ninja.result == 'failure' ||-        needs.test-fastdepends.result == 'failure'+        needs.test-fastdepends.result == 'failure' ||+        needs.test-mcp-python-sdk.result == 'failure' ||+        needs.test-fastmcp.result == 'failure'       )     permissions:       issues: write 

What the agent reported

The change is complete, localized, and verified.

Summary

What was requested: Add FastMCP (jlowin/fastmcp) and MCP's python-sdk (modelcontextprotocol/python-sdk) to pydantic's third-party test suite.

What was wrong / needed: This is a feature request, not a bug. The third-party test suite is defined entirely as GitHub Actions jobs in .github/workflows/third-party.yml (a daily cron that runs each downstream library's own tests against an editable install of latest pydantic). Neither library was present.

What I changed — all in .github/workflows/third-party.yml (+87/−1):

  • Added a test-fastmcp job (checks out jlowin/fastmcp) and a test-mcp job (checks out modelcontextprotocol/python-sdk), each mirroring the recent uv-based jobs (test-fastdepends/test-cadwyn): identical if: gate, pinned action SHAs, checkout of pydantic into pydantic-latest, install deps → uninstall pydantic → uv pip install -e ./pydantic-latest, list deps, run uv run --no-sync pytest. Matrix ['3.10','3.11','3.12','3.13']. test-mcp uses uv sync --all-extras (it ships several extras); test-fastmcp uses plain uv sync. Both install only pydantic editable (not the Rust pydantic-core), consistent with the reference jobs.
  • Wired both new job names into the create-issue-on-failure aggregator — its needs: list and its failure if: condition.

Test evidence: There is no runnable repository test for this CI-config file — ./bt-test runs pydantic's pytest suite, which does not exercise workflow YAML, and there is no local runner for the third-party workflow (it runs only in CI). I therefore verified by parsing the file with a YAML loader: it parses cleanly, both new jobs are present with the correct repository and name, and both appear in the aggregator's needs and if. git status confirms the diff touches only this one file (+87/−1). A reviewer subagent independently confirmed convention-match and that nothing unrelated changed.

Requirements disposition:

  1. Add FastMCP to third-party tests — implemented (test-fastmcp job + aggregator wiring); verified via YAML parse.
  2. Add MCP python-sdk to third-party tests — implemented (test-mcp job + aggregator wiring); verified via YAML parse.
  3. (Implicit convention) new jobs wired into create-issue-on-failure needs:/if: — implemented; verified.

Note: I could not confirm each downstream project's exact dependency-group/extra names or full green test runs without network access to those repos; the install steps follow the established pattern in this file and the workflow's own header instructions. Changes are left uncommitted in the working tree.