Backtestspydanticrun-20260929-204949#12435
Add FastMCP and MCP's python-sdk to third-party tests
The merged fix was judged better than our patch.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches add the two requested jobs (FastMCP + MCP python-sdk), wire them into `create-issue-on-failure` needs, and gate them identically. The deciding factors are correctness of the new jobs and their realism. Patch A's install step runs `uv pip install -e ./pydantic-latest/pydantic-core`, but pydantic-core is a separate repository (pydantic/pydantic-core), not a subdirectory of the pydantic checkout, so that path does not exist and the step would fail; Patch B correctly reinstalls only pydantic. Patch B also handles the practical realities of running these third-party suites: it excludes integration/subprocess/conformance markers and the smoke test gated on MCP_EXAMPLES_SMOKE, unsets FORCE_COLOR (which would otherwise break FastMCP's CLI text/JSON assertions), and applies timeouts and `-n auto`—all of which make the jobs likely to actually pass rather than fail on unrelated infra. Patch A hardcodes `jlowin/fastmcp` matching the issue link, while B uses `PrefectHQ/fastmcp`; that's a minor difference and repos redirect. Against Patch B is real scope creep: it also touches Beanie, ODMantic (a defensible git-clone workaround, but leaving a redundant `pytest tests ... tests/` double positional arg), Polar (adds a redis service), Dify, and Django Ninja/FastDepends with appended `tests/`. That churn is unrelated to the ask and slightly sloppy, but it adds little risk and doesn't obscure the new jobs. Neither patch adds test files, as expected for a CI-config change. On correctness of the actual deliverable, B is meaningfully more sound despite its extra noise.
The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.
Solve
5m 36s7 turns
Wall-clock time in the container
Full test suite
No new failures
12,298 passed · 501 skipped · 12,799 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
90vs 141 merged
Changed lines, ours vs merged
The issue
View on GitHubThe two patches
.github/workflows/third-party.ymlindex 6a057644..8db05eed 100644@@ -877,6 +877,90 @@ jobs: - name: Run FastDepends tests run: uv run --no-sync pytest + test-fastmcp:+ name: Test FastMCP (main branch) on Python ${{ matrix.python-version }}+ # If 'schedule' was the trigger, don't run it on contributors' forks+ if: |+ github.event_name == 'workflow_dispatch' ||+ (github.event_name == 'schedule' && github.repository == 'pydantic/pydantic') ||+ (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'third-party-tests'))++ runs-on: ubuntu-latest+ strategy:+ fail-fast: false+ matrix:+ python-version: ['3.10', '3.11', '3.12', '3.13']+ steps:+ - name: Checkout FastMCP+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+ with:+ repository: jlowin/fastmcp+ persist-credentials: false++ - name: Checkout Pydantic+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+ with:+ path: pydantic-latest+ persist-credentials: false++ - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0+ with:+ python-version: ${{ matrix.python-version }}++ - name: Install FastMCP dependencies+ run: |+ uv sync --locked+ uv pip uninstall pydantic pydantic-core+ uv pip install -e ./pydantic-latest -e ./pydantic-latest/pydantic-core++ - name: List installed dependencies+ run: uv pip list++ - name: Run FastMCP tests+ run: uv run --no-sync pytest tests++ test-mcp-python-sdk:+ name: Test MCP Python SDK (main branch) on Python ${{ matrix.python-version }}+ # If 'schedule' was the trigger, don't run it on contributors' forks+ if: |+ github.event_name == 'workflow_dispatch' ||+ (github.event_name == 'schedule' && github.repository == 'pydantic/pydantic') ||+ (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'third-party-tests'))++ runs-on: ubuntu-latest+ strategy:+ fail-fast: false+ matrix:+ python-version: ['3.10', '3.11', '3.12', '3.13']+ steps:+ - name: Checkout MCP Python SDK+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+ with:+ repository: modelcontextprotocol/python-sdk+ persist-credentials: false++ - name: Checkout Pydantic+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+ with:+ path: pydantic-latest+ persist-credentials: false++ - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0+ with:+ python-version: ${{ matrix.python-version }}++ - name: Install MCP Python SDK dependencies+ run: |+ uv sync --locked --all-extras+ uv pip uninstall pydantic pydantic-core+ uv pip install -e ./pydantic-latest -e ./pydantic-latest/pydantic-core++ - name: List installed dependencies+ run: uv pip list++ - name: Run MCP Python SDK tests+ run: uv run --no-sync pytest+ create-issue-on-failure: name: Create an issue if tests failed runs-on: ubuntu-latest@@ -897,6 +981,8 @@ jobs: - test-redis-om - test-django-ninja - test-fastdepends+ - test-fastmcp+ - test-mcp-python-sdk # Issue report disabled for now due to flakiness: if: | always() &&@@ -919,7 +1005,9 @@ jobs: needs.test-pydantic-xml.result == 'failure' || needs.test-redis-om.result == 'failure' || needs.test-django-ninja.result == 'failure' ||- needs.test-fastdepends.result == 'failure'+ needs.test-fastdepends.result == 'failure' ||+ needs.test-fastmcp.result == 'failure' ||+ needs.test-mcp-python-sdk.result == 'failure' ) permissions: issues: write
.github/workflows/third-party.ymlindex 6a057644a..bdc248775 100644@@ -167,7 +167,7 @@ jobs: run: pip list - name: Run Beanie tests- run: pytest -v+ run: pytest -v tests/ test-openapi-python-client: name: Test openapi-python-client (main branch) on Python ${{ matrix.python-version }}@@ -277,10 +277,10 @@ jobs: steps: - name: Checkout ODMantic- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0- with:- repository: art049/odmantic- persist-credentials: false+ # Not using actions/checkout, as the ODMantic repository has a stale gitlink entry+ # ('.mongodb-cluster-action', with no matching '.gitmodules' entry) that makes the+ # action's 'git submodule foreach' auth cleanup fail with exit code 128.+ run: git clone --depth=1 https://github.com/art049/odmantic.git . - name: Checkout Pydantic uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0@@ -313,7 +313,7 @@ jobs: - name: Run ODMantic tests # Disabled tests, as per https://github.com/art049/odmantic/issues/512:- run: uv run --no-sync pytest tests -k 'not test_custom_bson_serializable and not test_sync_custom_bson_serializable and not test_with_bson_serializer_override_builtin_bson'+ run: uv run --no-sync pytest tests -k 'not test_custom_bson_serializable and not test_sync_custom_bson_serializable and not test_with_bson_serializer_override_builtin_bson' tests/ env: TEST_MONGO_URI: mongodb://localhost:27017/ TEST_MONGO_MODE: standalone@@ -350,6 +350,16 @@ jobs: ports: - 5432:5432 + redis:+ image: redis:7.4-alpine+ options: >-+ --health-cmd "redis-cli ping"+ --health-interval 10s+ --health-timeout 5s+ --health-retries 5+ ports:+ - 6379:6379+ minio: image: bitnamilegacy/minio:2024.5.28 ports:@@ -435,7 +445,7 @@ jobs: - name: 🐍 Run polar tests (pytest) working-directory: ./server- run: uv run --no-sync pytest -n auto --no-cov+ run: uv run --no-sync pytest -n auto --no-cov tests/ test-bentoml: name: Test BentoML (main branch) on Python ${{ matrix.python-version }}@@ -627,9 +637,8 @@ jobs: with: python-version: ${{ matrix.python-version }} - - name: Install uv- shell: bash- run: pip install uv~=0.7.0+ - name: Set up uv+ uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 - name: Install Dify dependencies run: |@@ -641,7 +650,7 @@ jobs: run: uv pip --directory api list - name: Run Dify unit tests- run: uv run --no-sync --project api bash dev/pytest/pytest_unit_tests.sh+ run: uv run --no-sync --directory api pytest tests/unit_tests/ test-cadwyn: name: Test Cadwyn (main branch) on Python ${{ matrix.python-version }}@@ -833,7 +842,7 @@ jobs: run: pip list - name: Run Django Ninja tests- run: pytest+ run: pytest tests/ test-fastdepends: name: Test FastDepends (main branch) on Python ${{ matrix.python-version }}@@ -875,7 +884,107 @@ jobs: run: uv pip list - name: Run FastDepends tests- run: uv run --no-sync pytest+ run: uv run --no-sync pytest tests/++ test-mcp-python-sdk:+ name: Test MCP Python SDK (main branch) on Python ${{ matrix.python-version }}+ if: |+ github.event_name == 'workflow_dispatch' ||+ (github.event_name == 'schedule' && github.repository == 'pydantic/pydantic') ||+ (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'third-party-tests'))++ runs-on: ubuntu-latest+ strategy:+ fail-fast: false+ matrix:+ python-version: ['3.10', '3.11', '3.12', '3.13', '3.14']+ steps:+ - name: Checkout MCP Python SDK+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+ with:+ repository: modelcontextprotocol/python-sdk+ persist-credentials: false++ - name: Checkout Pydantic+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+ with:+ path: pydantic-latest+ persist-credentials: false++ - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0+ with:+ enable-cache: true+ python-version: ${{ matrix.python-version }}++ - name: Install MCP Python SDK dependencies+ run: |+ uv venv --python ${{ matrix.python-version }}+ uv sync --all-extras --python ${{ matrix.python-version }}+ uv pip uninstall pydantic+ uv pip install -e ./pydantic-latest++ - name: List installed dependencies+ run: uv pip list++ - name: Run MCP Python SDK tests+ shell: bash+ env:+ # tests/examples/test_stories_smoke.py is gated on this var; it spawns real+ # stdio + uvicorn subprocesses, so we don't run it for third-party tests:+ MCP_EXAMPLES_SMOKE: '0'+ run: uv run --frozen --no-sync pytest -n auto tests/++ test-fastmcp:+ name: Test FastMCP (main branch) on Python ${{ matrix.python-version }}+ if: |+ github.event_name == 'workflow_dispatch' ||+ (github.event_name == 'schedule' && github.repository == 'pydantic/pydantic') ||+ (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'third-party-tests'))++ runs-on: ubuntu-latest+ strategy:+ fail-fast: false+ matrix:+ python-version: ['3.10', '3.13']+ steps:+ - name: Checkout FastMCP+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+ with:+ repository: PrefectHQ/fastmcp+ persist-credentials: false++ - name: Checkout Pydantic+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+ with:+ path: pydantic-latest+ persist-credentials: false++ - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0+ with:+ enable-cache: true+ python-version: ${{ matrix.python-version }}++ - name: Install FastMCP dependencies+ run: |+ uv sync --no-progress+ uv pip uninstall pydantic+ uv pip install -e ./pydantic-latest++ - name: List installed dependencies+ run: uv pip list++ # unit tests only (see https://github.com/PrefectHQ/fastmcp/blob/main/.github/actions/run-pytest/action.yml):+ - name: Run FastMCP tests+ # The workflow-level FORCE_COLOR makes Rich emit ANSI codes in captured output,+ # breaking FastMCP's CLI tests (which assert on plain text/JSON output):+ run: |+ unset FORCE_COLOR+ uv run --frozen --no-sync pytest \+ --inline-snapshot=disable \+ --timeout=5 \+ --durations=50 \+ -m "not integration and not client_process and not subprocess_heavy and not conformance" \+ tests/ create-issue-on-failure: name: Create an issue if tests failed@@ -897,6 +1006,8 @@ jobs: - test-redis-om - test-django-ninja - test-fastdepends+ - test-mcp-python-sdk+ - test-fastmcp # Issue report disabled for now due to flakiness: if: | always() &&@@ -919,7 +1030,9 @@ jobs: needs.test-pydantic-xml.result == 'failure' || needs.test-redis-om.result == 'failure' || needs.test-django-ninja.result == 'failure' ||- needs.test-fastdepends.result == 'failure'+ needs.test-fastdepends.result == 'failure' ||+ needs.test-mcp-python-sdk.result == 'failure' ||+ needs.test-fastmcp.result == 'failure' ) permissions: issues: write
What the agent reported
The change is complete and verified. Here is my final report.
What the task asked
Add FastMCP (jlowin/fastmcp) and MCP's python-sdk (modelcontextprotocol/python-sdk) to pydantic's third-party test suite — the daily/labelled CI that runs downstream libraries' test suites against a local pydantic checkout to catch regressions.
What I found
The third-party test mechanism is not a Python test suite with a ./bt-test runner — it lives entirely in one GitHub Actions workflow, .github/workflows/third-party.yml. Each downstream library is a single job that: checks out the library, checks out pydantic into pydantic-latest, installs the library's deps, force-reinstalls pydantic/pydantic-core editable from ./pydantic-latest, then runs the library's tests. docs/contributing.md:264 directs contributors to request inclusion rather than documenting mechanical steps, so no docs change is required. There is nothing to reproduce with ./bt-test; the applicable gate for a workflow file is YAML validity plus fidelity to the in-file idiom.
What I changed (.github/workflows/third-party.yml, uncommitted)
- Added job
test-fastmcp(jlowin/fastmcp) —uv-based, mirroring the existing FastAPI/FastDepends jobs: pinned action SHAs, the fork/third-party-tests-label guard,uv sync --locked, editable reinstall of both pydantic and pydantic-core,uv run --no-sync pytest tests, matrix Python 3.10–3.13. - Added job
test-mcp-python-sdk(modelcontextprotocol/python-sdk) — same idiom, usinguv sync --locked --all-extras(the SDK ships optional extras) anduv run --no-sync pytest. - Registered both jobs in the
create-issue-on-failurejob'sneeds:list and itsif:failure-aggregation condition, keeping the boolean chain valid.
Network was blocked (backtest barrier), so I could not fetch the two projects' upstream workflows to copy exact sync flags; I modeled the jobs on the file's own established uv convention (uv sync --locked), which is the closest defensible match.
Test evidence
- No production code or existing tests changed — this is a CI-config-only addition, so no regression test applies and no existing test could exercise it.
- YAML parse + structural check:
test-fastmcpandtest-mcp-python-sdkboth present injobs;create-issue-on-failure.needsends with['test-fastdepends', 'test-fastmcp', 'test-mcp-python-sdk']. git diffconfirms only.github/workflows/third-party.ymlis touched; no other job reordered/reformatted; no scratch files.- Reviewer confirmed idiom fidelity (correct pinned SHAs, guards,
persist-credentials: false, dual editable reinstall,--no-sync), correct wiring into bothneeds:andif:, and a syntactically valid||chain.
Requirements checklist
- Add FastMCP (
jlowin/fastmcp) to third-party tests — ✅ Implemented (test-fastmcpjob + failure wiring). - Add MCP's python-sdk (
modelcontextprotocol/python-sdk) to third-party tests — ✅ Implemented (test-mcp-python-sdkjob + failure wiring). - Docs update — ⏭️ Skipped with evidence:
docs/contributing.md:264-277intentionally documents only inclusion criteria and points contributors to a feature request; the mechanical "how" lives solely in the workflow header comments, so no docs edit is warranted.
All changes are left uncommitted in the working tree.