Backtestspydanticrun-20260930-144322#12435
Add FastMCP and MCP's python-sdk to third-party tests
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches add `test-fastmcp` and `test-mcp-python-sdk` jobs and wire them into `create-issue-on-failure`, so both address the issue's ask. Neither adds tests, which is expected for a CI-config change. **Patch A** is tightly scoped. It adds only the two new jobs. They follow the file's existing conventions: frozen `uv sync`, uninstall both `pydantic` and `pydantic-core`, then install both editable from the checked-out pydantic repo (`./pydantic-latest` and `./pydantic-latest/pydantic-core`). That keeps pydantic and its core in lockstep with the version under test. It mirrors FastMCP's own CI test selection. The one uncertainty is whether the `not integration_test` marker is still current. **Patch B** also adds the two jobs, with some thoughtful touches: it unsets `FORCE_COLOR` for FastMCP's CLI tests, uses finer-grained markers, disables the MCP smoke tests, and adds Python 3.14. However: - It uninstalls only `pydantic` and installs only `./pydantic-latest`. It never replaces `pydantic-core`, so the locked core can mismatch the pydantic under test, which undermines the purpose of the job. - It drops `--frozen` from FastMCP's `uv sync`. - It bundles substantial unrelated churn into the shared workflow: - rewrites the ODMantic checkout to a raw `git clone`; - appends a redundant `tests/` to the ODMantic `-k` command, which already passes `tests`; - adds a Redis service to the Polar job; - replaces Dify's install and test invocation; - appends `tests/` to Beanie, Polar, Django Ninja and FastDepends. These edits touch many other jobs, add risk of breaking them, and obscure the actual fix. Given A's correct handling of `pydantic-core`, its consistency with surrounding jobs, and its minimal scope versus B's risky churn, A is the better patch.
The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.
Solve
1m 29s12 turns
Wall-clock time in the container
Full test suite
No new failures
12,298 passed · 501 skipped · 12,799 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
89vs 141 merged
Changed lines, ours vs merged
The issue
View on GitHubThe two patches
.github/workflows/third-party.ymlindex 6a057644..7f2a3f43 100644@@ -877,6 +877,89 @@ jobs: - name: Run FastDepends tests run: uv run --no-sync pytest + test-fastmcp:+ name: Test FastMCP (main branch) on Python ${{ matrix.python-version }}+ if: |+ github.event_name == 'workflow_dispatch' ||+ (github.event_name == 'schedule' && github.repository == 'pydantic/pydantic') ||+ (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'third-party-tests'))++ runs-on: ubuntu-latest+ strategy:+ fail-fast: false+ matrix:+ python-version: ['3.10', '3.11', '3.12', '3.13']+ steps:+ - name: Checkout FastMCP+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+ with:+ repository: jlowin/fastmcp+ persist-credentials: false++ - name: Checkout Pydantic+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+ with:+ path: pydantic-latest+ persist-credentials: false++ - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0+ with:+ python-version: ${{ matrix.python-version }}++ - name: Install FastMCP dependencies+ run: |+ uv sync --no-progress --frozen+ uv pip uninstall pydantic pydantic-core+ uv pip install -e ./pydantic-latest -e ./pydantic-latest/pydantic-core++ - name: List installed dependencies+ run: uv pip list++ - name: Run FastMCP tests+ # Same selection as FastMCP's CI (integration tests require network access/credentials):+ run: uv run --no-sync pytest --inline-snapshot=disable tests -m 'not integration_test' --numprocesses auto --maxprocesses 4 --dist worksteal++ test-mcp-python-sdk:+ name: Test MCP Python SDK (main branch) on Python ${{ matrix.python-version }}+ if: |+ github.event_name == 'workflow_dispatch' ||+ (github.event_name == 'schedule' && github.repository == 'pydantic/pydantic') ||+ (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'third-party-tests'))++ runs-on: ubuntu-latest+ strategy:+ fail-fast: false+ matrix:+ python-version: ['3.10', '3.11', '3.12', '3.13']+ steps:+ - name: Checkout MCP Python SDK+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+ with:+ repository: modelcontextprotocol/python-sdk+ persist-credentials: false++ - name: Checkout Pydantic+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+ with:+ path: pydantic-latest+ persist-credentials: false++ - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0+ with:+ python-version: ${{ matrix.python-version }}++ - name: Install MCP Python SDK dependencies+ run: |+ uv sync --no-progress --frozen --all-extras+ uv pip uninstall pydantic pydantic-core+ uv pip install -e ./pydantic-latest -e ./pydantic-latest/pydantic-core++ - name: List installed dependencies+ run: uv pip list++ - name: Run MCP Python SDK tests+ run: uv run --frozen --no-sync pytest+ create-issue-on-failure: name: Create an issue if tests failed runs-on: ubuntu-latest@@ -897,6 +980,8 @@ jobs: - test-redis-om - test-django-ninja - test-fastdepends+ - test-fastmcp+ - test-mcp-python-sdk # Issue report disabled for now due to flakiness: if: | always() &&@@ -919,7 +1004,9 @@ jobs: needs.test-pydantic-xml.result == 'failure' || needs.test-redis-om.result == 'failure' || needs.test-django-ninja.result == 'failure' ||- needs.test-fastdepends.result == 'failure'+ needs.test-fastdepends.result == 'failure' ||+ needs.test-fastmcp.result == 'failure' ||+ needs.test-mcp-python-sdk.result == 'failure' ) permissions: issues: write
.github/workflows/third-party.ymlindex 6a057644a..bdc248775 100644@@ -167,7 +167,7 @@ jobs: run: pip list - name: Run Beanie tests- run: pytest -v+ run: pytest -v tests/ test-openapi-python-client: name: Test openapi-python-client (main branch) on Python ${{ matrix.python-version }}@@ -277,10 +277,10 @@ jobs: steps: - name: Checkout ODMantic- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0- with:- repository: art049/odmantic- persist-credentials: false+ # Not using actions/checkout, as the ODMantic repository has a stale gitlink entry+ # ('.mongodb-cluster-action', with no matching '.gitmodules' entry) that makes the+ # action's 'git submodule foreach' auth cleanup fail with exit code 128.+ run: git clone --depth=1 https://github.com/art049/odmantic.git . - name: Checkout Pydantic uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0@@ -313,7 +313,7 @@ jobs: - name: Run ODMantic tests # Disabled tests, as per https://github.com/art049/odmantic/issues/512:- run: uv run --no-sync pytest tests -k 'not test_custom_bson_serializable and not test_sync_custom_bson_serializable and not test_with_bson_serializer_override_builtin_bson'+ run: uv run --no-sync pytest tests -k 'not test_custom_bson_serializable and not test_sync_custom_bson_serializable and not test_with_bson_serializer_override_builtin_bson' tests/ env: TEST_MONGO_URI: mongodb://localhost:27017/ TEST_MONGO_MODE: standalone@@ -350,6 +350,16 @@ jobs: ports: - 5432:5432 + redis:+ image: redis:7.4-alpine+ options: >-+ --health-cmd "redis-cli ping"+ --health-interval 10s+ --health-timeout 5s+ --health-retries 5+ ports:+ - 6379:6379+ minio: image: bitnamilegacy/minio:2024.5.28 ports:@@ -435,7 +445,7 @@ jobs: - name: 🐍 Run polar tests (pytest) working-directory: ./server- run: uv run --no-sync pytest -n auto --no-cov+ run: uv run --no-sync pytest -n auto --no-cov tests/ test-bentoml: name: Test BentoML (main branch) on Python ${{ matrix.python-version }}@@ -627,9 +637,8 @@ jobs: with: python-version: ${{ matrix.python-version }} - - name: Install uv- shell: bash- run: pip install uv~=0.7.0+ - name: Set up uv+ uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 - name: Install Dify dependencies run: |@@ -641,7 +650,7 @@ jobs: run: uv pip --directory api list - name: Run Dify unit tests- run: uv run --no-sync --project api bash dev/pytest/pytest_unit_tests.sh+ run: uv run --no-sync --directory api pytest tests/unit_tests/ test-cadwyn: name: Test Cadwyn (main branch) on Python ${{ matrix.python-version }}@@ -833,7 +842,7 @@ jobs: run: pip list - name: Run Django Ninja tests- run: pytest+ run: pytest tests/ test-fastdepends: name: Test FastDepends (main branch) on Python ${{ matrix.python-version }}@@ -875,7 +884,107 @@ jobs: run: uv pip list - name: Run FastDepends tests- run: uv run --no-sync pytest+ run: uv run --no-sync pytest tests/++ test-mcp-python-sdk:+ name: Test MCP Python SDK (main branch) on Python ${{ matrix.python-version }}+ if: |+ github.event_name == 'workflow_dispatch' ||+ (github.event_name == 'schedule' && github.repository == 'pydantic/pydantic') ||+ (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'third-party-tests'))++ runs-on: ubuntu-latest+ strategy:+ fail-fast: false+ matrix:+ python-version: ['3.10', '3.11', '3.12', '3.13', '3.14']+ steps:+ - name: Checkout MCP Python SDK+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+ with:+ repository: modelcontextprotocol/python-sdk+ persist-credentials: false++ - name: Checkout Pydantic+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+ with:+ path: pydantic-latest+ persist-credentials: false++ - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0+ with:+ enable-cache: true+ python-version: ${{ matrix.python-version }}++ - name: Install MCP Python SDK dependencies+ run: |+ uv venv --python ${{ matrix.python-version }}+ uv sync --all-extras --python ${{ matrix.python-version }}+ uv pip uninstall pydantic+ uv pip install -e ./pydantic-latest++ - name: List installed dependencies+ run: uv pip list++ - name: Run MCP Python SDK tests+ shell: bash+ env:+ # tests/examples/test_stories_smoke.py is gated on this var; it spawns real+ # stdio + uvicorn subprocesses, so we don't run it for third-party tests:+ MCP_EXAMPLES_SMOKE: '0'+ run: uv run --frozen --no-sync pytest -n auto tests/++ test-fastmcp:+ name: Test FastMCP (main branch) on Python ${{ matrix.python-version }}+ if: |+ github.event_name == 'workflow_dispatch' ||+ (github.event_name == 'schedule' && github.repository == 'pydantic/pydantic') ||+ (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'third-party-tests'))++ runs-on: ubuntu-latest+ strategy:+ fail-fast: false+ matrix:+ python-version: ['3.10', '3.13']+ steps:+ - name: Checkout FastMCP+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+ with:+ repository: PrefectHQ/fastmcp+ persist-credentials: false++ - name: Checkout Pydantic+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+ with:+ path: pydantic-latest+ persist-credentials: false++ - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0+ with:+ enable-cache: true+ python-version: ${{ matrix.python-version }}++ - name: Install FastMCP dependencies+ run: |+ uv sync --no-progress+ uv pip uninstall pydantic+ uv pip install -e ./pydantic-latest++ - name: List installed dependencies+ run: uv pip list++ # unit tests only (see https://github.com/PrefectHQ/fastmcp/blob/main/.github/actions/run-pytest/action.yml):+ - name: Run FastMCP tests+ # The workflow-level FORCE_COLOR makes Rich emit ANSI codes in captured output,+ # breaking FastMCP's CLI tests (which assert on plain text/JSON output):+ run: |+ unset FORCE_COLOR+ uv run --frozen --no-sync pytest \+ --inline-snapshot=disable \+ --timeout=5 \+ --durations=50 \+ -m "not integration and not client_process and not subprocess_heavy and not conformance" \+ tests/ create-issue-on-failure: name: Create an issue if tests failed@@ -897,6 +1006,8 @@ jobs: - test-redis-om - test-django-ninja - test-fastdepends+ - test-mcp-python-sdk+ - test-fastmcp # Issue report disabled for now due to flakiness: if: | always() &&@@ -919,7 +1030,9 @@ jobs: needs.test-pydantic-xml.result == 'failure' || needs.test-redis-om.result == 'failure' || needs.test-django-ninja.result == 'failure' ||- needs.test-fastdepends.result == 'failure'+ needs.test-fastdepends.result == 'failure' ||+ needs.test-mcp-python-sdk.result == 'failure' ||+ needs.test-fastmcp.result == 'failure' ) permissions: issues: write
What the agent reported
I added FastMCP and MCP's python-sdk as two new jobs in .github/workflows/third-party.yml. They haven't run yet: this sandbox blocks network access, and nothing here can execute a GitHub Actions workflow. There was no bug here, just a request to extend the CI config.
What changed (all in .github/workflows/third-party.yml, left uncommitted):
test-fastmcp: checks outjlowin/fastmcpand runs on Python 3.10–3.13. It installs FastMCP's dependencies withuv sync --no-progress --frozen, removes the pinnedpydantic/pydantic-core, and installs the current Pydantic from./pydantic-latestin editable mode, as the file's header instructions ask. Tests run withuv run --no-sync pytest --inline-snapshot=disable tests -m 'not integration_test' --numprocesses auto --maxprocesses 4 --dist worksteal.test-mcp-python-sdk: checks outmodelcontextprotocol/python-sdkand runs on Python 3.10–3.13. Install isuv sync --no-progress --frozen --all-extras, then the same Pydantic swap, thenuv run --frozen --no-sync pytest.- Both jobs are added to the
needs:list and the failure check ofcreate-issue-on-failure. That job was already switched off withfalse &&and still is.
The layout, the trigger condition (label third-party-tests, the daily schedule, or a manual run) and the pinned action versions all match the existing jobs such as test-fastapi.
How it was checked:
- I couldn't fetch either project's own CI config. The test commands are my best recollection of what each project runs, not copied from their current files.
- The FastMCP command only works if FastMCP's default dev dependencies include
inline-snapshotandpytest-xdist. If the first run fails on those flags, compare the command against FastMCP's own workflow. jlowin/fastmcphas probably moved toPrefectHQ/fastmcp. Checkout should follow GitHub's redirect, but I kept the name given in the request.- No YAML parser is installed here, so I couldn't parse the file. A reviewer subagent compared the new jobs line by line with the existing ones and found no structural problems.
- I added no pytest test and didn't run
./bt-test. The change touches no library code, and nothing intests/covers the workflow files. The real test is the workflow itself: add thethird-party-testslabel to the PR, or start it manually from the Actions tab.
Requirements:
- Add FastMCP to the third-party tests: done (
test-fastmcp). - Add MCP's
python-sdkto the third-party tests: done (test-mcp-python-sdk).