Backtestspydanticrun-20261002-175417#12435
Add FastMCP and MCP's python-sdk to third-party tests
Better
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches add the two requested jobs (FastMCP and the MCP python-sdk) to third-party.yml and wire them into create-issue-on-failure, so both address the issue's ask. Patch A is tightly scoped and follows the surrounding conventions. It installs both the local pydantic and `./pydantic-latest/pydantic-core` after uninstalling both packages. This matters if pydantic main depends on an unreleased pydantic-core in the same repo. Patch B has some genuinely useful refinements for FastMCP: - unsets FORCE_COLOR - excludes integration and subprocess markers - adds a timeout - disables the MCP smoke tests - uses the PrefectHQ repo name However, Patch B only uninstalls and reinstalls `pydantic`, not `pydantic-core`. That diverges from the pattern Patch A uses and risks a version mismatch or resolution failure against pydantic main. Patch B also brings substantial unrelated churn across other jobs: - adds `tests/` arguments to Beanie, Django Ninja and FastDepends - replaces actions/checkout with a raw `git clone` for ODMantic - duplicates the `tests` path in the ODMantic command - adds a redis service to Polar - rewrites Dify's uv setup and its test command None of this is requested, and it changes the behavior of existing jobs, which adds risk and obscures the fix. Neither patch adds tests, which is expected for CI config. Weighing correctness of the dependency install and scope, A is the cleaner, lower-risk fix. B's FastMCP filtering is a plus but does not outweigh the unrelated modifications and the missing pydantic-core install.
The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.
Solve
6m 41s
Wall-clock time in the container
Full test suite
No new failures
12,298 passed · 501 skipped · 12,799 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
90vs 141 merged
Changed lines, ours vs merged
The issue
View on GitHubThe two patches
Our patch+89 −1
.github/workflows/third-party.ymlindex 6a057644..9be0f004 100644@@ -877,6 +877,90 @@ jobs: - name: Run FastDepends tests run: uv run --no-sync pytest + test-fastmcp:+ name: Test FastMCP (main branch) on Python ${{ matrix.python-version }}+ # If 'schedule' was the trigger, don't run it on contributors' forks+ if: |+ github.event_name == 'workflow_dispatch' ||+ (github.event_name == 'schedule' && github.repository == 'pydantic/pydantic') ||+ (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'third-party-tests'))++ runs-on: ubuntu-latest+ strategy:+ fail-fast: false+ matrix:+ python-version: ['3.10', '3.11', '3.12', '3.13']+ steps:+ - name: Checkout FastMCP+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+ with:+ repository: jlowin/fastmcp+ persist-credentials: false++ - name: Checkout Pydantic+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+ with:+ path: pydantic-latest+ persist-credentials: false++ - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0+ with:+ python-version: ${{ matrix.python-version }}++ - name: Install FastMCP dependencies+ run: |+ uv sync --dev+ uv pip uninstall pydantic pydantic-core+ uv pip install -e ./pydantic-latest -e ./pydantic-latest/pydantic-core++ - name: List installed dependencies+ run: uv pip list++ - name: Run FastMCP tests+ run: uv run --no-sync pytest tests++ test-mcp-python-sdk:+ name: Test MCP Python SDK (main branch) on Python ${{ matrix.python-version }}+ # If 'schedule' was the trigger, don't run it on contributors' forks+ if: |+ github.event_name == 'workflow_dispatch' ||+ (github.event_name == 'schedule' && github.repository == 'pydantic/pydantic') ||+ (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'third-party-tests'))++ runs-on: ubuntu-latest+ strategy:+ fail-fast: false+ matrix:+ python-version: ['3.10', '3.11', '3.12', '3.13']+ steps:+ - name: Checkout MCP Python SDK+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+ with:+ repository: modelcontextprotocol/python-sdk+ persist-credentials: false++ - name: Checkout Pydantic+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+ with:+ path: pydantic-latest+ persist-credentials: false++ - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0+ with:+ python-version: ${{ matrix.python-version }}++ - name: Install MCP Python SDK dependencies+ run: |+ uv sync --frozen --all-extras+ uv pip uninstall pydantic pydantic-core+ uv pip install -e ./pydantic-latest -e ./pydantic-latest/pydantic-core++ - name: List installed dependencies+ run: uv pip list++ - name: Run MCP Python SDK tests+ run: uv run --frozen --no-sync pytest+ create-issue-on-failure: name: Create an issue if tests failed runs-on: ubuntu-latest@@ -897,6 +981,8 @@ jobs: - test-redis-om - test-django-ninja - test-fastdepends+ - test-fastmcp+ - test-mcp-python-sdk # Issue report disabled for now due to flakiness: if: | always() &&@@ -919,7 +1005,9 @@ jobs: needs.test-pydantic-xml.result == 'failure' || needs.test-redis-om.result == 'failure' || needs.test-django-ninja.result == 'failure' ||- needs.test-fastdepends.result == 'failure'+ needs.test-fastdepends.result == 'failure' ||+ needs.test-fastmcp.result == 'failure' ||+ needs.test-mcp-python-sdk.result == 'failure' ) permissions: issues: write
Merged fix · PR #13553+127 −14
.github/workflows/third-party.ymlindex 6a057644a..bdc248775 100644@@ -167,7 +167,7 @@ jobs: run: pip list - name: Run Beanie tests- run: pytest -v+ run: pytest -v tests/ test-openapi-python-client: name: Test openapi-python-client (main branch) on Python ${{ matrix.python-version }}@@ -277,10 +277,10 @@ jobs: steps: - name: Checkout ODMantic- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0- with:- repository: art049/odmantic- persist-credentials: false+ # Not using actions/checkout, as the ODMantic repository has a stale gitlink entry+ # ('.mongodb-cluster-action', with no matching '.gitmodules' entry) that makes the+ # action's 'git submodule foreach' auth cleanup fail with exit code 128.+ run: git clone --depth=1 https://github.com/art049/odmantic.git . - name: Checkout Pydantic uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0@@ -313,7 +313,7 @@ jobs: - name: Run ODMantic tests # Disabled tests, as per https://github.com/art049/odmantic/issues/512:- run: uv run --no-sync pytest tests -k 'not test_custom_bson_serializable and not test_sync_custom_bson_serializable and not test_with_bson_serializer_override_builtin_bson'+ run: uv run --no-sync pytest tests -k 'not test_custom_bson_serializable and not test_sync_custom_bson_serializable and not test_with_bson_serializer_override_builtin_bson' tests/ env: TEST_MONGO_URI: mongodb://localhost:27017/ TEST_MONGO_MODE: standalone@@ -350,6 +350,16 @@ jobs: ports: - 5432:5432 + redis:+ image: redis:7.4-alpine+ options: >-+ --health-cmd "redis-cli ping"+ --health-interval 10s+ --health-timeout 5s+ --health-retries 5+ ports:+ - 6379:6379+ minio: image: bitnamilegacy/minio:2024.5.28 ports:@@ -435,7 +445,7 @@ jobs: - name: 🐍 Run polar tests (pytest) working-directory: ./server- run: uv run --no-sync pytest -n auto --no-cov+ run: uv run --no-sync pytest -n auto --no-cov tests/ test-bentoml: name: Test BentoML (main branch) on Python ${{ matrix.python-version }}@@ -627,9 +637,8 @@ jobs: with: python-version: ${{ matrix.python-version }} - - name: Install uv- shell: bash- run: pip install uv~=0.7.0+ - name: Set up uv+ uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 - name: Install Dify dependencies run: |@@ -641,7 +650,7 @@ jobs: run: uv pip --directory api list - name: Run Dify unit tests- run: uv run --no-sync --project api bash dev/pytest/pytest_unit_tests.sh+ run: uv run --no-sync --directory api pytest tests/unit_tests/ test-cadwyn: name: Test Cadwyn (main branch) on Python ${{ matrix.python-version }}@@ -833,7 +842,7 @@ jobs: run: pip list - name: Run Django Ninja tests- run: pytest+ run: pytest tests/ test-fastdepends: name: Test FastDepends (main branch) on Python ${{ matrix.python-version }}@@ -875,7 +884,107 @@ jobs: run: uv pip list - name: Run FastDepends tests- run: uv run --no-sync pytest+ run: uv run --no-sync pytest tests/++ test-mcp-python-sdk:+ name: Test MCP Python SDK (main branch) on Python ${{ matrix.python-version }}+ if: |+ github.event_name == 'workflow_dispatch' ||+ (github.event_name == 'schedule' && github.repository == 'pydantic/pydantic') ||+ (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'third-party-tests'))++ runs-on: ubuntu-latest+ strategy:+ fail-fast: false+ matrix:+ python-version: ['3.10', '3.11', '3.12', '3.13', '3.14']+ steps:+ - name: Checkout MCP Python SDK+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+ with:+ repository: modelcontextprotocol/python-sdk+ persist-credentials: false++ - name: Checkout Pydantic+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+ with:+ path: pydantic-latest+ persist-credentials: false++ - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0+ with:+ enable-cache: true+ python-version: ${{ matrix.python-version }}++ - name: Install MCP Python SDK dependencies+ run: |+ uv venv --python ${{ matrix.python-version }}+ uv sync --all-extras --python ${{ matrix.python-version }}+ uv pip uninstall pydantic+ uv pip install -e ./pydantic-latest++ - name: List installed dependencies+ run: uv pip list++ - name: Run MCP Python SDK tests+ shell: bash+ env:+ # tests/examples/test_stories_smoke.py is gated on this var; it spawns real+ # stdio + uvicorn subprocesses, so we don't run it for third-party tests:+ MCP_EXAMPLES_SMOKE: '0'+ run: uv run --frozen --no-sync pytest -n auto tests/++ test-fastmcp:+ name: Test FastMCP (main branch) on Python ${{ matrix.python-version }}+ if: |+ github.event_name == 'workflow_dispatch' ||+ (github.event_name == 'schedule' && github.repository == 'pydantic/pydantic') ||+ (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'third-party-tests'))++ runs-on: ubuntu-latest+ strategy:+ fail-fast: false+ matrix:+ python-version: ['3.10', '3.13']+ steps:+ - name: Checkout FastMCP+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+ with:+ repository: PrefectHQ/fastmcp+ persist-credentials: false++ - name: Checkout Pydantic+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+ with:+ path: pydantic-latest+ persist-credentials: false++ - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0+ with:+ enable-cache: true+ python-version: ${{ matrix.python-version }}++ - name: Install FastMCP dependencies+ run: |+ uv sync --no-progress+ uv pip uninstall pydantic+ uv pip install -e ./pydantic-latest++ - name: List installed dependencies+ run: uv pip list++ # unit tests only (see https://github.com/PrefectHQ/fastmcp/blob/main/.github/actions/run-pytest/action.yml):+ - name: Run FastMCP tests+ # The workflow-level FORCE_COLOR makes Rich emit ANSI codes in captured output,+ # breaking FastMCP's CLI tests (which assert on plain text/JSON output):+ run: |+ unset FORCE_COLOR+ uv run --frozen --no-sync pytest \+ --inline-snapshot=disable \+ --timeout=5 \+ --durations=50 \+ -m "not integration and not client_process and not subprocess_heavy and not conformance" \+ tests/ create-issue-on-failure: name: Create an issue if tests failed@@ -897,6 +1006,8 @@ jobs: - test-redis-om - test-django-ninja - test-fastdepends+ - test-mcp-python-sdk+ - test-fastmcp # Issue report disabled for now due to flakiness: if: | always() &&@@ -919,7 +1030,9 @@ jobs: needs.test-pydantic-xml.result == 'failure' || needs.test-redis-om.result == 'failure' || needs.test-django-ninja.result == 'failure' ||- needs.test-fastdepends.result == 'failure'+ needs.test-fastdepends.result == 'failure' ||+ needs.test-mcp-python-sdk.result == 'failure' ||+ needs.test-fastmcp.result == 'failure' ) permissions: issues: write