Backtestsscrapyrun-20261025-211112#2578
Raising CloseSpider from DownloaderMiddleware doesn't work
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches fix the root cause identically: in Scraper.handle_downloader_output_async, a CloseSpider carried by the download Failure is now routed to engine.close_spider_async instead of to the request errback / 'Error downloading' log. The independent tests confirm both pass every scenario (process_request, process_response, async process_response, no reason, high-priority middleware). The difference is scope and risk. Patch A stays on the issue: a small scraper change factored into a `_close_spider_on_exception` helper reused by handle_spider_error, a docstring update, and a targeted robotstxt change so a CloseSpider raised while downloading robots.txt is re-raised instead of being swallowed by the generic `except Exception` (a real way the issue's symptom could persist). Its tests cover process_request, process_response, process_exception, the no-reason default, and the robots.txt path, and assert the spider stops early and nothing is logged as an error. Patch B bundles the same one-hunk fix with a large unrelated feature: a new DOWNLOADER_MIDDLEWARE_RESPONSE_EXCEPTIONS setting, rewritten process_response/process_exception chaining with None placeholders in the method deques, a changed `_process_exception` return type, a new DecompressionError, RETRY_EXCEPTIONS edits, a project-template default flip, and a deprecation warning that fires for any user middleware raising from process_response (including IgnoreRequest) when another user middleware defines process_exception. It also silently changes existing behaviour by no longer passing CloseSpider from process_request to process_exception methods. None of this is asked for by the issue, it buries the fix in ~600 lines of diff, and it introduces real behavioural risk for users. B's issue-specific tests are thinner than A's (two cases, no process_exception or no-reason case). On correctness the patches tie; on tests A is somewhat stronger; on scope B is clearly worse; on craft A's helper extraction is cleaner. A wins.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
16m 23s
Wall-clock time in the container
Full test suite
No new failures
2,197 passed · 323 skipped · 2,525 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
150vs 559 merged
Changed lines, ours vs merged
The issue
View on GitHubBehavior:
Raising a CloseSpider exception from a DownloaderMiddleware's process_response doesn't close the spider.
Instead, the scraper only outputs CloseSpider to stdout.
Expected behavior:
Spider should shut down.
Workarounds: