Backtestsscrapyrun-20261025-211112#3552

Issue #3552 Merged fix PR #8037 Run 25 Oct 2026 · run-20261025-211112 scrapy/scrapy

--pdb catches some harmless internal exceptions

Better

Our patch was judged better than the merged fix.

In this judgment, Patch A = our agent, Patch B = the merged fix.

Both patches remove `failure.startDebugMode()` (the root cause: it breaks into pdb on every Failure construction, including the handled OpenSSL SysCallError from the issue) and replace it with a hook that only fires for reported errors. Both pass the independent tests for the issue's exact scenario (handled SSL error, successful crawl, real spider error still debugged).

Correctness differences: Patch B only hooks Python logging records at ERROR with exc_info. Twisted-reported errors (notably 'Unhandled error in Deferred', which the Scrapy log shows at ERROR via the PythonLoggingObserver bridge that carries no exc_info) will no longer enter the debugger under B, a regression versus base for a class of real errors. Patch A additionally observes Twisted's log for isError events with a Failure, catches exceptions that propagate out of the command (excluding UsageError), dedups so the same exception is debugged once even if logged by several components, and guards against fake `_Traceback` objects from cleaned Failures (B would hand such an object to pdb.post_mortem). A also keeps tracebacks through `Failure.cleanFailure` so late-logged failures remain debuggable. That monkeypatch is invasive, but it is scoped to --pdb, context-managed, and restored; B instead leaves its handler on the root logger permanently.

The one independent test A failed did so because A entered the debugger for a different, genuinely reported error ('[Errno 98] Address already in use', Twisted-logged in that environment), not for the handled 'harmless' Deferred error the test targets. A's code produces no log or observer event for an errback-consumed failure, and A's own unit and subprocess tests cover exactly that case, so this result does not show A leaving the issue broken; it reflects A's broader (and consistent with the issue's criterion: debug only what is reported as an error) coverage.

Tests: A adds unit tests for every path (handled failure not debugged, logged error debugged once, warning not debugged, cleaned failure, Twisted log.err, unhandled Deferred, propagated exception, UsageError, patch restoration) plus subprocess tests asserting the debugger is entered exactly once for reported errors and never for handled ones, and asserts startDebugMode is never called. B's subprocess tests are reasonable but thinner (one asserts only that '(Pdb)' appears, without checking exit status) and do not cover Twisted-reported errors.

B is admirably minimal, but A resolves the issue while preserving the debugger for all reported errors and is better tested; the extra machinery is justified and well-explained.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

17m 34s

Wall-clock time in the container

Full test suite

No new failures

175 passed · 3 skipped · 180 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

363vs 94 merged

Changed lines, ours vs merged

Hello,

I'm getting an error when trying to scrape HTTPS pages. Let's look at www.google.com as an example:

scrapy fetch --pdb 'https://www.google.com'

2018-12-27 13:45:04 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: scrapybot)
2018-12-27 13:45:04 [scrapy.utils.log] INFO: Versions: lxml 4.2.5.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.1, w3lib 1.19.0, Twisted 18.9.0, Python 3.6.7 (default, Oct 22 2018, 11:32:17) - [GCC 8.2.0], pyOpenSSL 18.0.0 (OpenSSL 1.1.0j  20 Nov 2018), cryptography 2.4.2, Platform Linux-4.15.0-36-generic-x86_64-with-Ubuntu-18.04-bionic
2018-12-27 13:45:04 [scrapy.crawler] INFO: Overridden settings: {}
2018-12-27 13:45:04 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats']
2018-12-27 13:45:04 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-12-27 13:45:04 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-12-27 13:45:04 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2018-12-27 13:45:04 [scrapy.core.engine] INFO: Spider opened
2018-12-27 13:45:04 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-12-27 13:45:04 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-12-27 13:45:04 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.google.com> (referer: None)
<!doctype html><html dir="rtl" itemscope="" itemtype="http://schema.org/WebPage" lang="ar-AE"><head><meta content="text/html; charset=UTF-8" http-equiv="Content-Type"><meta content="/images/branding/googleg/1x/googleg_standard_color_128dp.png" itemprop="image"><title>Google</title><script nonce="y8SVUeCln6pIIL+tT3bpDA==">(function(){window.google={kEI:'MK0kXLv5Ne-mrgTclpzYDg',kEXPI:'0,1353747,57,1958,1016,1406,697,528,730,326,293,1180,30,524,27,144,532,895,546,6,2335661,186,32,68,329226,1294,12383,4855,32691,15248,867,6057,6106,12919,1410,2192,364,3319,5505,1112,129,1201,260,1028,4079,575,155,680,284,2,578,728,2431,1362,283,4040,3390,8,1569,774,2115,132,4352,395,525,626,2,1965,528,2067,182,283,3136,669,1050,464,1344,386,743,268,81,7,28,463,620,29,994,385,474,3204,644,3394,896,313,876,412,2,554,33,2332,2,267,381,3,283,948,1220,38,363,259,298,573,145,28,127,499,718,1361,106,381,47,589,491,1284,1085,323,44,49,741,9,2,258,443,56,1196,940,217,2045,5,136,159,2,145,531,44,18,31,285,600,823,22,408,252,97,249,247,1116,42,368,540,177,8,22,478,593,166,850,630,448,16,87,100,484,279,6,38,331,135,2,358,61,6,99,132,302,18,438,136,2,83,340,357,274,587,359,273,5970803,2554,235,5997366,90,2800095,4,1572,549,332,445,1,2,80,1,900,583,9,304,1,8,1,2,2132,1,1,1,1,1,414,1,889,59,351,2,373,3,7,2,360,81,3,6,71,1,9,9,2,1,18,1,2,4,1,1,2,1,1,100,1,29,8,22,1,1,1,1,1,1,1,1,1,1,1,1,2,28,1,1,3,1,2,1,1,1,1,1,1,68,8,6,1,3,1,2,1,1,3,1,3,31,10,6,2,4,2,2,2,4,12,4,3',authuser:0,kscs:'c9c918f0_MK0kXLv5Ne-mrgTclpzYDg',kGL:'AE'};google.kHL='ar-AE';})();google.time=function(){return(new Date).getTime()};(function(){google.lc=[];google.li=0;google.getEI=function(a){for(var b;a&&(!a.getAttribute||!(b=a.getAttribute("eid")));)a=a.parentNode;return b||google.kEI};google.getLEI=function(a){for(var b=null;a&&(!a.getAttribute||!(b=a.getAttribute("leid")));)a=a.parentNode;return b};google.https=function(){return"https:"==window.location.protocol};google.ml=function(){return null};google.log=function(a,b,e,c,g){if(a=google.logUrl(a,b,e,c,g)){b=new Image;var d=google.lc,f=google.li;d[f]=b;b.onerror=b.onload=b.onabort=function(){delete d[f]};google.vel&&google.vel.lu&&google.vel.lu(a);b.src=a;google.li=f+1}};google.logUrl=function(a,b,e,c,g){var d="",f=google.ls||"";e||-1!=b.search("&ei=")||(d="&ei="+google.getEI(c),-1==b.search("&lei=")&&(c=google.getLEI(c))&&(d+="&lei="+c));c="";!e&&google.cshid&&-1==b.search("&cshid=")&&"slh"!=a&&(c="&cshid="+google.cshid);a=e||"/"+(g||"gen_204")+"?atyp=i&ct="+a+"&cad="+b+d+f+"&zx="+google.time()+c;/^http:/i.test(a)&&google.https()&&(google.ml(Error("a"),!1,{src:a,glmm:1}),a="");return a};}).call(this);(function(){google.y={};google.x=function(a,b){if(a)var c=a.id;else{do c=Math.random();while(google.y[c])}google.y[c]=[a,b];return!1};google.lm=[];google.plm=function(a){google.lm.push.apply(google.lm,a)};google.lq=[];google.load=function(a,b,c){google.lq.push([[a],b,c])};google.loadAll=function(a,b){google.lq.push([a,b])};}).call(this);google.f={};</script><script nonce="y8SVUeCln6pIIL+tT3bpDA==">var a=window.location,b=a.href.indexOf("#");if(0<=b){var c=a.href.substring(b+1);/(^|&)q=/.test(c)&&-1==c.indexOf("#")&&a.replace("/search?"+c.replace(/(^|&)fp=[^&]*/g,"")+"&cad=h")};</script><style>#gbar,#guser{font-size:13px;padding-top:1px !important;}#gbar{height:22px}#guser{padding-bottom:7px !important;text-align:left}.gbh,.gbd{border-top:1px solid #c9d7f1;font-size:1px}.gbh{height:0;position:absolute;top:24px;width:100%}@media all{.gb1{height:22px;margin-left:.5em;vertical-align:top}#gbar{float:right}}a.gb1,a.gb4{text-decoration:underline !important}a.gb1,a.gb4{color:#00c !important}.gbi .gb4{color:#dd8e27 !important}.gbf .gb4{color:#900 !important}
</style><style>body,td,a,p,.h{font-family:arial,sans-serif}body{margin:0;overflow-y:scroll}#gog{padding:3px 8px 0}td{line-height:.8em}.gac_m td{line-height:17px}form{margin-bottom:20px}.h{color:#36c}.q{color:#00c}.ts td{padding:0}.ts{border-collapse:collapse}em{font-weight:bold;font-style:normal}.lst{height:25px;width:496px}.gsfi,.lst{font:18px arial,sans-serif}.gsfs{font:17px arial,sans-serif}.ds{display:inline-box;display:inline-block;margin:3px 0 4px;margin-right:4px}input{font-family:inherit}a.gb1,a.gb2,a.gb3,a.gb4{color:#11c !important}body{background:#fff;color:black}a{color:#11c;text-decoration:none}a:hover,a:active{text-decoration:underline}.fl a{color:#36c}a:visited{color:#551a8b}a.gb1,a.gb4{text-decoration:underline}a.gb3:hover{text-decoration:none}#ghead a.gb2:hover{color:#fff !important}.sblc{padding-top:5px}.sblc a{display:block;margin:2px 0;margin-right:13px;font-size:11px}.lsbb{background:#eee;border:solid 1px;border-color:#ccc #ccc #999 #999;height:30px}.lsbb{display:block}.ftl,#fll a{display:inline-block;margin:0 12px}.lsb{background:url(/images/nav_logo229.png) 0 -261px repeat-x;border:none;color:#000;cursor:pointer;height:30px;margin:0;outline:0;font:15px arial,sans-serif;vertical-align:top}.lsb:active{background:#ccc}.lst:focus{outline:none}.tiah{width:458px}</style><script nonce="y8SVUeCln6pIIL+tT3bpDA=="></script></head><body bgcolor="#fff"><script nonce="y8SVUeCln6pIIL+tT3bpDA==">(function(){var src='/images/nav_logo229.png';var iesg=false;document.body.onload = function(){window.n && window.n();if (document.images){new Image().src=src;}
if (!iesg){document.f&&document.f.q.focus();document.gbqf&&document.gbqf.q.focus();}
}
})();</script><div id="mngb"> <div id=gbar><nobr><b class=gb1>&#1576;&#1581;&#1579;</b> <a class=gb1 href="https://www.google.ae/imghp?hl=ar&tab=wi">&#1589;&#1608;&#1585;</a> <a class=gb1 href="https://maps.google.ae/maps?hl=ar&tab=wl">&#1582;&#1585;&#1575;&#1574;&#1591;</a> <a class=gb1 href="https://www.youtube.com/?gl=AE&tab=w1">YouTube</a> <a class=gb1 href="https://news.google.ae/nwshp?hl=ar&ned=ar_me&tab=wn">&#1575;&#1604;&#1571;&#1582;&#1576;&#1575;&#1585;</a> <a class=gb1 href="https://mail.google.com/mail/?tab=wm">Gmail</a> <a class=gb1 href="https://drive.google.com/?tab=wo">Drive</a> <a class=gb1 href="https://www.google.com/calendar?tab=wc">&#1578;&#1602;&#1608;&#1610;&#1605;</a> <a class=gb1 style="text-decoration:none" href="https://www.google.ae/intl/ar/about/products?tab=wh"><u>&#1575;&#1604;&#1605;&#1586;&#1610;&#1583;</u> &raquo;</a></nobr></div><div id=guser width=100%><nobr><span id=gbn class=gbi></span><span id=gbf class=gbf></span><span id=gbe></span><a href="http://www.google.ae/history/optout?hl=ar" class=gb4>&#1587;&#1616;&#1580;&#1604; &#1576;&#1581;&#1579; &#1575;&#1604;&#1608;&#1610;&#1576;</a> | <a  href="/preferences?hl=ar" class=gb4>&#1575;&#1604;&#1573;&#1593;&#1583;&#1575;&#1583;&#1575;&#1578;</a> | <a target=_top id=gb_70 href="https://accounts.google.com/ServiceLogin?hl=ar&passive=true&continue=https://www.google.com/" class=gb4>&#1578;&#1587;&#1580;&#1610;&#1604; &#1575;&#1604;&#1583;&#1582;&#1608;&#1604;</a></nobr></div><div class=gbh style=left:0></div><div class=gbh style=right:0></div> </div><center><br clear="all" id="lgpd"><div id="lga"><img alt="Google" height="92" src="/images/branding/googlelogo/1x/googlelogo_white_background_color_272x92dp.png" style="padding:28px 0 14px" width="272" id="hplogo" onload="window.lol&&lol()"><br><br></div><form action="/search" name="f"><table cellpadding="0" cellspacing="0"><tr valign="top"><td width="25%">&nbsp;</td><td align="center" nowrap=""><input name="ie" value="ISO-8859-1" type="hidden"><input value="ar-AE" name="hl" type="hidden"><input name="source" type="hidden" value="hp"><input name="biw" type="hidden"><input name="bih" type="hidden"><div class="ds" style="height:32px;margin:4px 0"><div style="position:relative;zoom:1"><input style="color:#000;margin:0;padding:5px 6px 0 8px;vertical-align:top;padding-left:38px" autocomplete="off" class="lst tiah" value="" title="&#1576;&#1581;&#1579; Google" maxlength="2048" name="q" size="57"><img src="/textinputassistant/tia.png" style="position:absolute;cursor:pointer;left:5px;top:4px;z-index:300" onclick="(function(){var src='/textinputassistant/11/ar_tia.js';var s=document.createElement('script');s.src=src;(document.getElementById('xjsc')||document.body).appendChild(s);})();" alt="" height="23" width="27"></div></div><br style="line-height:0"><span class="ds"><span class="lsbb"><input class="lsb" value="&#1576;&#1581;&#1579; Google" name="btnG" type="submit"></span></span><span class="ds"><span class="lsbb"><input class="lsb" value="&#1590;&#1585;&#1576;&#1577; &#1581;&#1592;" name="btnI" onclick="if(this.form.q.value)this.checked=1; else top.location='/doodles/'" type="submit"></span></span></td><td class="fl sblc" align="right" nowrap="" width="25%"><a href="/advanced_search?hl=ar-AE&amp;authuser=0">&#1576;&#1581;&#1579; &#1605;&#1578;&#1602;&#1583;&#1605;</a><a href="/language_tools?hl=ar-AE&amp;authuser=0">&#1571;&#1583;&#1608;&#1575;&#1578; &#1575;&#1604;&#1604;&#1594;&#1577;</a></td></tr></table><input id="gbv" name="gbv" type="hidden" value="1"><script nonce="y8SVUeCln6pIIL+tT3bpDA==">(function(){var a,b="1";if(document&&document.getElementById)if("undefined"!=typeof XMLHttpRequest)b="2";else if("undefined"!=typeof ActiveXObject){var c,d,e=["MSXML2.XMLHTTP.6.0","MSXML2.XMLHTTP.3.0","MSXML2.XMLHTTP","Microsoft.XMLHTTP"];for(c=0;d=e[c++];)try{new ActiveXObject(d),b="2"}catch(h){}}a=b;if("2"==a&&-1==location.search.indexOf("&gbv=2")){var f=google.gbvu,g=document.getElementById("gbv");g&&(g.value=a);f&&window.setTimeout(function(){location.href=f},0)};}).call(this);</script></form><div id="gac_scont"></div><div style="font-size:83%;min-height:3.5em"><br><div id="gws-output-pages-elements-homepage_additional_languages__als"><style>#gws-output-pages-elements-homepage_additional_languages__als{font-size:small;margin-bottom:24px}#SIvCob{display:inline-block;line-height:28px;}#SIvCob a{padding:0 3px;}.H6sW5{display:inline-block;margin:0 2px;white-space:nowrap}.z4hgWe{display:inline-block;margin:0 2px}</style><div id="SIvCob">&#1605;&#1581;&#1585;&#1617;&#1603; &#1576;&#1581;&#1579; Google &#1605;&#1578;&#1608;&#1601;&#1617;&#1585; &#1576;&#1575;&#1604;&#1604;&#1594;&#1577;:  <a href="https://www.google.com/setprefs?sig=0_Op0m1OssGjK0jPuLZxgPPuzH9Ek%3D&amp;hl=fa&amp;source=homepage&amp;sa=X&amp;ved=0ahUKEwi70byd6r_fAhVvk4sKHVwLB-sQ2ZgBCAU">&#1601;&#1575;&#1585;&#1587;&#1740;</a>    <a dir="ltr" href="https://www.google.com/setprefs?sig=0_Op0m1OssGjK0jPuLZxgPPuzH9Ek%3D&amp;hl=en&amp;source=homepage&amp;sa=X&amp;ved=0ahUKEwi70byd6r_fAhVvk4sKHVwLB-sQ2ZgBCAY">English</a>    <a dir="ltr" href="https://www.google.com/setprefs?sig=0_Op0m1OssGjK0jPuLZxgPPuzH9Ek%3D&amp;hl=hi&amp;source=homepage&amp;sa=X&amp;ved=0ahUKEwi70byd6r_fAhVvk4sKHVwLB-sQ2ZgBCAc">&#2361;&#2367;&#2344;&#2381;&#2342;&#2368;</a>    <a href="https://www.google.com/setprefs?sig=0_Op0m1OssGjK0jPuLZxgPPuzH9Ek%3D&amp;hl=ur&amp;source=homepage&amp;sa=X&amp;ved=0ahUKEwi70byd6r_fAhVvk4sKHVwLB-sQ2ZgBCAg">&#1575;&#1585;&#1583;&#1608;</a>  </div></div></div><span id="footer"><div style="font-size:10pt"><div style="margin:19px auto;text-align:center" id="fll"><a href="/intl/ar/ads/">&#1575;&#1604;&#1576;&#1585;&#1606;&#1575;&#1605;&#1580; &#1575;&#1604;&#1573;&#1593;&#1604;&#1575;&#1606;&#1610;</a><a href="http://www.google.com/intl/ar/services/">&#1581;&#1604;&#1608;&#1604; &#1575;&#1604;&#1588;&#1585;&#1603;&#1575;&#1578;</a><a href="https://plus.google.com/101532581614261957891" rel="publisher">+Google</a><a href="/intl/ar/about.html">&#1603;&#1604; &#1605;&#1575; &#1578;&#1581;&#1576; &#1605;&#1593;&#1585;&#1601;&#1578;&#1607; &#1593;&#1606; Google &#1607;&#1606;&#1575;</a><a dir="ltr" href="https://www.google.com/setprefdomain?prefdom=AE&amp;prev=https://www.google.ae/&amp;sig=K_LVsvsOYgNgpu3bZkCt1rVK7Ltrs%3D">Google.ae</a></div></div><p style="color:#767676;font-size:8pt">&copy; 2018 - <a href="/intl/ar/policies/privacy/">&#1575;&#1604;&#1582;&#1589;&#1608;&#1589;&#1610;&#1577;</a> - <a href="/intl/ar/policies/terms/">&#1575;&#1604;&#1576;&#1606;&#1608;&#1583;</a></p></span></center><script nonce="y8SVUeCln6pIIL+tT3bpDA==">(function(){window.google.cdo={height:0,width:0};(function(){var a=window.innerWidth,b=window.innerHeight;if(!a||!b){var c=window.document,d="CSS1Compat"==c.compatMode?c.documentElement:c.body;a=d.clientWidth;b=d.clientHeight}a&&b&&(a!=google.cdo.width||b!=google.cdo.height)&&google.log("","","/client_204?&atyp=i&biw="+a+"&bih="+b+"&ei="+google.kEI);}).call(this);})();(function(){var u='/xjs/_/js/k\x3dxjs.hp.en.-PpIeOYp7rw.O/m\x3dsb_he,d/am\x3dYsAs/rt\x3dj/d\x3d1/rs\x3dACT90oHak6Vea8_d5yZnUPgd2x9xqhMspg';var b={gen204:"xjsls",clearcut:31};setTimeout(function(){var a=document.createElement("script");a.src=u;google.timers&&google.timers.load&&google.tick&&google.tick("load",b);document.body.appendChild(a)},0);})();(function(){window.google.xjsu='/xjs/_/js/k\x3dxjs.hp.en.-PpIeOYp7rw.O/m\x3dsb_he,d/am\x3dYsAs/rt\x3dj/d\x3d1/rs\x3dACT90oHak6Vea8_d5yZnUPgd2x9xqhMspg';})();function _DumpException(e){throw e;}
(function(){var pmc='{\x22Qnk92g\x22:{},\x22U5B21g\x22:{},\x22YFCs/g\x22:{},\x22ZI/YVQ\x22:{},\x22d\x22:{},\x22sb_he\x22:{\x22agen\x22:true,\x22cgen\x22:true,\x22client\x22:\x22heirloom-hp\x22,\x22dh\x22:true,\x22dhqt\x22:true,\x22ds\x22:\x22\x22,\x22ffql\x22:\x22en\x22,\x22fl\x22:true,\x22host\x22:\x22google.com\x22,\x22isbh\x22:28,\x22jsonp\x22:true,\x22msgs\x22:{\x22cibl\x22:\x22&#1605;&#1581;&#1608; &#1575;&#1604;&#1576;&#1581;&#1579;\x22,\x22dym\x22:\x22&#1607;&#1604; &#1578;&#1602;&#1589;&#1583; :\x22,\x22lcky\x22:\x22&#1590;&#1585;&#1576;&#1577; &#1581;&#1592;\x22,\x22lml\x22:\x22&#1605;&#1586;&#1610;&#1583; &#1605;&#1606; &#1575;&#1604;&#1605;&#1593;&#1604;&#1608;&#1605;&#1575;&#1578;\x22,\x22oskt\x22:\x22&#1571;&#1583;&#1608;&#1575;&#1578; &#1575;&#1604;&#1573;&#1583;&#1582;&#1575;&#1604;\x22,\x22psrc\x22:\x22&#1578;&#1605;&#1578; &#1573;&#1586;&#1575;&#1604;&#1577; &#1607;&#1584;&#1575; &#1575;&#1604;&#1576;&#1581;&#1579; &#1605;&#1606; \\u003Ca href\x3d\\\x22/history\\\x22\\u003E&#1587;&#1616;&#1580;&#1604; &#1575;&#1604;&#1576;&#1581;&#1579;\\u003C/a\\u003E.\x22,\x22psrl\x22:\x22&#1573;&#1586;&#1575;&#1604;&#1577;\x22,\x22sbit\x22:\x22&#1575;&#1604;&#1576;&#1581;&#1579; &#1576;&#1581;&#1587;&#1576; &#1575;&#1604;&#1589;&#1608;&#1585;\x22,\x22srch\x22:\x22&#1576;&#1581;&#1579; Google\u200f\x22},\x22ovr\x22:{},\x22pq\x22:\x22\x22,\x22refpd\x22:true,\x22rfs\x22:[],\x22sbpl\x22:24,\x22sbpr\x22:24,\x22scd\x22:10,\x22sce\x22:5,\x22stok\x22:\x22w5imgy_Y5x5UzxatpHEl2smmvPE\x22,\x22uhde\x22:false}}';google.pmc=JSON.parse(pmc);})();(function(){var r=['aa','async','ipv6','mu','sf'];google.plm(r);})();</script>     </body></html>
2018-12-27 13:45:05 [scrapy.core.engine] INFO: Closing spider (finished)
2018-12-27 13:45:05 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 213,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 6323,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2018, 12, 27, 10, 45, 5, 11903),
 'log_count/DEBUG': 2,
 'log_count/INFO': 7,
 'memusage/max': 51888128,
 'memusage/startup': 51888128,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2018, 12, 27, 10, 45, 4, 773777)}
2018-12-27 13:45:05 [scrapy.core.engine] INFO: Spider closed (finished)
Jumping into debugger for post-mortem of exception '(-1, 'Unexpected EOF')':
> /root/scraper/lib/python3.6/site-packages/OpenSSL/SSL.py(1632)_raise_ssl_error()
-> raise SysCallError(-1, "Unexpected EOF")
(Pdb) where
  /root/scraper/lib/python3.6/site-packages/twisted/protocols/tls.py(274)_flushReceiveBIO()
-> bytes = self._tlsConnection.recv(2 ** 15)
  /root/scraper/lib/python3.6/site-packages/OpenSSL/SSL.py(1783)recv()
-> self._raise_ssl_error(self._ssl, result)
> /root/scraper/lib/python3.6/site-packages/OpenSSL/SSL.py(1632)_raise_ssl_error()
-> raise SysCallError(-1, "Unexpected EOF")
(Pdb) exit

Here is some info about my environment:

scrapy version -v

Scrapy       : 1.5.1
lxml         : 4.2.5.0
libxml2      : 2.9.8
cssselect    : 1.0.3
parsel       : 1.5.1
w3lib        : 1.19.0
Twisted      : 18.9.0
Python       : 3.6.7 (default, Oct 22 2018, 11:32:17) - [GCC 8.2.0]
pyOpenSSL    : 18.0.0 (OpenSSL 1.1.0j  20 Nov 2018)
cryptography : 2.4.2
Platform     : Linux-4.15.0-36-generic-x86_64-with-Ubuntu-18.04-bionic

pip freeze

asn1crypto==0.24.0
attrs==18.2.0
Automat==0.7.0
cffi==1.11.5
constantly==15.1.0
cryptography==2.4.2
cssselect==1.0.3
hyperlink==18.0.0
idna==2.8
incremental==17.5.0
lxml==4.2.5
parsel==1.5.1
pkg-resources==0.0.0
pyasn1==0.4.4
pyasn1-modules==0.2.2
pycparser==2.19
PyDispatcher==2.0.5
PyHamcrest==1.9.0
pyOpenSSL==18.0.0
queuelib==1.5.0
Scrapy==1.5.1
service-identity==18.1.0
six==1.12.0
Twisted==18.9.0
w3lib==1.19.0

No error is reported when I run the scrapy fetch command without the --pdb option. Also, fetching that page works fine with requests 2.21.0 and curl.

The two patches

Our patch+355 −8
scrapy/cmdline.pyindex e6d5ff96a..a2e62f3c5 100644@@ -5,6 +5,7 @@ import cProfile import inspect import os import sys+from contextlib import nullcontext from importlib.metadata import entry_points from typing import TYPE_CHECKING, ParamSpec @@ -12,6 +13,7 @@ import scrapy from scrapy.commands import BaseRunSpiderCommand, ScrapyCommand, ScrapyHelpFormatter from scrapy.crawler import AsyncCrawlerProcess, CrawlerProcess from scrapy.exceptions import UsageError+from scrapy.utils._post_mortem import post_mortem_on_error from scrapy.utils.misc import walk_modules_iter from scrapy.utils.project import get_project_settings, inside_project from scrapy.utils.python import garbage_collect@@ -216,10 +218,11 @@ def execute(argv: list[str] | None = None, settings: Settings | None = None) ->   def _run_command(cmd: ScrapyCommand, args: list[str], opts: argparse.Namespace) -> None:-    if opts.profile:-        _run_command_profiled(cmd, args, opts)-    else:-        cmd.run(args, opts)+    with post_mortem_on_error() if opts.pdb else nullcontext():+        if opts.profile:+            _run_command_profiled(cmd, args, opts)+        else:+            cmd.run(args, opts)   def _run_command_profiled(scrapy/commands/__init__.pyindex 9d0f405d2..af8d1dfbb 100644@@ -14,8 +14,6 @@ from abc import ABC, abstractmethod from pathlib import Path from typing import TYPE_CHECKING, Any, ClassVar -from twisted.python import failure- from scrapy import signals from scrapy.exceptions import ScrapyDeprecationWarning, UsageError from scrapy.extensions.feedexport import FeedExporter@@ -162,7 +160,6 @@ class ScrapyCommand(ABC):                 pass             else:                 sys.modules["pdb"] = ipdb-            failure.startDebugMode()      @abstractmethod     def run(self, args: list[str], opts: argparse.Namespace) -> None:scrapy/utils/_post_mortem.pynew file mode 100644index 000000000..53d2bf12c@@ -0,0 +1,120 @@+"""Post-mortem debugging support for the ``--pdb`` command-line option.++Twisted's own debug mode (:func:`twisted.python.failure.startDebugMode`)+enters the debugger whenever a :class:`~twisted.python.failure.Failure` is+created from an exception, including exceptions that are later handled+(e.g. ``OpenSSL.SSL.SysCallError`` raised by Twisted's TLS code when a server+closes the connection). Instead, the debugger is only entered for errors that+are actually reported: errors logged with a traceback at ``ERROR`` level or+higher, errors logged by Twisted (e.g. unhandled errors in Deferreds) and+exceptions that propagate out of the command.+"""++from __future__ import annotations++import gc+import logging+import sys+from contextlib import contextmanager+from types import TracebackType+from typing import TYPE_CHECKING, Any++from twisted.python import log as twisted_log+from twisted.python.failure import Failure++from scrapy.exceptions import UsageError++if TYPE_CHECKING:+    from collections.abc import Callable, Iterator+++class _PostMortemDebugger:+    def __init__(self) -> None:+        # Exceptions already debugged, so that an exception reported more than+        # once (e.g. by different components) only enters the debugger once.+        self._debugged: list[BaseException] = []++    def debug(self, exc_value: BaseException | None, tb: Any) -> None:+        # Fake tracebacks (e.g. from cleaned Failures) cannot be debugged.+        if exc_value is None or not isinstance(tb, TracebackType):+            return+        if any(debugged is exc_value for debugged in self._debugged):+            return+        self._debugged.append(exc_value)+        try:+            strrepr = str(exc_value)+        except Exception:+            strrepr = "broken str"+        print(f"Jumping into debugger for post-mortem of exception {strrepr!r}:")+        # Imported here so that ipdb is used if it was installed as "pdb".+        import pdb  # noqa: T100,PLC0415++        pdb.post_mortem(tb)++    def debug_failure(self, failure: Failure) -> None:+        self.debug(failure.value, failure.tb)+++class _PostMortemLogHandler(logging.Handler):+    def __init__(self, debugger: _PostMortemDebugger):+        super().__init__(logging.ERROR)+        self._debugger = debugger++    def emit(self, record: logging.LogRecord) -> None:+        if record.exc_info:+            self._debugger.debug(record.exc_info[1], record.exc_info[2])+++def _keep_traceback(+    clean_failure: Callable[[Failure], None],+) -> Callable[[Failure], None]:+    """Wrap :meth:`Failure.cleanFailure` so that it keeps the traceback.++    Older Twisted versions clean the failures that reach the end of a+    Deferred callback chain, dropping their traceback before they are+    logged."""++    def cleanFailure(self: Failure) -> None:+        tb = self.tb+        value = self.value+        value_tb = getattr(value, "__traceback__", None)+        clean_failure(self)+        self.tb = tb+        if value is not None and value_tb is not None:+            value.__traceback__ = value_tb++    return cleanFailure+++@contextmanager+def post_mortem_on_error() -> Iterator[None]:+    """Enter the post-mortem debugger for errors reported within the+    context."""+    debugger = _PostMortemDebugger()++    def twisted_observer(event: dict[str, Any]) -> None:+        failure = event.get("failure")+        if event.get("isError") and isinstance(failure, Failure):+            debugger.debug_failure(failure)++    handler = _PostMortemLogHandler(debugger)+    original_clean_failure = Failure.cleanFailure+    Failure.cleanFailure = _keep_traceback(original_clean_failure)  # type: ignore[method-assign]+    logging.root.addHandler(handler)+    twisted_log.addObserver(twisted_observer)+    try:+        try:+            yield+        finally:+            # Report unhandled errors of Deferreds that are not referenced+            # anymore while the debugger is still enabled.+            gc.collect()+    except UsageError:+        raise+    except Exception:+        debugger.debug(*sys.exc_info()[1:])+        raise+    finally:+        twisted_log.removeObserver(twisted_observer)+        logging.root.removeHandler(handler)+        Failure.cleanFailure = original_clean_failure  # type: ignore[method-assign]tests/test_commands.pyindex 08882fd52..18985f179 100644@@ -100,7 +100,10 @@ class TestCommandSettings:         assert dict(self.command.settings["FEEDS"]) == json.loads(feeds_json)      def test_pdb_uses_ipdb_if_installed(self, monkeypatch):-        monkeypatch.setattr(failure, "startDebugMode", lambda: None)+        def start_debug_mode() -> None:+            raise AssertionError("Twisted debug mode must not be enabled")++        monkeypatch.setattr(failure, "startDebugMode", start_debug_mode)         fake_ipdb: Any = argparse.Namespace(post_mortem=lambda tb: None)         monkeypatch.setitem(sys.modules, "pdb", pdb)         monkeypatch.setitem(sys.modules, "ipdb", fake_ipdb)@@ -188,6 +191,94 @@ class MySpider(scrapy.Spider):         assert returncode == 0, err         assert "It works!" in err +    def test_pdb_handled_exception(self, tmp_path: Path) -> None:+        """Exceptions that are caught and handled, e.g. the+        OpenSSL.SSL.SysCallError that Twisted handles on TLS connection loss,+        do not trigger the debugger."""+        spider_path = tmp_path / "myspider.py"+        spider_path.write_text(+            """+import scrapy+from twisted.internet.defer import Deferred+from twisted.python.failure import Failure++def fail(_):+    raise ValueError("harmless")++class MySpider(scrapy.Spider):+    name = "myspider"++    async def start(self):+        try:+            raise ValueError("harmless")+        except ValueError:+            Failure()+        d = Deferred()+        d.addCallback(fail)+        d.addErrback(lambda failure: None)+        d.callback(None)+        self.logger.debug("It works!")+        return+        yield+""",+            encoding="utf-8",+        )+        returncode, out, err = proc("runspider", str(spider_path), "--pdb", input="")+        assert returncode == 0, err+        assert "It works!" in err+        assert "Jumping into debugger" not in out++    @pytest.mark.parametrize(+        "error_code",+        [+            # An exception logged as an error by Scrapy.+            """+        yield scrapy.Request("data:,", callback=self.parse_error)++    def parse_error(self, response):+        raise ValueError("expected error")+""",+            # An unhandled error in a Deferred, logged by Twisted.+            """+        d = Deferred()+        d.addCallback(fail)+        d.callback(None)+        del d+        return+        yield+""",+        ],+        ids=["logged_error", "unhandled_deferred_error"],+    )+    def test_pdb_reported_exception(self, tmp_path: Path, error_code: str) -> None:+        spider_path = tmp_path / "myspider.py"+        spider_path.write_text(+            """+import scrapy+from twisted.internet.defer import Deferred++def fail(_):+    raise ValueError("expected error")++class MySpider(scrapy.Spider):+    name = "myspider"++    async def start(self):"""+            + error_code,+            encoding="utf-8",+        )+        returncode, out, err = proc(+            "runspider", str(spider_path), "--pdb", input="continue\n"+        )+        assert returncode == 0, err+        assert (+            out.count(+                "Jumping into debugger for post-mortem of exception 'expected error'"+            )+            == 1+        ), out+        assert "Spider closed (finished)" in err+  class TestSettingsCommand:     @pytest.mark.parametrize(tests/test_utils_post_mortem.pynew file mode 100644index 000000000..64eec0350@@ -0,0 +1,136 @@+from __future__ import annotations++import argparse+import logging+import sys+from types import TracebackType+from typing import Any++import pytest+from twisted.internet.defer import Deferred+from twisted.python import log as twisted_log+from twisted.python.failure import Failure++from scrapy.exceptions import UsageError+from scrapy.utils._post_mortem import post_mortem_on_error+from scrapy.utils.log import failure_to_exc_info++logger = logging.getLogger(__name__)+++@pytest.fixture+def post_mortems(monkeypatch: pytest.MonkeyPatch) -> list[TracebackType]:+    """Replace pdb with a fake one, return the tracebacks it is called with."""+    calls: list[TracebackType] = []+    fake_pdb: Any = argparse.Namespace(post_mortem=calls.append)+    monkeypatch.setitem(sys.modules, "pdb", fake_pdb)+    return calls+++def _failing_callback(_: Any) -> None:+    raise ValueError("boom")+++def test_handled_failure_does_not_debug(post_mortems: list[TracebackType]) -> None:+    with post_mortem_on_error():+        try:+            raise ValueError("harmless")+        except ValueError:+            Failure()+        d: Deferred[None] = Deferred()+        d.addCallback(_failing_callback)+        d.addErrback(lambda f: None)+        d.callback(None)+    assert post_mortems == []+++def test_logged_error_debugs(post_mortems: list[TracebackType]) -> None:+    with post_mortem_on_error():+        try:+            raise ValueError("boom")+        except ValueError as e:+            exc = e+            logger.error("Error", exc_info=True)+            # The same exception reported again is not debugged again.+            logger.error("Error", exc_info=True)+    assert post_mortems == [exc.__traceback__]+++def test_logged_warning_does_not_debug(post_mortems: list[TracebackType]) -> None:+    with post_mortem_on_error():+        try:+            raise ValueError("boom")+        except ValueError:+            logger.warning("Warning", exc_info=True)+    assert post_mortems == []+++def test_logged_failure_debugs(post_mortems: list[TracebackType]) -> None:+    with post_mortem_on_error():+        d: Deferred[None] = Deferred()+        d.addCallback(_failing_callback)+        failures: list[Failure] = []+        d.addErrback(failures.append)+        d.callback(None)+        logger.error("Error", exc_info=failure_to_exc_info(failures[0]))+    assert len(post_mortems) == 1+    assert isinstance(post_mortems[0], TracebackType)+++def test_cleaned_failure_debugs(post_mortems: list[TracebackType]) -> None:+    """Older Twisted versions clean failures at the end of Deferred callback+    chains, the traceback must survive that."""+    with post_mortem_on_error():+        try:+            raise ValueError("boom")+        except ValueError as e:+            tb = e.__traceback__+            f = Failure()+        f.cleanFailure()+        logger.error("Error", exc_info=failure_to_exc_info(f))+    assert post_mortems == [tb]+++def test_clean_failure_restored(post_mortems: list[TracebackType]) -> None:+    original = Failure.cleanFailure+    with post_mortem_on_error():+        assert Failure.cleanFailure is not original+    assert Failure.cleanFailure is original+    try:+        raise ValueError("boom")+    except ValueError:+        logger.error("Error", exc_info=True)+    assert post_mortems == []+++def test_twisted_logged_failure_debugs(post_mortems: list[TracebackType]) -> None:+    with post_mortem_on_error():+        try:+            raise ValueError("boom")+        except ValueError as e:+            tb = e.__traceback__+            twisted_log.err(Failure(), "Error")+    assert post_mortems == [tb]+++def test_unhandled_deferred_error_debugs(post_mortems: list[TracebackType]) -> None:+    with post_mortem_on_error():+        d: Deferred[None] = Deferred()+        d.addCallback(_failing_callback)+        d.callback(None)+        del d+    assert len(post_mortems) == 1+    assert isinstance(post_mortems[0], TracebackType)+++def test_propagated_exception_debugs(post_mortems: list[TracebackType]) -> None:+    with pytest.raises(ValueError, match="boom"), post_mortem_on_error():+        raise ValueError("boom")+    assert len(post_mortems) == 1+    assert isinstance(post_mortems[0], TracebackType)+++def test_usage_error_does_not_debug(post_mortems: list[TracebackType]) -> None:+    with pytest.raises(UsageError), post_mortem_on_error():+        raise UsageError("bad usage")+    assert post_mortems == [] 
Merged fix · PR #8037+86 −8
scrapy/commands/__init__.pyindex 9d0f405d2..e8e446ba4 100644@@ -14,8 +14,6 @@ from abc import ABC, abstractmethod from pathlib import Path from typing import TYPE_CHECKING, Any, ClassVar -from twisted.python import failure- from scrapy import signals from scrapy.exceptions import ScrapyDeprecationWarning, UsageError from scrapy.extensions.feedexport import FeedExporter@@ -34,6 +32,17 @@ if TYPE_CHECKING: logger = logging.getLogger(__name__)  +class _PdbHandler(logging.Handler):+    """Start a post-mortem debugging session on every logged record that comes+    with an exception traceback."""++    def emit(self, record: logging.LogRecord) -> None:+        if record.exc_info and record.exc_info[2]:+            import pdb  # noqa: T100,PLC0415++            pdb.post_mortem(record.exc_info[2])++ class ScrapyCommand(ABC):     """Base class for all Scrapy commands.""" @@ -162,7 +171,7 @@ class ScrapyCommand(ABC):                 pass             else:                 sys.modules["pdb"] = ipdb-            failure.startDebugMode()+            logging.root.addHandler(_PdbHandler(level=logging.ERROR))      @abstractmethod     def run(self, args: list[str], opts: argparse.Namespace) -> None:tests/test_commands.pyindex 08882fd52..8b3e8b7c6 100644@@ -2,13 +2,13 @@ from __future__ import annotations  import argparse import json+import logging import pdb  # noqa: T100 import sys from pathlib import Path from typing import TYPE_CHECKING, Any  import pytest-from twisted.python import failure  import scrapy from scrapy.cmdline import _pop_command_name, execute@@ -100,13 +100,16 @@ class TestCommandSettings:         assert dict(self.command.settings["FEEDS"]) == json.loads(feeds_json)      def test_pdb_uses_ipdb_if_installed(self, monkeypatch):-        monkeypatch.setattr(failure, "startDebugMode", lambda: None)         fake_ipdb: Any = argparse.Namespace(post_mortem=lambda tb: None)         monkeypatch.setitem(sys.modules, "pdb", pdb)         monkeypatch.setitem(sys.modules, "ipdb", fake_ipdb)         opts, args = self.parser.parse_known_args(args=["--pdb", "spider.py"])-        self.command.process_options(args, opts)-        assert sys.modules["pdb"] is fake_ipdb+        handlers = logging.root.handlers[:]+        try:+            self.command.process_options(args, opts)+            assert sys.modules["pdb"] is fake_ipdb+        finally:+            logging.root.handlers[:] = handlers      def test_help_formatter(self):         formatter = ScrapyHelpFormatter(prog="scrapy")@@ -184,9 +187,75 @@ class MySpider(scrapy.Spider):         assert pidfile.read_text(encoding="utf-8").strip().isdigit()      def test_pdb(self, spider_path: Path) -> None:-        returncode, _, err = proc("runspider", str(spider_path), "--pdb")+        returncode, out, err = proc("runspider", str(spider_path), "--pdb", input="")         assert returncode == 0, err         assert "It works!" in err+        assert "(Pdb)" not in out++    def test_pdb_on_logged_error(self, tmp_path: Path) -> None:+        path = tmp_path / "myspider.py"+        path.write_text(+            """+import scrapy++class MySpider(scrapy.Spider):+    name = "myspider"++    async def start(self):+        raise ValueError("boom")+        yield+""",+            encoding="utf-8",+        )+        _, out, err = proc("runspider", str(path), "--pdb", input="")+        assert "(Pdb)" in out, err++    def test_pdb_on_logged_error_without_exception(self, tmp_path: Path) -> None:+        path = tmp_path / "myspider.py"+        path.write_text(+            """+import logging+import scrapy++logger = logging.getLogger(__name__)++class MySpider(scrapy.Spider):+    name = "myspider"++    async def start(self):+        logger.error("boom")+        return+        yield+""",+            encoding="utf-8",+        )+        returncode, out, err = proc("runspider", str(path), "--pdb", input="")+        assert returncode == 0, err+        assert "(Pdb)" not in out++    def test_pdb_on_handled_failure(self, tmp_path: Path) -> None:+        path = tmp_path / "myspider.py"+        path.write_text(+            """+import scrapy+from scrapy.exceptions import IgnoreRequest++class IgnoreEverything:+    def process_request(self, request, spider):+        raise IgnoreRequest++class MySpider(scrapy.Spider):+    name = "myspider"+    custom_settings = {"DOWNLOADER_MIDDLEWARES": {IgnoreEverything: 500}}++    async def start(self):+        yield scrapy.Request("https://example.com")+""",+            encoding="utf-8",+        )+        returncode, out, err = proc("runspider", str(path), "--pdb", input="")+        assert returncode == 0, err+        assert "(Pdb)" not in out   class TestSettingsCommand: