Backtestsscrapyrun-20261025-211112#6550

Issue #6550 Merged fix PR #7879 Run 25 Oct 2026 · run-20261025-211112 scrapy/scrapy

Improve open_in_browser function's handling of base tags in response.py

Better

Our patch was judged better than the merged fix.

In this judgment, Patch A = our agent, Patch B = the merged fix.

Both patches make every independent test pass, so the decision rests on robustness, scope, and tests. Patch A inserts the base tag after the real opening <head> if present, else after <html>, else after a doctype, else at the start (preserving a UTF-8 BOM), with HTML comments masked so commented-out <head>/<base> tags are ignored and offsets stay exact. It keeps the existing 'do not inject when a base tag already exists' behaviour, matches <base> case-insensitively without false-matching <basefont>, HTML-escapes the URL in the attribute, and skips injection for encodings like UTF-16 where ASCII bytes would corrupt the body. Its tests cover each fallback position, uppercase tags, newline-separated attributes, commented heads, basefont, escaping, BOM and UTF-16. Patch B takes a simpler route: always prepend a base tag (after a doctype only if the doctype is at the very start) and rely on the browser to hoist it into <head>. That works in browsers, but it changes semantics (a second base tag is always injected even when the response has one, with the existing test deleted and replaced), does not escape the URL, and only recognises a doctype at byte 0 (modulo whitespace), so a leading comment before the doctype would put <base> ahead of it and push the browser into quirks mode, which its own code comment says it wants to avoid. It also bumps the minimum w3lib requirement in pyproject.toml and tox.ini, an unexplained dependency change unrelated to the reported issue that adds risk for users, and it deletes the _remove_html_comments helper and its unit tests. Patch B's updated tests are reasonable but less thorough than A's. A resolves the issue more faithfully, with stronger tests and no unrelated churn.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

5m 24s

Wall-clock time in the container

Full test suite

No new failures

165 passed · 2 skipped · 169 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

158vs 112 merged

Changed lines, ours vs merged

The open_in_browser function in response.py uses a workaround to insert a <base> tag into the HTML response for local file viewing. As mentioned in the function's docstring, is "dirty" and could fail in situations where <head> tags are missing or improperly formed which can lead to unintended effects when rendering HTML.

Steps to Reproduce:

  1. Use open_in_browser with an HTML response that does not contain a proper <head> tag.
  2. Observe to see if the function does not handle this smoothly.

Suggested Fix:
Edit the function to handle missing or properly formed <head> tags so that a correct <base> tag can be inserted.

The two patches

Our patch+154 −4
scrapy/utils/response.pyindex b3622c159..4d040270e 100644@@ -5,10 +5,12 @@ scrapy.http.Response objects  from __future__ import annotations +import codecs import os import re import tempfile import webbrowser+from html import escape from typing import TYPE_CHECKING, Any from weakref import WeakKeyDictionary @@ -71,6 +73,54 @@ def _remove_html_comments(body: bytes) -> bytes:     return body  +def _mask_html_comments(body: bytes) -> bytes:+    """Return *body* with every HTML comment (including an unterminated+    trailing one) replaced by spaces of the same length, so that offsets in+    the result match offsets in *body*."""+    parts: list[bytes] = []+    pos = 0+    start = body.find(b"<!--")+    while start != -1:+        end = body.find(b"-->", start + 1)+        end = len(body) if end == -1 else end + 3+        parts.append(body[pos:start])+        parts.append(b" " * (end - start))+        pos = end+        start = body.find(b"<!--", pos)+    parts.append(body[pos:])+    return b"".join(parts)+++_BASE_TAG_RE = re.compile(rb"<base(?=[\s/>])", re.IGNORECASE)+_HEAD_TAG_RE = re.compile(rb"<head(?=[\s/>])[^<>]*>", re.IGNORECASE)+_HTML_TAG_RE = re.compile(rb"<html(?=[\s/>])[^<>]*>", re.IGNORECASE)+_DOCTYPE_RE = re.compile(rb"<!doctype[^<>]*>", re.IGNORECASE)+++def _inject_base_tag(body: bytes, url: str) -> bytes:+    """Return *body* with a ``<base>`` tag pointing to *url* inserted, unless+    *body* already declares one.++    The tag is inserted right after the opening ``<head>`` tag if there is+    one, otherwise right after the opening ``<html>`` tag, otherwise right+    after the doctype declaration, otherwise at the start of *body*. Tags+    inside HTML comments are ignored.+    """+    masked = _mask_html_comments(body)+    if _BASE_TAG_RE.search(masked):+        return body+    for pattern in (_HEAD_TAG_RE, _HTML_TAG_RE, _DOCTYPE_RE):+        match = pattern.search(masked)+        if match:+            pos = match.end()+            break+    else:+        # keep a leading UTF-8 byte order mark at the start of the body+        pos = len(codecs.BOM_UTF8) if body.startswith(codecs.BOM_UTF8) else 0+    base_tag = to_bytes(f'<base href="{escape(url, quote=True)}">')+    return body[:pos] + base_tag + body[pos:]++ def open_in_browser(     response: TextResponse,     _openfunc: Callable[[str], Any] = webbrowser.open,@@ -98,12 +148,11 @@ def open_in_browser(     # circular imports     from scrapy.http import HtmlResponse, TextResponse  # noqa: PLC0415 -    # XXX: this implementation is a bit dirty and could be improved     body = response.body     if isinstance(response, HtmlResponse):-        if b"<base" not in _remove_html_comments(body):-            repl = rf'\g<0><base href="{response.url}">'-            body = re.sub(rb"<head(?:[^<>]*?>)", to_bytes(repl), body, count=1)+        # an ASCII tag cannot be inserted into e.g. UTF-16 bodies+        if "<base>".encode(response.encoding) == b"<base>":+            body = _inject_base_tag(body, response.url)         ext = ".html"     elif isinstance(response, TextResponse):         ext = ".txt"tests/test_utils_response.pyindex 3638c52b7..04f69f276 100644@@ -322,3 +322,104 @@ def test_open_in_browser_raises_for_unsupported_response_type():     response = Response("http://www.example.com", body=b"binary")     with pytest.raises(TypeError):         open_in_browser(response, _openfunc=lambda _: True)  # type: ignore[arg-type]+++def _open_and_read(body: bytes, url: str = "http://www.example.com") -> bytes:+    result: list[bytes] = []++    def check(burl: str) -> bool:+        result.append(_read_browser_output(burl))+        return True++    assert open_in_browser(HtmlResponse(url, body=body), _openfunc=check)+    return result[0]+++@pytest.mark.parametrize(+    ("body", "expected"),+    [+        pytest.param(+            b"<html><body>no head</body></html>",+            b'<html><base href="http://www.example.com"><body>',+            id="missing head",+        ),+        pytest.param(+            b'<html lang="en"><body>no head</body></html>',+            b'<html lang="en"><base href="http://www.example.com"><body>',+            id="missing head, html with attributes",+        ),+        pytest.param(+            b"<!DOCTYPE html><title>T</title><p>hi</p>",+            b'<!DOCTYPE html><base href="http://www.example.com"><title>',+            id="missing head and html",+        ),+        pytest.param(+            b"<p>fragment</p>",+            b'<base href="http://www.example.com"><p>fragment</p>',+            id="bare fragment",+        ),+        pytest.param(+            b"<html><body><header>top</header></body></html>",+            b'<html><base href="http://www.example.com"><body><header>top',+            id="header element is not head",+        ),+        pytest.param(+            b"<HTML><HEAD><TITLE>T</TITLE></HEAD><BODY></BODY></HTML>",+            b'<HEAD><base href="http://www.example.com"><TITLE>',+            id="uppercase head",+        ),+        pytest.param(+            b'<html><head\nclass="x"><title>T</title></head></html>',+            b'<head\nclass="x"><base href="http://www.example.com"><title>',+            id="head with newline before attributes",+        ),+        pytest.param(+            b"<html><!-- <head> --><head><title>T</title></head></html>",+            b'<!-- <head> --><head><base href="http://www.example.com"><title>',+            id="head in comment is skipped",+        ),+        pytest.param(+            b"<html><!-- no head here --><body></body></html>",+            b'<html><base href="http://www.example.com"><!-- no head here -->',+            id="missing head, comment preserved",+        ),+        pytest.param(+            b"<html><body><basefont size=3></body></html>",+            b'<html><base href="http://www.example.com"><body><basefont',+            id="basefont is not base",+        ),+    ],+)+def test_open_in_browser_injects_base_without_proper_head(+    body: bytes, expected: bytes+) -> None:+    bbody = _open_and_read(body)+    assert bbody.count(b"<base href=") == 1+    assert expected in bbody+++def test_open_in_browser_does_not_inject_base_when_uppercase_base_present():+    body = b'<html><head><BASE HREF="http://real.com"></head></html>'+    bbody = _open_and_read(body)+    assert bbody == body+++def test_open_in_browser_escapes_base_url():+    url = 'http://www.example.com/?a=1&b="2"'+    bbody = _open_and_read(b"<html><head></head></html>", url=url)+    assert (+        b'<head><base href="http://www.example.com/?a=1&amp;b=&quot;2&quot;"></head>'+        in bbody+    )+++def test_open_in_browser_keeps_utf8_bom_first():+    bbody = _open_and_read(b"\xef\xbb\xbf<p>fragment</p>")+    assert bbody == b'\xef\xbb\xbf<base href="http://www.example.com"><p>fragment</p>'+++def test_open_in_browser_does_not_corrupt_utf16_body():+    body = "<p>hé</p>".encode("utf-16")+    bbody = _open_and_read(body)+    assert bbody == body+    assert bbody.decode("utf-16") == "<p>hé</p>" 
Merged fix · PR #7879+60 −52
pyproject.tomlindex 13aae01ac..81f3c062b 100644@@ -21,7 +21,7 @@ dependencies = [     "queuelib>=1.6.1",     "service_identity>=24.2.0",     "tldextract",-    "w3lib>=1.17.0",+    "w3lib>=2.1.0",     "zope.interface>=5.1.0",     # Platform-specific dependencies     'PyDispatcher>=2.0.5; platform_python_implementation == "CPython"',scrapy/utils/response.pyindex b3622c159..41bdac163 100644@@ -60,15 +60,7 @@ def response_status_message(status: bytes | float | str) -> str:     return f"{status_int} {to_unicode(message)}"  -def _remove_html_comments(body: bytes) -> bytes:-    start = body.find(b"<!--")-    while start != -1:-        end = body.find(b"-->", start + 1)-        if end == -1:-            return body[:start]-        body = body[:start] + body[end + 3 :]-        start = body.find(b"<!--")-    return body+_DOCTYPE_RE = re.compile(rb"\s*<!doctype[^<>]*>", re.IGNORECASE)   def open_in_browser(@@ -98,12 +90,17 @@ def open_in_browser(     # circular imports     from scrapy.http import HtmlResponse, TextResponse  # noqa: PLC0415 -    # XXX: this implementation is a bit dirty and could be improved     body = response.body     if isinstance(response, HtmlResponse):-        if b"<base" not in _remove_html_comments(body):-            repl = rf'\g<0><base href="{response.url}">'-            body = re.sub(rb"<head(?:[^<>]*?>)", to_bytes(repl), body, count=1)+        # Web browsers move a base tag that precedes the head element into it,+        # so the head element does not need to be found, which is not always+        # possible. The base tag must come after the doctype declaration, if+        # any, to keep the browser out of quirks mode. It takes precedence over+        # any base tag of the response, and matches it when there is one.+        doctype = _DOCTYPE_RE.match(body)+        index = doctype.end() if doctype else 0+        base_tag = to_bytes(f'<base href="{get_base_url(response)}">')+        body = body[:index] + base_tag + body[index:]         ext = ".html"     elif isinstance(response, TextResponse):         ext = ".txt"tests/test_utils_response.pyindex 3638c52b7..8269f0ad0 100644@@ -9,7 +9,6 @@ import pytest from scrapy.http import HtmlResponse, Response, TextResponse from scrapy.utils.python import to_bytes from scrapy.utils.response import (-    _remove_html_comments,     get_base_url,     get_meta_refresh,     open_in_browser,@@ -169,6 +168,31 @@ def test_response_status_message():     </html>""",             id="Conditional comment",         ),+        pytest.param(+            b"""+    <html>+        <body><p>Hello world.</p></body>+    </html>""",+            id="No <head>",+        ),+        pytest.param(+            b"<p>Hello world.</p>",+            id="No <html>",+        ),+        pytest.param(+            b"""<!DOCTYPE html>+    <html>+        <head><title>Dummy</title></head>+        <body><p>Hello world.</p></body>+    </html>""",+            id="Doctype",+        ),+        pytest.param(+            b"""+    <!-- <head><base href="http://example.org"></head> -->+    <p>Hello world.</p>""",+            id="Only commented-out <head> and <base>",+        ),     ], ) def test_inject_base_url(body: bytes) -> None:@@ -176,8 +200,13 @@ def test_inject_base_url(body: bytes) -> None:      def check_base_url(burl):         bbody = _read_browser_output(burl)-        assert bbody.count(b'><base href="' + to_bytes(url) + b'">') == 1-        assert b"<head" in bbody+        base_tag = b'<base href="' + to_bytes(url) + b'">'+        assert bbody.count(base_tag) == 1+        index = bbody.index(base_tag)+        # The base tag is not commented out.+        assert bbody.rfind(b"<!--", 0, index) <= bbody.rfind(b"-->", 0, index)+        # The base tag comes after the doctype declaration, if any.+        assert b"<!DOCTYPE" not in bbody[index:]         return True      resp = HtmlResponse(url, body=body)@@ -211,24 +240,6 @@ def test_open_in_browser_redos_head():     _assert_open_in_browser_is_fast(b"<head\t" * 80_000)  -@pytest.mark.parametrize(-    ("input_body", "output_body"),-    [-        (b"a<!--", b"a"),-        (b"a<!---->b", b"ab"),-        (b"a<!--b-->c", b"ac"),-        (b"a<!--b-->c<!--", b"ac"),-        (b"a<!--b-->c<!--d", b"ac"),-        (b"a<!--b-->c<!---->d", b"acd"),-        (b"a<!--b--><!--c-->d", b"ad"),-        (b"a<!-- <!-- inner --> -->b", b"a -->b"),-        (b"<!-- <head>fake</head> --><head>real</head>", b"<head>real</head>"),-    ],-)-def test_remove_html_comments(input_body: bytes, output_body: bytes) -> None:-    assert _remove_html_comments(input_body) == output_body-- def test_open_in_browser_preserves_html_comments():     url = "http://www.example.com"     body = (@@ -248,19 +259,24 @@ def test_open_in_browser_preserves_html_comments():     assert open_in_browser(response, _openfunc=check)  -def test_open_in_browser_does_not_inject_base_when_present():-    url = "http://www.example.com"-    body = (-        b"<html>"-        b'<head><base href="http://real.com"><title>T</title></head>'-        b"<body>hi</body>"-        b"</html>"-    )+@pytest.mark.parametrize(+    ("base_tag", "expected_base_url"),+    [+        (b'<base href="http://real.com/">', b"http://real.com/"),+        (b'<BASE HREF="http://real.com/">', b"http://real.com/"),+        (b'<base href="/img/">', b"http://www.example.com/img/"),+        (b'<base target="_blank">', b"http://www.example.com/page.html"),+    ],+)+def test_open_in_browser_keeps_base_url_of_response(+    base_tag: bytes, expected_base_url: bytes+):+    url = "http://www.example.com/page.html"+    body = b"<html><head>" + base_tag + b"<title>T</title></head><body>hi</body></html>"      def check(burl):         bbody = _read_browser_output(burl)-        assert b'<base href="' + to_bytes(url) + b'">' not in bbody-        assert b'<base href="http://real.com">' in bbody+        assert bbody.startswith(b'<base href="' + expected_base_url + b'">')         return True      response = HtmlResponse(url, body=body)@@ -286,7 +302,7 @@ def test_open_in_browser_injects_base_when_only_in_comment():     assert open_in_browser(response, _openfunc=check)  -def test_open_in_browser_injects_base_at_real_head_not_commented_head():+def test_open_in_browser_injects_base_before_head_contents():     url = "http://www.example.com"     body = (         b"<html>"tox.iniindex e807317b3..d2a682415 100644@@ -149,7 +149,7 @@ deps =     pyOpenSSL==24.3.0     queuelib==1.6.1     service_identity==24.2.0-    w3lib==1.17.0+    w3lib==2.1.0     zope.interface==5.1.0     {[test-requirements]deps} setenv =@@ -307,11 +307,6 @@ deps =     pyOpenSSL==24.3.0     queuelib==1.6.1     service_identity==24.2.0-    # w3lib 1.17 fails to import on PyPy 3.11 because its encoding regex uses-    # an inline flag placement that Python 3.11 treats as an error: global-    # flags not at the start of the expression. w3lib 1.18 stopped encoding []-    # in URLs until 2.1.0 brought that behavior back. Tests for verbatim_url-    # rely on that encoding.     w3lib==2.1.0     zope.interface==5.1.0 commands =