Backtestsscrapyrun-20261025-211112#5532
LinkExtractor calls process_value before applying allow and deny
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
The issue's stated expected behaviour is that allow/deny filter links before process_value is invoked. Patch A implements exactly that: when process_value is given, LxmlParserLinkExtractor builds a Link from the raw attribute value and runs the extractor's _link_allowed check before calling process_value, then still filters the processed result. It sensibly lets non-URL values (e.g. javascript: hrefs that process_value converts) through to process_value so the documented javascript example keeps working, only wires the filter when process_value is set (no cost otherwise), and updates the docstring to describe the new semantics. It adds meaningful tests covering each filter parameter, the javascript case, and the re-filtering of the returned value; the independent tests derived from the issue go from FAIL on base to PASS with A. Weaknesses of A: it is a behaviour change that can break users whose process_value rewrites a valid URL into one that only then matches allow (A's own test enshrines that this is now dropped), and it does a redundant safe_url_string/Link construction per link; these are real but modest risks. Patch B is docs-only, declaring the existing order intentional and pointing to Rule.process_links. The issue did allow for 'by design, but document it', so B is a defensible stance, but it leaves the reporter's actual expected behaviour unmet, adds no tests, and every independent test still fails under B. On correctness against the reported expectation and on tests, A is clearly ahead; its scope and craft are reasonable, so A wins.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
4m 6s
Wall-clock time in the container
Full test suite
No new failures
48 passed · 0 skipped · 48 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
131vs 6 merged
Changed lines, ours vs merged
The issue
View on GitHubDescription
I'm rather new to scrapy, created some 100 spiders and now moved the data to a MySql database.
To check if the page is scraped before I decided to use a process_value function, in connection with a list retrieved from the database.
That part is working OK.
But, when I did that, I saw multiple "new" links in my tests, which should be filtered by "allow".
This can be by design, but at least the documentation does not state that.
Expected behavior: [What you expect to happen]
"allow" and "deny" lists should filter the links before calling process_value callable.
Actual behavior: [What actually happens]
"allow" and "deny" lists are ignored.
Reproduces how often: [What percentage of the time does it reproduce?]
Tried on 2 test spiders with the same results.
Additional context
I also could not find anything on web or in the issues here. Perhaps CrawlSpider/LinkExtractor/process_value seems not be used much...
The two patches
scrapy/linkextractors/lxmlhtml.pyindex 3fb741d7a..f7f39fc58 100644@@ -78,6 +78,7 @@ class LxmlParserLinkExtractor: unique: bool = False, strip: bool = True, canonicalized: bool = False,+ link_filter: Callable[[Link], bool] | None = None, ): # mypy doesn't infer types for operator.* and also for partial() self.scan_tag: Callable[[str], bool] = (@@ -93,6 +94,9 @@ class LxmlParserLinkExtractor: self.process_attr: Callable[[Any], Any] = ( process if callable(process) else _identity )+ # Called with the link built from the unprocessed URL before *process*,+ # so that links rejected by it never reach *process*.+ self.link_filter: Callable[[Link], bool] | None = link_filter self.unique: bool = unique self.strip: bool = strip self.link_key: Callable[[Link], str] = (@@ -131,6 +135,8 @@ class LxmlParserLinkExtractor: except ValueError: continue # skipping bogus links else:+ if not self._raw_link_allowed(el, attr_val, response_encoding):+ continue url = self.process_attr(attr_val) if url is None: continue@@ -150,6 +156,26 @@ class LxmlParserLinkExtractor: links.append(link) return self._deduplicate_if_needed(links) + def _raw_link_allowed(+ self, el: HtmlElement, url: str, response_encoding: str+ ) -> bool:+ """Return whether the link built from the unprocessed *url* passes+ :attr:`link_filter`.++ Values that are not valid URLs yet (e.g. ``javascript:`` hrefs that+ *process* turns into URLs) are let through, to be filtered once+ processed.+ """+ if self.link_filter is None:+ return True+ try:+ url = safe_url_string(url, encoding=response_encoding)+ except ValueError:+ return True+ if not _is_valid_url(url):+ return True+ return self.link_filter(Link(url, _collect_string_content(el) or ""))+ def extract_links(self, response: TextResponse) -> list[Link]: base_url = get_base_url(response) return self._extract_links(@@ -265,6 +291,15 @@ class LxmlLinkExtractor: new one, or return ``None`` to ignore the link altogether. If not given, ``process_value`` defaults to ``lambda x: x``. + Links whose (absolute) URL is rejected by the filtering parameters+ above (``allow``, ``deny``, ``allow_domains``, ``deny_domains``,+ ``deny_extensions`` and ``restrict_text``) are discarded before+ ``process_value`` is called, so ``process_value`` only receives+ values of links that would otherwise be extracted. Values that are+ not valid URLs yet, such as the ``javascript:`` value below, are+ still passed to ``process_value``. The value returned by+ ``process_value`` is filtered again.+ .. highlight:: html For example, to extract links from this code::@@ -327,6 +362,7 @@ class LxmlLinkExtractor: process=process_value, strip=strip, canonicalized=not canonicalize,+ link_filter=self._link_allowed if process_value is not None else None, ) self.allow_res: list[re.Pattern[str]] = self._compile_regexes(allow) self.deny_res: list[re.Pattern[str]] = self._compile_regexes(deny)tests/test_linkextractors.pyindex 95a6aee54..d051421c6 100644@@ -377,6 +377,101 @@ class Base: Link(url="http://example.org/other/page.html", text="Text") ] + def test_process_value_javascript_with_allow(self):+ html = b"""+<a href="javascript:goToPage('../other/page.html'); return false">Text</a>+<a href="javascript:goToPage('../denied/page.html'); return false">Denied</a>+ """+ response = HtmlResponse("http://example.org/somepage/index.html", body=html)++ def process_value(value):+ m = re.search(r"javascript:goToPage\('(.*?)'", value)+ return m.group(1) if m else None++ lx = self.extractor_cls(allow="other", process_value=process_value)+ assert lx.extract_links(response) == [+ Link(url="http://example.org/other/page.html", text="Text")+ ]++ @pytest.mark.parametrize(+ ("kwargs", "expected"),+ [+ (+ {"allow": "allowed"},+ [+ "http://example.org/allowed.html",+ "http://other.example/allowed.html",+ ],+ ),+ (+ {"deny": "denied"},+ [+ "http://example.org/allowed.html",+ "http://example.org/photo.jpg",+ "http://other.example/allowed.html",+ ],+ ),+ (+ {"allow_domains": "example.org"},+ [+ "http://example.org/allowed.html",+ "http://example.org/denied.html",+ "http://example.org/photo.jpg",+ ],+ ),+ (+ {"deny_domains": "example.org"},+ ["http://other.example/allowed.html"],+ ),+ (+ {"deny_extensions": ["html"]},+ ["http://example.org/photo.jpg"],+ ),+ ({"restrict_text": "Photo"}, ["http://example.org/photo.jpg"]),+ ],+ )+ def test_process_value_called_after_filtering(self, kwargs, expected):+ html = b"""+<a href="/allowed.html">Allowed</a>+<a href="/denied.html">Denied</a>+<a href="/photo.jpg">Photo</a>+<a href="http://other.example/allowed.html">Other</a>+ """+ response = HtmlResponse("http://example.org/index.html", body=html)+ seen = []++ def process_value(value):+ seen.append(value)+ return value++ kwargs.setdefault("deny_extensions", [])+ lx = self.extractor_cls(process_value=process_value, **kwargs)+ assert [link.url for link in lx.extract_links(response)] == expected+ assert seen == expected++ def test_process_value_cannot_unfilter_denied_link(self):+ html = b'<a href="/denied.html">Denied</a>'+ response = HtmlResponse("http://example.org/index.html", body=html)+ seen = []++ def process_value(value):+ seen.append(value)+ return value.replace("denied", "allowed")++ lx = self.extractor_cls(allow="allowed", process_value=process_value)+ assert lx.extract_links(response) == []+ assert seen == []++ def test_process_value_result_is_filtered(self):+ html = b'<a href="/allowed.html">Allowed</a>'+ response = HtmlResponse("http://example.org/index.html", body=html)++ def process_value(value):+ return value.replace("allowed", "denied")++ lx = self.extractor_cls(deny="denied", process_value=process_value)+ assert lx.extract_links(response) == []+ def test_base_url_with_restrict_xpaths(self): html = b"""<html><head><title>Page title</title><base href="http://otherdomain.com/base/" /></head> <body><p><a href="item/12.html">Item 12</a></p>
scrapy/linkextractors/lxmlhtml.pyindex 3fb741d7a..46ac39a28 100644@@ -282,6 +282,12 @@ class LxmlLinkExtractor: if m: return m.group(1) + ``process_value`` is called before the filtering parameters, such as+ ``allow`` and ``deny``, which match the value that it returns. To drop+ links based on their final URL, use the ``process_links`` parameter of+ :class:`~scrapy.spiders.Rule`, which only receives links that those+ parameters kept.+ :type process_value: collections.abc.Callable :param strip: whether to strip whitespaces from extracted attributes.