Backtestsjsouprun-20261006-181938#2543

Issue #2543 Merged fix PR # Run 6 Oct 2026 · run-20261006-181938 jhy/jsoup

Correct offsite link detection in Cleaner

Better

Our patch was judged better than the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

Both patches replace the string-prefix check with a case-insensitive hostname comparison, so both fix the two core symptoms: a prefixed host like example.com.test being treated as local, and same-host links outside the base path being treated as external.

Patch B also covers the issue's other two requirements.

- **Relative links stay local.** Patch B detects a relative href directly: it resolves the raw href against two different placeholder hosts and checks whether it takes the placeholder's host both times. This keeps relative links local even when the base URI has no host, such as `file:///home/x/`. Patch A instead requires a non-empty base host, so a relative link under a `file:` base gets `rel=nofollow`. That contradicts "relative links should remain local."
- **Absolute links are external with no base URI.** Patch B handles the case where the base is only the internal `DummyUri` placeholder, so an absolute link to the placeholder host itself is still external. Under Patch A, an absolute link to the dummy host would match the placeholder base and be treated as local. This is an edge case, but it is part of the issue's stated semantics.

On tests, Patch B's are much more thorough. They cover:
- a lookalike host, including userinfo (`example.com@evil.com`);
- a base with a path, both with and without `preserveRelativeLinks`;
- case-insensitive hosts;
- the no-base case, including the placeholder host and protocol-relative links;
- a base with no host (`file:`).

Patch A's single test covers only the two headline cases.

On craft, Patch B's two-placeholder trick is somewhat clever and adds complexity. It is well documented, though, and moved into named helpers. Patch A is simpler but misses stated requirements. Overall, Patch B is the more correct and better-tested fix.

The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.

Solve

10m 15s

Wall-clock time in the container

Full test suite

No new failures

2,088 passed · 0 skipped · 2,088 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

85vs 27 merged

Changed lines, ours vs merged

The Cleaner uses a string-prefix comparison to determine whether a link points to the same site as the document base URI.

This can incorrectly treat a prefixed hostname as local when the base URI has no trailing slash, and can treat local links as external when the base URI contains a path.

Parse and compare the resolved link and base URI hostnames instead.

Relative links should remain local, while absolute links should be considered external when no base URI is available.

The two patches

Our patch+82 −3
CHANGES.mdindex 2993c271..561d8e77 100644@@ -16,6 +16,7 @@ * Fixed `W3CDom` XML conversion so plain XML elements don't serialize with the reserved XML namespace as the default namespace. Explicit XML namespaces and `xml:*` attributes are still preserved. [#2504](https://github.com/jhy/jsoup/issues/2504) * Preserve control characters in parsed tag names [#2358](https://github.com/jhy/jsoup/issues/2538) * Updated HTTP redirects to follow the specification: 307 and 308 preserve the request method and content, 301 and 302 only change POST to GET, and `Location` is followed only for 301, 302, 303, 307, and 308 responses. Streamed request bodies are not buffered; if an automatic redirect would require replaying one, execution fails so the caller can resend with a fresh stream. [#2540](https://github.com/jhy/jsoup/pull/2540)+* In the `Cleaner`, the check that decides whether a link is offsite (and so gets an enforced `rel=nofollow`) now compares the hostnames of the resolved link and the base URI, instead of checking whether one string starts with the other. Previously a lookalike host such as `http://example.com.evil.com/` was treated as local when the base URI had no trailing slash, and same-host links outside the base URI's path were treated as external. Relative links stay local; when no base URI is given, absolute links are treated as external.  ### Build Changes * Cleaned up the Maven build for the multi-release JAR so Java 8 and Java 11+ sources compile as separate source sets. This avoids spurious Java 8 compiler warnings from newer-language overlay sources, keeps long-running parser checks behind an explicit profile, and preserves the same published artifacts and runtime behavior.src/main/java/org/jsoup/safety/Cleaner.javaindex a1c10d31..9db5435f 100644@@ -1,6 +1,7 @@ package org.jsoup.safety;  import org.jsoup.helper.Validate;+import org.jsoup.internal.StringUtil; import org.jsoup.nodes.Attribute; import org.jsoup.nodes.Attributes; import org.jsoup.nodes.DataNode;@@ -14,6 +15,8 @@ import org.jsoup.parser.ParseErrorList; import org.jsoup.parser.Parser; import org.jsoup.select.NodeVisitor; +import java.net.MalformedURLException;+import java.net.URL; import java.util.List;  import static org.jsoup.internal.SharedConstants.DummyUri;@@ -212,9 +215,7 @@ public class Cleaner {         Attributes enforcedAttrs = safelist.getEnforcedAttributes(sourceTag);         // special case for <a href rel=nofollow>, only apply to external links:         if (sourceEl.nameIs("a") && enforcedAttrs.get("rel").equals("nofollow")) {-            String href = sourceEl.absUrl("href");-            String sourceBase = sourceEl.baseUri();-            if (!href.isEmpty() && !sourceBase.isEmpty() && href.startsWith(sourceBase)) { // same site, so don't set the nofollow+            if (isSameSiteLink(sourceEl)) { // same site, so don't set the nofollow                 enforcedAttrs.remove("rel");             }         }@@ -228,6 +229,45 @@ public class Cleaner {         return new ElementMeta(dest, numDiscarded);     } +    /**+     Tests if the element's href links to the same site as its base URI, by comparing the hostnames of the resolved link+     and the base. Relative links are local. If there is no base URI (or only the placeholder {@link+     org.jsoup.internal.SharedConstants#DummyUri DummyUri}), absolute links are considered external.+     */+    private static boolean isSameSiteLink(Element sourceEl) {+        String href = sourceEl.absUrl("href");+        String sourceBase = sourceEl.baseUri();+        if (href.isEmpty() || sourceBase.isEmpty()) return false; // unresolvable, or no base: treat as external++        // a relative link resolves to the placeholder host against both placeholders; an absolute link keeps its own+        // host, so can't match both. Relative links are local, even if the base has no host (e.g. file:)+        String rawHref = sourceEl.attr("href");+        if (hostsMatch(StringUtil.resolve(DummyUri, rawHref), DummyUri)+            && hostsMatch(StringUtil.resolve(AltDummyUri, rawHref), AltDummyUri))+            return true;+        if (sourceBase.equals(DummyUri)) return false; // no real base, so absolute links are external+        return hostsMatch(href, sourceBase);+    }++    private static final String AltDummyUri = "https://alt.dummy.example/";++    /** Tests if both URLs have the same non-empty host (case-insensitive). False if either can't be parsed. */+    private static boolean hostsMatch(String url, String otherUrl) {+        String host = hostOf(url);+        String otherHost = hostOf(otherUrl);+        return !host.isEmpty() && host.equalsIgnoreCase(otherHost);+    }++    private static String hostOf(String url) {+        if (url.isEmpty()) return "";+        try {+            String host = new URL(url).getHost();+            return host == null ? "" : host;+        } catch (MalformedURLException e) {+            return ""; // not a parsable URL, so has no host to match; caller treats as external+        }+    }+     private static class ElementMeta {         Element el;         int numAttribsDiscarded;src/test/java/org/jsoup/safety/CleanerTest.javaindex f22eb826..fa8a70ba 100644@@ -593,6 +593,44 @@ public class CleanerTest {         assertEquals("<a href=\"http://external.com/\" rel=\"nofollow\">One</a> <a href=\"/relative/\">Two</a> <a href=\"../other/\">Three</a> <a href=\"http://example.com/bar\" rel=\"nofollow\">Four</a>", clean4);     } +    @Test void nofollowOnPrefixedHostWhenBaseHasNoTrailingSlash() {+        // a host that merely shares the base's string prefix is offsite+        Safelist basic = Safelist.basic();+        String html = "<a href='http://example.com.evil.com/'>One</a> <a href='http://example.com@evil.com/'>Two</a> <a href='http://example.com/ok'>Three</a>";+        String clean = Jsoup.clean(html, "http://example.com", basic);+        assertEquals("<a href=\"http://example.com.evil.com/\" rel=\"nofollow\">One</a> <a href=\"http://evil.com/\" rel=\"nofollow\">Two</a> <a href=\"http://example.com/ok\">Three</a>", clean); // userinfo is stripped on abs resolve+    }++    @Test void noNofollowOnSameHostWhenBaseHasPath() {+        // links on the same host are local, even if not under the base URI's path+        Safelist basic = Safelist.basic().preserveRelativeLinks(true);+        String html = "<a href='http://example.com/other/page'>One</a> <a href='/root'>Two</a> <a href='HTTPS://EXAMPLE.COM/x'>Three</a> <a href='http://external.com/a/b/c'>Four</a>";+        String clean = Jsoup.clean(html, "http://example.com/a/b/c", basic);+        assertEquals("<a href=\"http://example.com/other/page\">One</a> <a href=\"/root\">Two</a> <a href=\"HTTPS://EXAMPLE.COM/x\">Three</a> <a href=\"http://external.com/a/b/c\" rel=\"nofollow\">Four</a>", clean);++        // and without preserving relative links (so they are made absolute)+        String clean2 = Jsoup.clean("<a href='/root'>One</a> <a href='http://example.com.evil.com/a/b/c'>Two</a>", "http://example.com/a/b/c", Safelist.basic());+        assertEquals("<a href=\"http://example.com/root\">One</a> <a href=\"http://example.com.evil.com/a/b/c\" rel=\"nofollow\">Two</a>", clean2);+    }++    @Test void absoluteLinksExternalWithoutBaseUri() {+        // with no base URI, relative links remain local but any absolute link is external, including the placeholder host+        Safelist basic = Safelist.basic().preserveRelativeLinks(true);+        String html = "<a href='/relative'>One</a> <a href='https://dummy.example/'>Two</a> <a href='//dummy.example/x'>Three</a> <a href='http://example.com/'>Four</a>";+        String clean = Jsoup.clean(html, basic);+        assertEquals("<a href=\"/relative\">One</a> <a href=\"https://dummy.example/\" rel=\"nofollow\">Two</a> <a href=\"//dummy.example/x\" rel=\"nofollow\">Three</a> <a href=\"http://example.com/\" rel=\"nofollow\">Four</a>", clean);++        Document dirty = Jsoup.parse("<a href='http://example.com/'>One</a>");+        assertEquals("<a href=\"http://example.com/\" rel=\"nofollow\">One</a>", new Cleaner(Safelist.basic()).clean(dirty).body().html());+    }++    @Test void relativeLinksLocalWhenBaseHasNoHost() {+        Safelist safelist = Safelist.basic().preserveRelativeLinks(true).addProtocols("a", "href", "file");+        String html = "<a href='a.html'>One</a> <a href='http://example.com/'>Two</a>";+        String clean = Jsoup.clean(html, "file:///home/x/", safelist);+        assertEquals("<a href=\"a.html\">One</a> <a href=\"http://example.com/\" rel=\"nofollow\">Two</a>", clean);+    }+     @Test void canonicalizesEnforcedAttributes() {         Document customDirty = Jsoup.parse("<a REL='external'>One</a>", "",             Parser.htmlParser().settings(ParseSettings.preserveCase)); 
Merged fix · PR #+23 −4
CHANGES.mdindex 2993c271..fa9037bd 100644@@ -16,6 +16,7 @@ * Fixed `W3CDom` XML conversion so plain XML elements don't serialize with the reserved XML namespace as the default namespace. Explicit XML namespaces and `xml:*` attributes are still preserved. [#2504](https://github.com/jhy/jsoup/issues/2504) * Preserve control characters in parsed tag names [#2358](https://github.com/jhy/jsoup/issues/2538) * Updated HTTP redirects to follow the specification: 307 and 308 preserve the request method and content, 301 and 302 only change POST to GET, and `Location` is followed only for 301, 302, 303, 307, and 308 responses. Streamed request bodies are not buffered; if an automatic redirect would require replaying one, execution fails so the caller can resend with a fresh stream. [#2540](https://github.com/jhy/jsoup/pull/2540)+* Corrected the Cleaner's same-site link detection to compare hostnames rather than URL prefixes when applying `rel=nofollow`. [2543](https://github.com/jhy/jsoup/issues/2543)  ### Build Changes * Cleaned up the Maven build for the multi-release JAR so Java 8 and Java 11+ sources compile as separate source sets. This avoids spurious Java 8 compiler warnings from newer-language overlay sources, keeps long-running parser checks behind an explicit profile, and preserves the same published artifacts and runtime behavior.src/main/java/org/jsoup/safety/Cleaner.javaindex a1c10d31..6f419f8d 100644@@ -14,6 +14,8 @@ import org.jsoup.parser.ParseErrorList; import org.jsoup.parser.Parser; import org.jsoup.select.NodeVisitor; +import java.net.MalformedURLException;+import java.net.URL; import java.util.List;  import static org.jsoup.internal.SharedConstants.DummyUri;@@ -208,14 +210,18 @@ public class Cleaner {                 numDiscarded++;         } -         Attributes enforcedAttrs = safelist.getEnforcedAttributes(sourceTag);         // special case for <a href rel=nofollow>, only apply to external links:         if (sourceEl.nameIs("a") && enforcedAttrs.get("rel").equals("nofollow")) {             String href = sourceEl.absUrl("href");-            String sourceBase = sourceEl.baseUri();-            if (!href.isEmpty() && !sourceBase.isEmpty() && href.startsWith(sourceBase)) { // same site, so don't set the nofollow-                enforcedAttrs.remove("rel");+            if (!href.isEmpty()) {+                try {+                    URL baseUrl = new URL(sourceEl.baseUri());+                    URL linkUrl = new URL(href);+                    String baseHost = baseUrl.getHost();+                    if (!baseHost.isEmpty() && baseHost.equalsIgnoreCase(linkUrl.getHost())) // same site, so don't set the nofollow+                        enforcedAttrs.remove("rel");+                } catch (MalformedURLException ignored) {}             }         } src/test/java/org/jsoup/safety/CleanerTest.javaindex f22eb826..f877e6a0 100644@@ -593,6 +593,18 @@ public class CleanerTest {         assertEquals("<a href=\"http://external.com/\" rel=\"nofollow\">One</a> <a href=\"/relative/\">Two</a> <a href=\"../other/\">Three</a> <a href=\"http://example.com/bar\" rel=\"nofollow\">Four</a>", clean4);     } +    @Test void nofollowComparesLinkHosts() {+        Safelist basic = Safelist.basic();++        // A hostname that extends the base hostname is still external+        String external = Jsoup.clean("<a href='https://example.com.test/path'>Link</a>", "https://example.com", basic);+        assertEquals("<a href=\"https://example.com.test/path\" rel=\"nofollow\">Link</a>", external);++        // The base URI path, scheme, and port do not make another URL on the same host external+        String internal = Jsoup.clean("<a href='http://example.com:8080/other'>Link</a>", "https://example.com:8443/path/page", basic);+        assertEquals("<a href=\"http://example.com:8080/other\">Link</a>", internal);+    }+     @Test void canonicalizesEnforcedAttributes() {         Document customDirty = Jsoup.parse("<a REL='external'>One</a>", "",             Parser.htmlParser().settings(ParseSettings.preserveCase));