Backtestsjsouprun-20261006-181938#2542

Issue #2542 Merged fix PR # Run 6 Oct 2026 · run-20261006-181938 jhy/jsoup

Parse CDATA as a bogus comment in HTML content

Worse

The merged fix was judged better than our patch.

In this judgment, Patch B = our agent, Patch A = the merged fix.

Both patches correctly make `<![CDATA[` in HTML-namespace content a bogus comment that ends at the next `>`. Both keep CDATA in SVG, MathML and the XML parser.

The difference is in fragment parsing, where the spec's 'adjusted current node' is the context element.

Patch A keeps the real context tag, including its namespace, when it builds the fragment context element. Fragments parsed inside `<svg>` or `<math>` therefore behave as foreign content all the way down: nested children get the SVG/MathML namespace, and CDATA inside those nested children is still recognized. It also fixes the knock-on effects of keeping the namespace:
- tokeniser start state only switches to script, plaintext or template mode for HTML contexts;
- noscript handling only applies to HTML contexts;
- `resetInsertionMode` treats non-HTML nodes without `continue`, which also removes a latent null dereference.
These changes are broader, but each follows from honouring the context namespace and is covered by tests (SVG and MathML fragments, HTML integration points, foreign script/plaintext/noscript contexts, custom foreign data tags).

Patch B adds a side field, `contextNamespace`, and consults it only while the context element is the sole item on the stack. The context copy itself stays in the HTML namespace. As a result, `svg.html("<g><![CDATA[x]]></g>")` would create an HTML-namespace `<g>`, and the CDATA inside it would wrongly become a bogus comment. Its fix is partial for fragments.

Patch B's tokenizer change (`matches && allowsCdata && matchConsume`, falling through to the existing bogus-comment branch) and its virtual `allowsCdata` in `TreeBuilder` are tidy.

On tests, both update the existing CDATA tests and add HTML-vs-foreign cases. Patch A's tests are more thorough: integration points, nested SVG, XML with the HTML namespace, error position, and fragment namespaces.

Patch A wins on correctness and test depth.

The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.

Solve

11m 49s

Wall-clock time in the container

Full test suite

No new failures

2,092 passed · 0 skipped · 2,092 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

183vs 301 merged

Changed lines, ours vs merged

The HTML parser currently recognizes <![CDATA[...]]> in all contexts and creates a CDataNode.

Per the HTML specification, CDATA sections are recognized only in foreign content such as SVG and MathML.

In the HTML namespace, <![CDATA[ is a parse error and should start a bogus comment, which ends at the next >.

Update the tokenizer to consider the adjusted current node’s namespace so that HTML parsing aligns with browser behavior.

CDATA handling should remain unchanged for SVG, MathML, and XML parsing.

The two patches

Our patch+148 −35
CHANGES.mdindex 3ca16dd0..2d7035b3 100644@@ -18,6 +18,7 @@ * Preserve control characters in parsed tag names [#2358](https://github.com/jhy/jsoup/issues/2538) * Updated HTTP redirects to follow the specification: 307 and 308 preserve the request method and content, 301 and 302 only change POST to GET, and `Location` is followed only for 301, 302, 303, 307, and 308 responses. Streamed request bodies are not buffered; if an automatic redirect would require replaying one, execution fails so the caller can resend with a fresh stream. [#2540](https://github.com/jhy/jsoup/pull/2540) * Corrected the Cleaner's same-site link detection to compare hostnames rather than URL prefixes when applying `rel=nofollow`. [2543](https://github.com/jhy/jsoup/issues/2543)+* In HTML content, `<![CDATA[...]]>` is now parsed as a bogus comment (ending at the next `>`), per the HTML spec and browser behavior. CDATA sections are still recognized in foreign content (SVG and MathML), and in the XML parser.  ### Build Changes * Cleaned up the Maven build for the multi-release JAR so Java 8 and Java 11+ sources compile as separate source sets. This avoids spurious Java 8 compiler warnings from newer-language overlay sources, keeps long-running parser checks behind an explicit profile, and preserves the same published artifacts and runtime behavior.src/main/java/org/jsoup/parser/HtmlTreeBuilder.javaindex 8fc68a43..2814a4aa 100644@@ -45,6 +45,7 @@ public class HtmlTreeBuilder extends TreeBuilder {     private @Nullable Element headElement; // the current head element     private @Nullable FormElement formElement; // the current form element     private @Nullable Element contextElement; // fragment parse root; name only copy of context. could be null even if fragment parsing+    private @Nullable String contextNamespace; // namespace of the original fragment context; the copy above is in the default namespace     ArrayList<Element> formattingElements; // active (open) formatting elements     private ArrayList<HtmlTreeBuilderState> tmplInsertMode; // stack of Template Insertion modes     private @Nullable NoscriptState noscriptState; // active noscript island state@@ -75,6 +76,7 @@ public class HtmlTreeBuilder extends TreeBuilder {         headElement = null;         formElement = null;         contextElement = null;+        contextNamespace = null;         formattingElements = new ArrayList<>();         tmplInsertMode = new ArrayList<>();         noscriptState = null;@@ -93,6 +95,7 @@ public class HtmlTreeBuilder extends TreeBuilder {         if (context != null) {             final String contextName = context.normalName();             contextElement = new Element(tagFor(contextName, contextName, defaultNamespace(), settings), baseUri);+            contextNamespace = context.tag().namespace();             if (context.ownerDocument() != null) // quirks setup:                 doc.quirksMode(context.ownerDocument().quirksMode()); @@ -230,6 +233,21 @@ public class HtmlTreeBuilder extends TreeBuilder {         return true;     } +    /**+     Per spec, a CDATA section is only recognized when the adjusted current node is not an element in the HTML+     namespace (i.e. in SVG or MathML foreign content). Otherwise, {@code <![CDATA[} starts a bogus comment.+     */+    @Override boolean allowsCdata() {+        String ns;+        if (stack.isEmpty())+            ns = defaultNamespace();+        else if (stack.size() == 1 && stack.get(0) == contextElement && contextNamespace != null)+            ns = contextNamespace; // the adjusted current node is the context element+        else+            ns = currentElement().tag().namespace();+        return !NamespaceHtml.equals(ns);+    }+     boolean useCurrentOrForeignInsert(Token token) {         // https://html.spec.whatwg.org/multipage/parsing.html#tree-construction         // If the stack of open elements is emptysrc/main/java/org/jsoup/parser/Tokeniser.javaindex 2a5db368..876a0e9c 100644@@ -31,6 +31,7 @@ final class Tokeniser {         Arrays.sort(notCharRefCharsSorted);     } +    private final TreeBuilder treeBuilder; // consulted for parse state the tokeniser depends on (e.g. CDATA in foreign content)     private final CharacterReader reader; // html input     private final ParseErrorList errors; // errors found while tokenising @@ -57,10 +58,16 @@ final class Tokeniser {         tagPending = startPending  = new Token.StartTag(treeBuilder);         endPending = new Token.EndTag(treeBuilder);         xmlDeclPending = new Token.XmlDecl(treeBuilder);+        this.treeBuilder = treeBuilder;         this.reader = treeBuilder.reader;         this.errors = treeBuilder.parser.getErrors();     } +    /** If a CDATA section is allowed here: in XML, or when the adjusted current node is not in the HTML namespace. */+    boolean allowsCdata() {+        return treeBuilder.allowsCdata();+    }+     Token read() {         while (!isEmitPending) {             state.read(this, reader);src/main/java/org/jsoup/parser/TokeniserState.javaindex e61b8c27..035537e0 100644@@ -944,10 +944,8 @@ enum TokeniserState {                 t.transition(CommentStart);             } else if (r.matchConsumeIgnoreCase("DOCTYPE")) {                 t.transition(Doctype);-            } else if (r.matchConsume("[CDATA[")) {-                // todo: should actually check current namespace, and only non-html allows cdata. until namespace-                // is implemented properly, keep handling as cdata-                //} else if (!t.currentNodeInHtmlNS() && r.matchConsume("[CDATA[")) {+            } else if (r.matches("[CDATA[") && t.allowsCdata() && r.matchConsume("[CDATA[")) {+                // CDATA only in foreign content (or XML); in the HTML namespace, falls through to a bogus comment                 t.createTempBuffer();                 t.transition(CdataSection);             } else {src/main/java/org/jsoup/parser/TreeBuilder.javaindex db7069c7..085de2d7 100644@@ -129,6 +129,14 @@ abstract class TreeBuilder {      abstract boolean process(Token token); +    /**+     Checks if a {@code <![CDATA[} section may be opened at the current position. Default (XML) is always; the HTML+     tree builder only allows CDATA in foreign content.+     */+    boolean allowsCdata() {+        return true;+    }+     boolean processStartTag(String name) {         // these are "virtual" start tags (auto-created by the treebuilder), so not tracking the start position         final Token.StartTag start = this.start;src/test/java/org/jsoup/nodes/ElementTest.javaindex 827fbdb0..9dbb6cd0 100644@@ -3132,7 +3132,7 @@ public class ElementTest {         el.appendChild(cdata);         assertEquals("Some CData", el.data()); -        Document parse = Jsoup.parse("One <![CDATA[Hello]]>");+        Document parse = Jsoup.parse("One <svg><![CDATA[Hello]]></svg>");         assertEquals("Hello", parse.data());     } src/test/java/org/jsoup/parser/HtmlParserTest.javaindex 10f88db2..23234bae 100644@@ -345,26 +345,89 @@ public class HtmlParserTest {     }      @Test public void handlesCdata() {-        // todo: as this is html namespace, should actually treat as bogus comment, not cdata. keep as cdata for now-        String h = "<div id=1><![CDATA[<html>\n <foo><&amp;]]></div>"; // the &amp; in there should remain literal+        // CDATA is only recognized in foreign content (svg, math)+        String h = "<div id=1><svg><![CDATA[<html>\n <foo><&amp;]]></svg></div>"; // the &amp; in there should remain literal         Document doc = Jsoup.parse(h);-        Element div = doc.getElementById("1");-        assertEquals("<html>\n <foo><&amp;", div.text());-        assertEquals(0, div.children().size());-        assertEquals(1, div.childNodeSize()); // no elements, one text node+        Element svg = doc.expectFirst("#1 > svg");+        assertEquals(Parser.NamespaceSvg, svg.tag().namespace());+        assertEquals("<html>\n <foo><&amp;", svg.text());+        assertEquals(0, svg.children().size());+        assertEquals(1, svg.childNodeSize()); // no elements, one text node+        assertTrue(svg.childNode(0) instanceof CDataNode);+    }++    @Test public void handlesCdataInMathml() {+        Document doc = Jsoup.parse("<math><mi>x</mi><![CDATA[<y> & z]]></math>");+        doc.outputSettings().prettyPrint(false);+        Element math = doc.expectFirst("math");+        assertEquals(Parser.NamespaceMathml, math.tag().namespace());+        CDataNode cdata = (CDataNode) math.childNode(1);+        assertEquals("<y> & z", cdata.text());+        assertEquals("<math><mi>x</mi><![CDATA[<y> & z]]></math>", math.outerHtml());+    }++    @Test public void cdataInHtmlNamespaceIsBogusComment() {+        // https://html.spec.whatwg.org/multipage/parsing.html#markup-declaration-open-state - in HTML, <![CDATA[ is+        // a cdata-in-html-content parse error, and starts a bogus comment, which ends at the first >+        Parser parser = Parser.htmlParser().setTrackErrors(10);+        Document doc = Jsoup.parse("<div id=1><![CDATA[<b>One</b>]]></div>", parser);+        doc.outputSettings().prettyPrint(false);+        Element div = doc.expectFirst("#1");++        List<Node> nodes = div.childNodes();+        assertEquals(3, nodes.size());+        Comment comment = (Comment) nodes.get(0);+        assertEquals("[CDATA[<b", comment.getData());+        assertEquals("One", ((TextNode) nodes.get(1)).getWholeText()); // the <b> was consumed by the bogus comment+        assertEquals("]]>", ((TextNode) nodes.get(2)).getWholeText()); // </b> is ignored+        assertEquals(0, div.select("b").size());+        assertEquals(0, doc.select("::cdata").size());+        assertEquals("<div id=\"1\"><!--[CDATA[<b-->One]]&gt;</div>", div.outerHtml());+        assertFalse(parser.getErrors().isEmpty());+    }++    @Test public void cdataInHtmlIntegrationPointIsBogusComment() {+        // foreignObject is an HTML integration point, but its own namespace is svg, so CDATA is recognized directly in it.+        // Once in an html element within it, CDATA is a bogus comment+        Document doc = Jsoup.parse("<svg><foreignObject><![CDATA[One]]><p><![CDATA[Two]]></p></foreignObject><![CDATA[Three]]></svg>");+        Element fo = doc.expectFirst("foreignObject");+        assertTrue(fo.childNode(0) instanceof CDataNode);+        Element p = doc.expectFirst("p");+        assertEquals(Parser.NamespaceHtml, p.tag().namespace());+        Comment comment = (Comment) p.childNode(0);+        assertEquals("[CDATA[Two]]", comment.getData());+        Element svg = doc.expectFirst("svg");+        CDataNode three = (CDataNode) svg.childNode(1);+        assertEquals("Three", three.text());+        Document after = Jsoup.parse("<svg></svg><div>Zero<![CDATA[Four]]></div>");+        after.outputSettings().prettyPrint(false);+        assertEquals("<div>Zero<!--[CDATA[Four]]--></div>", after.expectFirst("div").outerHtml());+    }++    @Test public void cdataInFragmentUsesContextNamespace() {+        Document doc = Jsoup.parse("<div></div><svg></svg>");+        doc.outputSettings().prettyPrint(false);+        Element div = doc.expectFirst("div");+        div.html("<![CDATA[One]]>");+        assertEquals("<div><!--[CDATA[One]]--></div>", div.outerHtml());++        Element svg = doc.expectFirst("svg");+        svg.html("<![CDATA[Two]]>");+        assertTrue(svg.childNode(0) instanceof CDataNode);+        assertEquals("<svg><![CDATA[Two]]></svg>", svg.outerHtml());     }      @Test public void roundTripsCdata() {-        String h = "<div id=1><![CDATA[\n<html>\n <foo><&amp;]]></div>";+        String h = "<svg id=1><![CDATA[\n<html>\n <foo><&amp;]]></svg>";         Document doc = Jsoup.parse(h);-        Element div = doc.getElementById("1");-        assertEquals("<html>\n <foo><&amp;", div.text());-        assertEquals(0, div.children().size());-        assertEquals(1, div.childNodeSize()); // no elements, one text node+        Element svg = doc.getElementById("1");+        assertEquals("<html>\n <foo><&amp;", svg.text());+        assertEquals(0, svg.children().size());+        assertEquals(1, svg.childNodeSize()); // no elements, one text node -        assertEquals("<div id=\"1\"><![CDATA[\n<html>\n <foo><&amp;]]></div>", div.outerHtml());+        assertEquals("<svg id=\"1\"><![CDATA[\n<html>\n <foo><&amp;]]></svg>", svg.outerHtml()); -        CDataNode cdata = (CDataNode) div.textNodes().get(0);+        CDataNode cdata = (CDataNode) svg.textNodes().get(0);         assertEquals("\n<html>\n <foo><&amp;", cdata.text());     } @@ -374,11 +437,11 @@ public class HtmlParserTest {             sb.append("A suitable amount of CData.\n");         }         String cdata = sb.toString();-        String h = "<div><![CDATA[" + cdata + "]]></div>";+        String h = "<svg><![CDATA[" + cdata + "]]></svg>";         Document doc = Jsoup.parse(h);-        Element div = doc.selectFirst("div");+        Element svg = doc.selectFirst("svg"); -        CDataNode node = (CDataNode) div.textNodes().get(0);+        CDataNode node = (CDataNode) svg.textNodes().get(0);         assertEquals(cdata, node.text());     } @@ -399,15 +462,23 @@ public class HtmlParserTest {      @Test public void handlesUnclosedCdataAtEOF() {         // https://github.com/jhy/jsoup/issues/349 would crash, as character reader would try to seek past EOF-        String h = "<![CDATA[]]";+        String h = "<svg><![CDATA[]]";         Document doc = Jsoup.parse(h);-        assertEquals(1, doc.body().childNodeSize());+        Element svg = doc.expectFirst("svg");+        assertEquals(1, svg.childNodeSize());+        assertTrue(svg.childNode(0) instanceof CDataNode);++        // and in html, an unclosed bogus comment at EOF+        Document htmlDoc = Jsoup.parse("<p><![CDATA[]]");+        Element p = htmlDoc.expectFirst("p");+        assertEquals(1, p.childNodeSize());+        assertEquals("[CDATA[]]", ((Comment) p.childNode(0)).getData());     }      @Test public void handleCDataInText() {-        String h = "<p>One <![CDATA[Two <&]]> Three</p>";+        String h = "<svg>One <![CDATA[Two <&]]> Three</svg>";         Document doc = Jsoup.parse(h);-        Element p = doc.selectFirst("p");+        Element p = doc.selectFirst("svg");          List<Node> nodes = p.childNodes();         assertEquals("One ", ((TextNode) nodes.get(0)).getWholeText());@@ -419,9 +490,9 @@ public class HtmlParserTest {     }      @Test public void cdataNodesAreTextNodes() {-        String h = "<p>One <![CDATA[ Two <& ]]> Three</p>";+        String h = "<svg>One <![CDATA[ Two <& ]]> Three</svg>";         Document doc = Jsoup.parse(h);-        Element p = doc.selectFirst("p");+        Element p = doc.selectFirst("svg");          List<TextNode> nodes = p.textNodes();         assertEquals("One ", nodes.get(0).text());src/test/java/org/jsoup/parser/PositionTest.javaindex 847c1b03..f49008e6 100644@@ -305,7 +305,7 @@ class PositionTest {     }      @Test void tracksMarkup() {-        String html = "<!doctype\nhtml>\n<title>jsoup &copy;\n2022</title><body>\n<![CDATA[\n<jsoup>\n]]>";+        String html = "<!doctype\nhtml>\n<title>jsoup &copy;\n2022</title><body>\n<svg><![CDATA[\n<jsoup>\n]]>";         Document doc = Jsoup.parse(html, TrackingHtmlParser);          DocumentType doctype = doc.documentType();@@ -321,9 +321,18 @@ class PositionTest {         assertEquals("3,1:16-3,8:23", title.sourceRange().toString());         assertEquals("3,8:23-4,5:40", titleText.sourceRange().toString()); -        CDataNode cdata = (CDataNode) doc.body().childNode(1);+        CDataNode cdata = (CDataNode) doc.expectFirst("svg").childNode(0);         assertEquals("\n<jsoup>\n", cdata.text());-        assertEquals("5,1:55-7,4:76", cdata.sourceRange().toString());+        assertEquals("5,6:60-7,4:81", cdata.sourceRange().toString());+    }++    @Test void tracksCdataInHtmlAsBogusComment() {+        String html = "<body>\n<![CDATA[\n<jsoup>\n]]>";+        Document doc = Jsoup.parse(html, TrackingHtmlParser);++        Comment comment = (Comment) doc.body().childNode(1);+        assertEquals("[CDATA[\n<jsoup", comment.getData());+        assertEquals("2,1:7-3,8:24", comment.sourceRange().toString());     }      @Test void tracksDataNodes() {src/test/java/org/jsoup/parser/TokeniserTest.javaindex d6846b25..b3c6266d 100644@@ -119,10 +119,10 @@ public class TokeniserTest {             sb.append("Quite a lot of CDATA <><><><>");         } while (sb.length() < BufferSize);         String cdata = sb.toString();-        String html = "<p><![CDATA[" + cdata + "]]></p>";+        String html = "<p><svg><![CDATA[" + cdata + "]]></svg></p>";          Document doc = Jsoup.parse(html);-        Elements els = doc.select("p");+        Elements els = doc.select("p > svg");         assertEquals(1, els.size());         Element el = els.first(); @@ -194,18 +194,19 @@ public class TokeniserTest {     }      @Test public void canParseCdataEndingAtEdgeOfBuffer() {+        String svgStart = "<svg>"; // cdata only recognized in foreign content         String cdataStart = "<![CDATA[";         String cdataEnd = "]]>";-        int bufLen = BufferSize - cdataStart.length() - 1;    // also breaks with -2, but not with -3 or 0+        int bufLen = BufferSize - svgStart.length() - cdataStart.length() - 1;    // also breaks with -2, but not with -3 or 0         char[] cdataContentsArray = new char[bufLen];         Arrays.fill(cdataContentsArray, 'x');         String cdataContents = new String(cdataContentsArray);-        String testMarkup = cdataStart + cdataContents + cdataEnd;+        String testMarkup = svgStart + cdataStart + cdataContents + cdataEnd;         Parser parser = new Parser(new HtmlTreeBuilder());          Document doc = parser.parseInput(testMarkup, ""); -        Node cdataNode = doc.body().childNode(0);+        Node cdataNode = doc.expectFirst("svg").childNode(0);         assertTrue(cdataNode instanceof CDataNode, "Expected CDATA node");         assertEquals(cdataContents, ((CDataNode)cdataNode).text());     } 
Merged fix · PR #+232 −69
CHANGES.mdindex 3ca16dd0..fdd6a17a 100644@@ -11,6 +11,7 @@ * `<noscript>` fallback markup is now parsed into an inspectable DOM subtree in both the document head and body. The fallback acts as a contained parsing island, so malformed markup cannot disrupt the surrounding document structure, while normal HTML tokenization still applies within it. This also improves round-trip serialization. [#2537](https://github.com/jhy/jsoup/pull/2537) * Improved redirect credential handling as a defense-in-depth measure: explicit authorization headers and request cookies are no longer forwarded across origins, reducing exposure through open redirects and aligning with HTTP guidance. Cookies managed by a `CookieStore` continue to follow their configured scope. [#2540](https://github.com/jhy/jsoup/pull/2540) * Elements can now append their outer HTML, including their own tags, directly to an `Appendable` with `Node#outerHtml(Appendable)`, without first creating a `String`. This complements `Element#html(Appendable)`, which appends inner HTML only. [#2532](https://github.com/jhy/jsoup/issues/2532)+* Aligned CDATA tokenization with the HTML spec: CDATA syntax in HTML content is parsed as a bogus comment, while it remains supported in SVG, MathML, and XML. Also improved namespace-aware fragment parsing so SVG and MathML contexts, HTML integration points, and context-sensitive tokenizer states are handled correctly. [#2542](https://github.com/jhy/jsoup/issues/2542)  ### Bug Fixes * Fixed HTML parsing of mixed-case RCDATA end tags after tag-shaped text. For example, `<title><p>Foo</TiTLE>` and `<textarea><img src=x></TeXtArEa>` now keep the tag-shaped content as text instead of promoting it to markup. [#2503](https://github.com/jhy/jsoup/issues/2503)src/main/java/org/jsoup/parser/HtmlTreeBuilder.javaindex 8fc68a43..6c1a6b96 100644@@ -44,7 +44,7 @@ public class HtmlTreeBuilder extends TreeBuilder {     private boolean baseUriSetFromDoc;     private @Nullable Element headElement; // the current head element     private @Nullable FormElement formElement; // the current form element-    private @Nullable Element contextElement; // fragment parse root; name only copy of context. could be null even if fragment parsing+    private @Nullable Element contextElement; // fragment parse root; shallow copy of context, may be null during fragment parsing     ArrayList<Element> formattingElements; // active (open) formatting elements     private ArrayList<HtmlTreeBuilderState> tmplInsertMode; // stack of Template Insertion modes     private @Nullable NoscriptState noscriptState; // active noscript island state@@ -92,30 +92,37 @@ public class HtmlTreeBuilder extends TreeBuilder {          if (context != null) {             final String contextName = context.normalName();-            contextElement = new Element(tagFor(contextName, contextName, defaultNamespace(), settings), baseUri);+            contextElement = new Element(context.tag(), baseUri);+            contextElement.attributes().addAll(context.attributes());             if (context.ownerDocument() != null) // quirks setup:                 doc.quirksMode(context.ownerDocument().quirksMode()); -            // initialise the tokeniser state:+            // initialise the tokeniser state+            Tag contextTag = contextElement.tag();+            boolean htmlContext = NamespaceHtml.equals(contextTag.namespace());+            TokeniserState contextState = contextTag.textState(); // style, xmp, title, textarea, etc; or custom+            if (contextState == null) contextState = TokeniserState.Data;+             switch (contextName) {                 case "script":-                    tokeniser.transition(TokeniserState.ScriptData);+                    if (htmlContext) {+                        contextState = TokeniserState.ScriptData;+                    } else if (NamespaceSvg.equals(contextTag.namespace())) {+                        // svg script enters script data during document parsing, but fragments start in data so markup creates svg children+                        contextState = TokeniserState.Data;+                    }                     break;                 case "plaintext":-                    tokeniser.transition(TokeniserState.PLAINTEXT);+                    if (htmlContext) contextState = TokeniserState.PLAINTEXT;                     break;                 case "template":-                    tokeniser.transition(TokeniserState.Data);-                    pushTemplateMode(HtmlTreeBuilderState.InTemplate);+                    if (htmlContext) {+                        contextState = TokeniserState.Data;+                        pushTemplateMode(HtmlTreeBuilderState.InTemplate);+                    }                     break;-                default:-                    Tag tag = contextElement.tag();-                    TokeniserState textState = tag.textState();-                    if (textState != null)-                        tokeniser.transition(textState); // style, xmp, title, textarea, etc; or custom-                    else-                        tokeniser.transition(TokeniserState.Data);             }+            tokeniser.transition(contextState);             doc.appendChild(contextElement);             push(contextElement);             resetInsertionMode();@@ -131,7 +138,7 @@ public class HtmlTreeBuilder extends TreeBuilder {                 formSearch = formSearch.parent();             } -            if (contextName.equals("noscript")) enterNoscript(contextElement);+            if (htmlContext && contextName.equals("noscript")) enterNoscript(contextElement);         }     } @@ -693,9 +700,7 @@ public class HtmlTreeBuilder extends TreeBuilder {                 if (fragmentParsing)                     node = contextElement;             }-            String name = node != null ? node.normalName() : "";-            if (!NamespaceHtml.equals(node.tag().namespace()))-                continue; // only looking for HTML elements here+            String name = node != null && NamespaceHtml.equals(node.tag().namespace()) ? node.normalName() : "";              switch (name) {                 case "select":src/main/java/org/jsoup/parser/Tokeniser.javaindex 2a5db368..4487b615 100644@@ -33,6 +33,7 @@ final class Tokeniser {      private final CharacterReader reader; // html input     private final ParseErrorList errors; // errors found while tokenising+    private final TreeBuilder treeBuilder; // current tree state for contextual tokenisation      private TokeniserState state = TokeniserState.Data; // current tokenisation state     @Nullable private Token emitPending = null; // the token we are about to emit on next read@@ -53,6 +54,7 @@ final class Tokeniser {     private int markupStartPos, charStartPos = 0; // reader pos at the start of markup / characters. markup updated on state transition, char on token emit.      Tokeniser(TreeBuilder treeBuilder) {+        this.treeBuilder = treeBuilder;         syntax = treeBuilder instanceof XmlTreeBuilder ? Document.OutputSettings.Syntax.xml : Document.OutputSettings.Syntax.html;         tagPending = startPending  = new Token.StartTag(treeBuilder);         endPending = new Token.EndTag(treeBuilder);@@ -260,6 +262,12 @@ final class Tokeniser {         dataBuffer.reset();     } +    /** Test if CDATA sections are allowed at the adjusted current node */+    boolean isCdataAllowed() {+        return syntax == Document.OutputSettings.Syntax.xml+            || !Parser.NamespaceHtml.equals(treeBuilder.currentElement().tag().namespace());+    }+     boolean isAppropriateEndTagToken() {         return lastStartTag != null && tagPending.name().equalsIgnoreCase(lastStartTag);     }src/main/java/org/jsoup/parser/TokeniserState.javaindex e61b8c27..37911b45 100644@@ -945,11 +945,15 @@ enum TokeniserState {             } else if (r.matchConsumeIgnoreCase("DOCTYPE")) {                 t.transition(Doctype);             } else if (r.matchConsume("[CDATA[")) {-                // todo: should actually check current namespace, and only non-html allows cdata. until namespace-                // is implemented properly, keep handling as cdata-                //} else if (!t.currentNodeInHtmlNS() && r.matchConsume("[CDATA[")) {-                t.createTempBuffer();-                t.transition(CdataSection);+                if (t.isCdataAllowed()) {+                    t.createTempBuffer();+                    t.transition(CdataSection);+                } else {+                    t.error(this);+                    t.createBogusCommentPending();+                    t.commentPending.append("[CDATA[");+                    t.transition(BogusComment);+                }             } else {                 if (t.syntax == xml && r.matchesAsciiAlpha()) {                     t.createXmlDeclPending(true);src/test/java/org/jsoup/nodes/ElementTest.javaindex 827fbdb0..229fe210 100644@@ -3132,8 +3132,8 @@ public class ElementTest {         el.appendChild(cdata);         assertEquals("Some CData", el.data()); -        Document parse = Jsoup.parse("One <![CDATA[Hello]]>");-        assertEquals("Hello", parse.data());+        Document xml = Jsoup.parse("One <![CDATA[Hello]]>", "", Parser.xmlParser());+        assertEquals("Hello", xml.data());     }      @Test void datanodesOutputCdataInXhtml() {src/test/java/org/jsoup/parser/HtmlParserTest.javaindex 10f88db2..1ab9de2e 100644@@ -344,27 +344,167 @@ public class HtmlParserTest {         assertEquals("https://example.net/img.jpg", el.absUrl("src"));     } -    @Test public void handlesCdata() {-        // todo: as this is html namespace, should actually treat as bogus comment, not cdata. keep as cdata for now-        String h = "<div id=1><![CDATA[<html>\n <foo><&amp;]]></div>"; // the &amp; in there should remain literal-        Document doc = Jsoup.parse(h);-        Element div = doc.getElementById("1");-        assertEquals("<html>\n <foo><&amp;", div.text());-        assertEquals(0, div.children().size());-        assertEquals(1, div.childNodeSize()); // no elements, one text node+    @Test public void handlesCdataAsBogusCommentInHtml() {+        String h = "<div id=1><![CDATA[<html>\n <foo><&amp;]]></div>";+        Parser parser = Parser.htmlParser().setTrackErrors(1);+        Document doc = Jsoup.parse(h, "", parser);+        Element div = doc.expectFirst("#1");++        Comment comment = assertInstanceOf(Comment.class, div.childNode(0));+        assertEquals("[CDATA[<html", comment.getData()); // closes at the first '>'+        Element foo = div.expectFirst("foo"); // remaining source continues as HTML+        assertEquals("<&]]>", foo.text());++        // reports after the recognized CDATA opener+        assertEquals(1, parser.getErrors().size());+        ParseError error = parser.getErrors().get(0);+        assertEquals(h.indexOf("<html>"), error.getPosition());+        assertEquals("1:20", error.getCursorPos());+    }++    @Test void cdataParsingDependsOnNamespaceAndParser() {+        String htmlInput =+            "<div id=html><![CDATA[Html > tail]]></div>" ++            "<svg><![CDATA[Svg > data]]><foreignObject><![CDATA[Integrated SVG > data]]>" ++            "<p><![CDATA[Foreign HTML > tail]]></p></foreignObject>" ++            "<g><![CDATA[Nested SVG > data]]></g></svg>" ++            "<math><![CDATA[Math > data]]></math>";+        Document html = Jsoup.parse(htmlInput);++        // in the HTML namespace this is a bogus comment, which closes at the first '>'+        Element div = html.expectFirst("#html");+        Comment comment = assertInstanceOf(Comment.class, div.childNode(0));+        assertEquals("[CDATA[Html ", comment.getData());+        assertEquals(" tail]]>", assertInstanceOf(TextNode.class, div.childNode(1)).getWholeText());++        // foreign content recognizes CDATA, including after returning from an HTML integration point+        Element svg = html.expectFirst("svg");+        assertEquals("Svg > data", assertInstanceOf(CDataNode.class, svg.childNode(0)).getWholeText());+        Element nestedSvg = svg.expectFirst("g");+        assertEquals(Parser.NamespaceSvg, nestedSvg.tag().namespace());+        assertEquals("Nested SVG > data",+            assertInstanceOf(CDataNode.class, nestedSvg.childNode(0)).getWholeText());+        Element math = html.expectFirst("math");+        assertEquals(Parser.NamespaceMathml, math.tag().namespace());+        assertEquals("Math > data", assertInstanceOf(CDataNode.class, math.childNode(0)).getWholeText());++        // direct content uses the SVG namespace, while HTML children return to bogus comment parsing+        Element foreignObject = svg.expectFirst("foreignObject");+        assertEquals("Integrated SVG > data",+            assertInstanceOf(CDataNode.class, foreignObject.childNode(0)).getWholeText());+        Element foreignHtml = foreignObject.expectFirst("p");+        assertEquals(Parser.NamespaceHtml, foreignHtml.tag().namespace());+        Comment foreignComment = assertInstanceOf(Comment.class, foreignHtml.childNode(0));+        assertEquals("[CDATA[Foreign HTML ", foreignComment.getData());+        assertEquals(" tail]]>",+            assertInstanceOf(TextNode.class, foreignHtml.childNode(1)).getWholeText());++        // the XML parser recognizes CDATA even when the element is in the HTML namespace+        String xmlInput = String.format(+            "<p xmlns='%s'><![CDATA[Nine > Ten]]></p>", Parser.NamespaceHtml);+        Document xml = Jsoup.parse(xmlInput, "", Parser.xmlParser());+        assertEquals(Parser.NamespaceHtml, xml.expectFirst("p").tag().namespace());+        assertEquals("Nine > Ten",+            assertInstanceOf(CDataNode.class, xml.expectFirst("p").childNode(0)).getWholeText());+    }++    @Test void fragmentParsingUsesContextNamespace() {+        String fragment = "<![CDATA[One > Two]]><child />";++        Element html = Jsoup.parse("<div></div>").expectFirst("div");+        html.html(fragment);+        Comment comment = assertInstanceOf(Comment.class, html.childNode(0));+        assertEquals("[CDATA[One ", comment.getData());+        assertEquals(" Two]]>", assertInstanceOf(TextNode.class, html.childNode(1)).getWholeText());+        assertEquals(Parser.NamespaceHtml, html.expectFirst("child").tag().namespace());+        assertEquals(3, html.childNodeSize());++        Element svg = Jsoup.parse("<svg></svg>").expectFirst("svg");+        svg.html(fragment);+        assertEquals("One > Two",+            assertInstanceOf(CDataNode.class, svg.childNode(0)).getWholeText());+        assertEquals(Parser.NamespaceSvg, svg.expectFirst("child").tag().namespace());+        assertEquals(2, svg.childNodeSize());++        Element math = Jsoup.parse("<math></math>").expectFirst("math");+        math.html(fragment);+        assertEquals("One > Two",+            assertInstanceOf(CDataNode.class, math.childNode(0)).getWholeText());+        assertEquals(Parser.NamespaceMathml, math.expectFirst("child").tag().namespace());+        assertEquals(2, math.childNodeSize());++        Document xmlDoc = Jsoup.parse("<root />", "", Parser.xmlParser());+        Element xml = xmlDoc.expectFirst("root");+        xml.html(fragment);+        assertEquals("One > Two",+            assertInstanceOf(CDataNode.class, xml.childNode(0)).getWholeText());+        assertEquals(Parser.NamespaceXml, xml.expectFirst("child").tag().namespace());+        assertEquals(2, xml.childNodeSize());+    }++    @Test void fragmentParsingPreservesHtmlIntegrationPoints() {+        Document doc = Jsoup.parse(+            "<svg><foreignObject></foreignObject></svg>" ++            "<math><annotation-xml encoding=text/html></annotation-xml></math>");++        Element foreignObject = doc.expectFirst("foreignObject");+        foreignObject.html("<section />");+        assertEquals(1, foreignObject.childNodeSize());+        assertEquals(Parser.NamespaceHtml, foreignObject.expectFirst("section").tag().namespace());++        Element annotation = doc.expectFirst("annotation-xml");+        annotation.html("<section />");+        assertEquals(1, annotation.childNodeSize());+        assertEquals(Parser.NamespaceHtml, annotation.expectFirst("section").tag().namespace());+    }++    @Test void fragmentTokeniserStateUsesContextNamespace() {+        Document doc = Jsoup.parse(+            "<svg><noscript id=n></noscript><plaintext id=p></plaintext><script id=ss></script></svg>" ++            "<math><script id=s></script></math>");++        Element svgNoscript = doc.expectFirst("#n");+        svgNoscript.html("<child></child>");+        Element noscriptChild = assertInstanceOf(Element.class, svgNoscript.childNode(0));+        assertEquals(Parser.NamespaceSvg, noscriptChild.tag().namespace());+        assertEquals(1, svgNoscript.childNodeSize());++        Element svgPlaintext = doc.expectFirst("#p");+        svgPlaintext.html("<child></child>");+        Element plaintextChild = assertInstanceOf(Element.class, svgPlaintext.childNode(0));+        assertEquals(Parser.NamespaceSvg, plaintextChild.tag().namespace());+        assertEquals(1, svgPlaintext.childNodeSize());++        Element svgScript = doc.expectFirst("#ss");+        svgScript.html("<child></child>");+        Element svgScriptChild = assertInstanceOf(Element.class, svgScript.childNode(0));+        assertEquals(Parser.NamespaceSvg, svgScriptChild.tag().namespace());+        assertEquals(1, svgScript.childNodeSize());++        Element mathScript = doc.expectFirst("#s");+        mathScript.html("<child></child>");+        Element scriptChild = assertInstanceOf(Element.class, mathScript.childNode(0));+        assertEquals(Parser.NamespaceMathml, scriptChild.tag().namespace());+        assertEquals(1, mathScript.childNodeSize());++        Element htmlScript = Jsoup.parse("<script></script>").expectFirst("script");+        htmlScript.html("<child></child>");+        DataNode scriptData = assertInstanceOf(DataNode.class, htmlScript.childNode(0));+        assertEquals("<child></child>", scriptData.getWholeData());+        assertEquals(1, htmlScript.childNodeSize());     }      @Test public void roundTripsCdata() {-        String h = "<div id=1><![CDATA[\n<html>\n <foo><&amp;]]></div>";+        String h = "<svg id=1><![CDATA[\n<html>\n <foo><&amp;]]></svg>";         Document doc = Jsoup.parse(h);-        Element div = doc.getElementById("1");-        assertEquals("<html>\n <foo><&amp;", div.text());-        assertEquals(0, div.children().size());-        assertEquals(1, div.childNodeSize()); // no elements, one text node+        Element svg = doc.expectFirst("#1");+        assertEquals("<html>\n <foo><&amp;", svg.text());+        assertEquals(0, svg.children().size());+        assertEquals(1, svg.childNodeSize()); // no elements, one text node -        assertEquals("<div id=\"1\"><![CDATA[\n<html>\n <foo><&amp;]]></div>", div.outerHtml());+        assertEquals("<svg id=\"1\"><![CDATA[\n<html>\n <foo><&amp;]]></svg>", svg.outerHtml()); -        CDataNode cdata = (CDataNode) div.textNodes().get(0);+        CDataNode cdata = assertInstanceOf(CDataNode.class, svg.textNodes().get(0));         assertEquals("\n<html>\n <foo><&amp;", cdata.text());     } @@ -374,11 +514,11 @@ public class HtmlParserTest {             sb.append("A suitable amount of CData.\n");         }         String cdata = sb.toString();-        String h = "<div><![CDATA[" + cdata + "]]></div>";+        String h = "<svg><![CDATA[" + cdata + "]]></svg>";         Document doc = Jsoup.parse(h);-        Element div = doc.selectFirst("div");+        Element svg = doc.expectFirst("svg"); -        CDataNode node = (CDataNode) div.textNodes().get(0);+        CDataNode node = assertInstanceOf(CDataNode.class, svg.childNode(0));         assertEquals(cdata, node.text());     } @@ -397,33 +537,34 @@ public class HtmlParserTest {         // see - not a cdata node, because in script. contrast with XmlTreeBuilder - will be cdata.     } -    @Test public void handlesUnclosedCdataAtEOF() {+    @Test public void handlesUnclosedBogusCommentAtEOF() {         // https://github.com/jhy/jsoup/issues/349 would crash, as character reader would try to seek past EOF         String h = "<![CDATA[]]";         Document doc = Jsoup.parse(h);-        assertEquals(1, doc.body().childNodeSize());+        Comment comment = assertInstanceOf(Comment.class, doc.childNode(0));+        assertEquals("[CDATA[]]", comment.getData());     }      @Test public void handleCDataInText() {-        String h = "<p>One <![CDATA[Two <&]]> Three</p>";+        String h = "<svg><text>One <![CDATA[Two <&]]> Three</text></svg>";         Document doc = Jsoup.parse(h);-        Element p = doc.selectFirst("p");+        Element text = doc.expectFirst("text"); -        List<Node> nodes = p.childNodes();+        List<Node> nodes = text.childNodes();         assertEquals("One ", ((TextNode) nodes.get(0)).getWholeText());         assertEquals("Two <&", ((TextNode) nodes.get(1)).getWholeText());         assertEquals("Two <&", ((CDataNode) nodes.get(1)).getWholeText());         assertEquals(" Three", ((TextNode) nodes.get(2)).getWholeText()); -        assertEquals(h, p.outerHtml());+        assertEquals("<text>One <![CDATA[Two <&]]> Three</text>", text.outerHtml());     }      @Test public void cdataNodesAreTextNodes() {-        String h = "<p>One <![CDATA[ Two <& ]]> Three</p>";+        String h = "<svg><text>One <![CDATA[ Two <& ]]> Three</text></svg>";         Document doc = Jsoup.parse(h);-        Element p = doc.selectFirst("p");+        Element text = doc.expectFirst("text"); -        List<TextNode> nodes = p.textNodes();+        List<TextNode> nodes = text.textNodes();         assertEquals("One ", nodes.get(0).text());         assertEquals(" Two <& ", nodes.get(1).text());         assertEquals(" Three", nodes.get(2).text());@@ -2361,13 +2502,19 @@ public class HtmlParserTest {         Tag dataTag = new Tag("data", Parser.NamespaceSvg);         dataTag.set(Tag.Data);         TagSet tagSet = TagSet.HtmlTagSet.add(dataTag);-        String html = "<svg><data>a < b</data></svg>";+        String html = "<svg><data>a < b</data><data id=fragment></data></svg>";         Document doc = Jsoup.parse(html, Parser.htmlParser().tagSet(tagSet));         Element data = doc.expectFirst("data");         assertEquals(Parser.NamespaceSvg, data.tag().namespace);         assertEquals("", data.text());         assertEquals("a < b", data.data());         assertEquals("<data>a < b</data>", data.outerHtml());++        // custom foreign data tags retain their configured fragment state+        Element fragment = doc.expectFirst("#fragment");+        fragment.html("<child></child>");+        assertEquals("<child></child>", fragment.data());+        assertInstanceOf(DataNode.class, fragment.childNode(0));     }      @Test void dropsNullsFromBody() {src/test/java/org/jsoup/parser/PositionTest.javaindex 847c1b03..3883f7e8 100644@@ -5,7 +5,6 @@ import org.jsoup.TextUtil; import org.jsoup.integration.TestServer; import org.jsoup.nodes.Attribute; import org.jsoup.nodes.Attributes;-import org.jsoup.nodes.CDataNode; import org.jsoup.nodes.Comment; import org.jsoup.nodes.DataNode; import org.jsoup.nodes.Document;@@ -321,9 +320,13 @@ class PositionTest {         assertEquals("3,1:16-3,8:23", title.sourceRange().toString());         assertEquals("3,8:23-4,5:40", titleText.sourceRange().toString()); -        CDataNode cdata = (CDataNode) doc.body().childNode(1);-        assertEquals("\n<jsoup>\n", cdata.text());-        assertEquals("5,1:55-7,4:76", cdata.sourceRange().toString());+        Comment comment = (Comment) doc.body().childNode(1);+        assertEquals("[CDATA[\n<jsoup", comment.getData());+        assertEquals("5,1:55-6,8:72", comment.sourceRange().toString());++        TextNode commentTail = (TextNode) doc.body().childNode(2);+        assertEquals("\n]]>", commentTail.getWholeText());+        assertEquals("6,8:72-7,4:76", commentTail.sourceRange().toString());     }      @Test void tracksDataNodes() {src/test/java/org/jsoup/parser/TokeniserTest.javaindex d6846b25..67c91ded 100644@@ -119,15 +119,11 @@ public class TokeniserTest {             sb.append("Quite a lot of CDATA <><><><>");         } while (sb.length() < BufferSize);         String cdata = sb.toString();-        String html = "<p><![CDATA[" + cdata + "]]></p>";+        String xml = "<p><![CDATA[" + cdata + "]]></p>"; -        Document doc = Jsoup.parse(html);-        Elements els = doc.select("p");-        assertEquals(1, els.size());-        Element el = els.first();--        assertNotNull(el);-        TextNode child = (TextNode) el.childNode(0);+        Document doc = Jsoup.parse(xml, "", Parser.xmlParser());+        Element el = doc.expectFirst("p");+        CDataNode child = assertInstanceOf(CDataNode.class, el.childNode(0));         assertEquals(cdata, el.text());         assertEquals(cdata, child.getWholeText());     }@@ -201,13 +197,12 @@ public class TokeniserTest {         Arrays.fill(cdataContentsArray, 'x');         String cdataContents = new String(cdataContentsArray);         String testMarkup = cdataStart + cdataContents + cdataEnd;-        Parser parser = new Parser(new HtmlTreeBuilder());+        Parser parser = Parser.xmlParser();          Document doc = parser.parseInput(testMarkup, ""); -        Node cdataNode = doc.body().childNode(0);-        assertTrue(cdataNode instanceof CDataNode, "Expected CDATA node");-        assertEquals(cdataContents, ((CDataNode)cdataNode).text());+        CDataNode cdataNode = assertInstanceOf(CDataNode.class, doc.childNode(0));+        assertEquals(cdataContents, cdataNode.text());     }      @Test void tokenDataToString() {