Backtestsjsouprun-20261006-181938#2593
jsoup bodyStream() silently truncates responses at Integer.MAX_VALUE despite maxBodySize(0)
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches fix the root cause the same way. `SimpleBufferedInput` used `Integer.MAX_VALUE` as a stand-in for 'unlimited' and decremented it, so reads stopped after 2,147,483,647 bytes. Each patch replaces the sentinel with an explicit `capped` flag and an `uncap()` method, and applies it everywhere the sentinel was used (`read`, `reset`, `max`). Patch A stops there. It is small and easy to verify. It covers the issue's scenario, because `bodyStream()` reads through `buff`, which now starts uncapped. Its unit tests use a virtual stream larger than `Integer.MAX_VALUE` and would fail before the change. They cover a fresh `bodyStream()`, the cap-then-uncap path via `max()`, and `mark`/`reset`. One gap: `readPos` can overflow on the direct-read path, but that only affects progress reporting, not the bytes delivered. Patch B also fixes the bug, adds `readPos` saturation, and adds a good integration test with a gzip body that expands past 2GB. However, it bundles in a major behavior change. `bodyStream()` now goes through the `ControllableInputStream`, so it enforces `maxBodySize` and the timeout. Because the default max body size is 2MB, existing users who stream large downloads without `maxBodySize(0)` would silently get only 2MB. The patch also adds a new public API (`isTruncated`), single-byte lookahead probing, and a `skip` override. It rewrites existing tests (`remainingAfterFirstRead`, `noLimitAfterFirstRead`) to match the new semantics, which confirms the change breaks prior behavior. This extra scope is not neutral: it adds real regression risk and makes the actual fix harder to see. Since both patches are equally correct on the reported issue and A's tests target it directly, A is the better fix.
The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.
Solve
8m 46s
Wall-clock time in the container
Full test suite
No new failures
2,223 passed · 0 skipped · 2,223 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
117vs 436 merged
Changed lines, ours vs merged
The issue
View on GitHubWhen using Connection.maxBodySize(0) together with Response.bodyStream() to download a large binary response, the response appears to be silently truncated at exactly Integer.MAX_VALUE (2,147,483,647) bytes.
I understand that jsoup is primarily an HTML parser rather than a dedicated large-file download tool. However, the documentation for maxBodySize(0) explicitly describes 0 as:
an infinite amount (bounded only by your patience and the memory available on your machine)
This makes the observed behaviour somewhat unexpected. More importantly, the truncation is silent: the InputStream reaches EOF normally, so the calling application has no indication that the response was incomplete.
Steps to reproduce
We use the Blender example.
Compare with command line wget
wget "https://download.blender.org/demo/movies/Blender_reel_2013.mov"
--2026-09-06 15:44:16-- https://download.blender.org/demo/movies/Blender_reel_2013.mov
Resolving download.blender.org (download.blender.org)... 104.20.41.146, 172.66.172.236, 2606:4700:10::6814:2992, ...
Connecting to download.blender.org (download.blender.org)|104.20.41.146|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2516125285 (2,3G) [video/quicktime]
Expected file size is: 2516125285
String url = "https://download.blender.org/demo/movies/Blender_reel_2013.mov";
Connection con = Jsoup.connect(url.toString())
.timeout(10800000) // 3 hours
.method(Method.GET)
.ignoreContentType(true)
.maxBodySize(0)
.followRedirects(false)
.ignoreHttpErrors(true);
Response res = con.execute();
LOG.info("Content-Length: " + res.header("Content-Length"));
LOG.info("Content-Type: " + res.contentType());
try (InputStream in = res.bodyStream();
OutputStream out = new FileOutputStream("/home/test/file.mov")) {
byte[] buffer = new byte[1024 * 1024]; // 1 MB
int bytesRead;
while ((bytesRead = in.read(buffer)) != -1) {
out.write(buffer, 0, bytesRead);
}
}
Validate the resulting file on disk -> 2,0 GiB (2.147.483.647)
The two patches
src/main/java/org/jsoup/internal/ControllableInputStream.javaindex 7deb4ef2..262665e7 100644@@ -84,7 +84,8 @@ public class ControllableInputStream extends FilterInputStream { if (capped && len > remaining) len = remaining; // don't read more than desired, even if available- buff.capRemaining(capped ? remaining : Integer.MAX_VALUE);+ if (capped) buff.capRemaining(remaining);+ else buff.uncap(); while (true) { // loop trying to read until we get some data or hit the overall timeout, if we have one if (expired())@@ -160,7 +161,7 @@ public class ControllableInputStream extends FilterInputStream { buff.capRemaining(remaining); } else { remaining = 0;- buff.capRemaining(Integer.MAX_VALUE);+ buff.uncap(); } readPos = markPos; // readPos is used for progress emits markPos = -1;@@ -197,7 +198,8 @@ public class ControllableInputStream extends FilterInputStream { remaining += newMax - maxSize; // update remaining to reflect the difference in the new maxsize if (remaining < 0) remaining = 0; maxSize = newMax;- buff.capRemaining(newMax == 0 ? Integer.MAX_VALUE : remaining);+ if (newMax == 0) buff.uncap();+ else buff.capRemaining(remaining); } public void allowClose(boolean allowClose) {src/main/java/org/jsoup/internal/SimpleBufferedInput.javaindex daeeaed9..abbdabd1 100644@@ -17,7 +17,8 @@ import static org.jsoup.internal.SharedConstants.DefaultBufferSize; class SimpleBufferedInput extends FilterInputStream { static final int BufferSize = DefaultBufferSize; static final SoftPool<byte[]> BufferPool = new SoftPool<>(() -> new byte[BufferSize]);- private int capRemaining = Integer.MAX_VALUE; // how many bytes we are allowed to pull from the underlying stream+ private boolean capped = false; // if false, there is no limit on how many bytes we may pull from the underlying stream+ private int capRemaining = 0; // when capped, how many bytes we are allowed to pull from the underlying stream private byte @Nullable [] byteBuf; // the byte buffer; recycled via SoftPool. Created in fill if required private int bufPos;@@ -73,24 +74,24 @@ class SimpleBufferedInput extends FilterInputStream { compact(); bufLength = bufPos;- int toRead = Math.min(byteBuf.length - bufPos, capRemaining);+ int toRead = allowedRead(byteBuf.length - bufPos); if (toRead <= 0) return; int read = in.read(byteBuf, bufPos, toRead); if (read > 0) { bufLength = read + bufPos;- capRemaining -= read;- while (byteBuf.length - bufLength > 0 && capRemaining > 0) { // read in more if we have space, without blocking+ consumeCap(read);+ while (byteBuf.length - bufLength > 0 && (!capped || capRemaining > 0)) { // read in more if we have space, without blocking try { if (in.available() < 1) break; } catch (IOException e) { break; // available() is advisory; keep the bytes we've already buffered }- toRead = Math.min(byteBuf.length - bufLength, capRemaining);+ toRead = allowedRead(byteBuf.length - bufLength); if (toRead <= 0) break; read = in.read(byteBuf, bufLength, toRead); if (read <= 0) break; bufLength += read;- capRemaining -= read;+ consumeCap(read); } } if (read == -1) inReadFully = true;@@ -123,10 +124,26 @@ class SimpleBufferedInput extends FilterInputStream { return inReadFully ? 0 : in.available(); } + /** Limit how many more bytes may be pulled from the underlying stream. */ void capRemaining(int newRemaining) {+ capped = true; capRemaining = Math.max(0, newRemaining); } + /** Remove any limit on how many bytes may be pulled from the underlying stream (unbounded; not a finite budget). */+ void uncap() {+ capped = false;+ capRemaining = 0;+ }++ private int allowedRead(int space) {+ return capped ? Math.min(space, capRemaining) : space;+ }++ private void consumeCap(int read) {+ if (capped) capRemaining -= read;+ }+ void setMark() { bufMark = bufPos; }src/test/java/org/jsoup/internal/ControllableInputStreamTest.javaindex fc90fe85..edc63586 100644@@ -61,6 +61,86 @@ class ControllableInputStreamTest { in.close(); } + // a body larger than Integer.MAX_VALUE, which an unlimited (maxSize 0) stream must deliver in full+ private static final long OverIntMax = (long) Integer.MAX_VALUE + 10_000_000L;++ @Test+ void unlimitedInputStreamReadsPastIntMax() throws IOException {+ // the Response.bodyStream() path, including a temporary cap then uncap via max() as DataUtil does+ ControllableInputStream in = ControllableInputStream.wrap(new LargeInputStream(OverIntMax), 0);+ in.max(1024);+ byte[] buf = new byte[1024 * 1024];+ int first = in.read(buf);+ assertEquals(1024, first);+ in.max(0);++ long total = first;+ try (InputStream body = in.inputStream()) {+ int read;+ while ((read = body.read(buf)) != -1) total += read;+ }+ assertEquals(OverIntMax, total, "unlimited stream must not be truncated at Integer.MAX_VALUE");+ }++ @Test+ void unlimitedFreshInputStreamReadsPastIntMax() throws IOException {+ // the Response.bodyStream() path with no prior reads or max() calls; relies on the default (uncapped) buffer state+ ControllableInputStream in = ControllableInputStream.wrap(new LargeInputStream(OverIntMax), 0);+ byte[] buf = new byte[1024 * 1024];+ long total = 0;+ try (InputStream body = in.inputStream()) {+ int read;+ while ((read = body.read(buf)) != -1) total += read;+ }+ assertEquals(OverIntMax, total, "unlimited stream must not be truncated at Integer.MAX_VALUE");+ }++ @Test+ void unlimitedControllableReadPastIntMaxAfterReset() throws IOException {+ // the direct read path, including mark / reset (which re-applies the unlimited state)+ ControllableInputStream in = ControllableInputStream.wrap(new LargeInputStream(OverIntMax), 0);+ in.mark(1024);+ byte[] buf = new byte[1024 * 1024];+ assertEquals(1024, in.read(buf, 0, 1024));+ in.reset();++ long total = 0;+ int read;+ while ((read = in.read(buf)) != -1) total += read;+ assertEquals(OverIntMax, total, "unlimited stream must not be truncated at Integer.MAX_VALUE");+ assertTrue(in.baseReadFully());+ in.close();+ }++ /** Produces {@code size} bytes (contents unspecified) without allocating them. */+ private static final class LargeInputStream extends InputStream {+ private long remaining;++ LargeInputStream(long size) {+ remaining = size;+ }++ @Override+ public int read() {+ if (remaining <= 0) return -1;+ remaining--;+ return 0;+ }++ @Override+ public int read(byte[] b, int off, int len) {+ if (remaining <= 0) return -1;+ int n = (int) Math.min(len, remaining);+ remaining -= n;+ return n;+ }++ @Override+ public int available() {+ return (int) Math.min(remaining, Integer.MAX_VALUE);+ }+ }+ private static final class CountingInputStream extends FilterInputStream { int count = 0;
CHANGES.mdindex 8b245d86..14e8f85b 100644@@ -4,10 +4,12 @@ ### Changes * Removed APIs previously deprecated and scheduled for removal in 1.24.1. [#2597](https://github.com/jhy/jsoup/pull/2597)+* `Response.bodyStream()` now observes the configured max body size and request timeout, consistent with the other response body methods. The default max size is 2 MB. You can configure that with `maxBodySize(0)` before executing the request to disable the cap. `Response.isTruncated()` reports when the decoded response content exceeded the configured limit. [#2598](https://github.com/jhy/jsoup/pull/2598) ### Bug Fixes * Standardized parser normalization of tag and attribute names so that HTML comparisons use ASCII-only case folding, and accepted control characters are preserved, aligning name handling to the HTML and XML specs. [#2594](https://github.com/jhy/jsoup/issues/2594) * Named character references without a semicolon before `-` or `_` now decode correctly in attribute values, matching HTML and browser behavior (e.g., `©-` becomes `©-`). [#2588](https://github.com/jhy/jsoup/issues/2588)+* Support downloads > 2GB via `Response.bodyStream()` when `maxBodySize(0)` is configured. [#2593](https://github.com/jhy/jsoup/issues/2593) ## 1.23.2 (2026-Aug-26) src/main/java/org/jsoup/Connection.javaindex 62e5ec8e..450d7b72 100644@@ -1005,13 +1005,24 @@ public interface Connection { throw new UnsupportedOperationException(); } + /**+ Tests whether the response body was truncated because it exceeded the configured {@link Request#maxBodySize()}.+ <p>The result is authoritative once the body has been consumed by {@link #parse()}, {@link #readFully()}, or+ {@link #bodyStream()} through end of stream. Before then, a {@code false} result is provisional.</p>+ @return true if decoded response content was omitted+ @since 1.24.1+ */+ default boolean isTruncated() {+ return false;+ }+ /** Get the body of the response as a (buffered) InputStream. You should close the input stream when you're done with it. <p>Other body methods (like readFully, body, parse, etc) will generally not work in conjunction with this method, as it consumes the InputStream.</p>- <p>Any configured max size or maximum read timeout applied to the connection will not be applied to this stream,- unless {@link #readFully()} is called prior.</p>+ <p>The configured maximum body size and request timeout apply to this stream. Set {@link #maxBodySize(int)} to+ {@code 0} for an unlimited body size.</p> <p>This method is useful for writing large responses to disk, without buffering them completely into memory first.</p> @return the response body input streamsrc/main/java/org/jsoup/Progress.javaindex f36c5b16..c93e5667 100644@@ -5,7 +5,8 @@ package org.jsoup; public interface Progress<ProgressContext> { /** Called to report progress. Note that this will be executed by the same thread that is doing the work, so either- don't take to long, or hand it off to another thread.+ don't take too long, or hand it off to another thread.+ <p>Byte counts saturate at {@link Integer#MAX_VALUE} for responses > ~ 2GB.</p> @param processed the number of bytes processed so far. @param total the total number of expected bytes, or -1 if unknown. @param percent the percentage of completion, 0.0..100.0. If the expected total is unknown, % will remain at zerosrc/main/java/org/jsoup/helper/HttpConnection.javaindex 8d054658..292e5076 100644@@ -843,6 +843,7 @@ public class HttpConnection implements Connection { private @Nullable String charset; @Nullable String contentType; int contentLength;+ private boolean truncated; private boolean executed = false; private boolean inputStreamRead = false; private int numRedirects = 0;@@ -984,6 +985,7 @@ public class HttpConnection implements Connection { stream, DefaultBufferSize, req.maxBodySize()) .timeout(startTime, req.timeout()); + // todo: for compressed responses, count transport progress before decoding so processed bytes and Content-Length use the same representation if (req.responseProgress != null) // set response progress listener res.bodyStream.onProgress(res.contentLength, req.responseProgress, res); } else {@@ -1048,6 +1050,7 @@ public class HttpConnection implements Connection { @Override public Document parse() throws IOException { ControllableInputStream stream = prepareParse(); Document doc = DataUtil.parseInputStream(stream, charset, url.toExternalForm(), req.parser());+ truncated |= stream.checkTruncated(); doc.connection(new HttpConnection(req, this)); // because we're static, don't have the connection obj. // todo - maybe hold in the req? charset = doc.outputSettings().charset().name(); // update charset from meta-equiv, possibly safeClose();@@ -1080,6 +1083,7 @@ public class HttpConnection implements Connection { Validate.isFalse(inputStreamRead, "Request has already been read (with .parse())"); try { byteData = DataUtil.readToByteBuffer(bodyStream, req.maxBodySize());+ truncated |= bodyStream.checkTruncated(); } finally { inputStreamRead = true; safeClose();@@ -1153,12 +1157,18 @@ public class HttpConnection implements Connection { return bodyStream.inputStream(); } + @Override+ public boolean isTruncated() {+ return truncated || bodyStream != null && bodyStream.isTruncated();+ }+ /** * Call on completion of stream read, to close the body (or error) stream. The connection.disconnect allows * keep-alives to work (as the underlying connection is actually held open, despite the name). */ private void safeClose() { if (bodyStream != null) {+ truncated |= bodyStream.isTruncated(); try { bodyStream.close(); } catch (IOException e) {src/main/java/org/jsoup/internal/ControllableInputStream.javaindex 7deb4ef2..ff669317 100644@@ -26,7 +26,9 @@ public class ControllableInputStream extends FilterInputStream { private int remaining; // how many bytes may still be returned to caller under the current cap private int markPos; // logical readPos snapshot for InputStream.mark/reset (not a buffer cursor) private boolean interrupted; // true if Thread.interrupted() was detected, used to latch interrupted state+ private boolean truncated; // true if there is content beyond the body cap private boolean allowClose = true; // for cases where we want to re-read the input, can ignore .close() from the parser+ private final byte[] singleByte = new byte[1]; // avoids allocation when routing single-byte reads through controls // if we are tracking progress, will have the expected content length and progress callback state private @Nullable ProgressState<?> progress;@@ -69,22 +71,35 @@ public class ControllableInputStream extends FilterInputStream { return wrap(in, maxSize); } + @Override+ public int read() throws IOException {+ int read = read(singleByte, 0, 1);+ return read == -1 ? -1 : singleByte[0] & 0xff;+ }+ @Override public int read(byte[] b, int off, int len) throws IOException { if (readPos == 0) emitProgress(); // emits a progress boolean capped = maxSize != 0;- if (interrupted || capped && remaining <= 0)+ if (interrupted) return -1; if (Thread.currentThread().isInterrupted()) { // interrupted latches, because parse() may call twice interrupted = true; return -1; }+ if (capped && remaining <= 0) {+ if (checkTruncated()) return -1;+ contentLength = readPos;+ emitProgress();+ return -1;+ } if (capped && len > remaining) len = remaining; // don't read more than desired, even if available- buff.capRemaining(capped ? remaining : Integer.MAX_VALUE);+ if (capped) buff.capRemaining(remaining);+ else buff.uncap(); while (true) { // loop trying to read until we get some data or hit the overall timeout, if we have one if (expired())@@ -98,7 +113,8 @@ public class ControllableInputStream extends FilterInputStream { if (capped && read > 0) { remaining -= read; // track bytes returned to the caller }- readPos += read;+ // todo: use long progress values in the public API; saturate until that is available+ readPos = read > Integer.MAX_VALUE - readPos ? Integer.MAX_VALUE : readPos + read; } emitProgress(); return read;@@ -109,6 +125,25 @@ public class ControllableInputStream extends FilterInputStream { } } + @Override+ public long skip(long requested) throws IOException {+ // implemented here so our cap accounting aligns+ if (requested <= 0) return 0;++ byte[] skipBuffer = SimpleBufferedInput.BufferPool.borrow();+ long skipped = 0;+ try {+ while (skipped < requested) {+ int read = read(skipBuffer, 0, (int) Math.min(requested - skipped, skipBuffer.length));+ if (read == -1) break;+ skipped += read;+ }+ return skipped;+ } finally {+ SimpleBufferedInput.BufferPool.release(skipBuffer);+ }+ }+ @Override public boolean markSupported() { return true;@@ -155,12 +190,13 @@ public class ControllableInputStream extends FilterInputStream { if (markPos < 0) throw new IOException("Resetting to invalid mark"); buff.rewindToMark(); buff.clearMark();+ truncated = false; if (maxSize != 0) { remaining = maxSize - markPos; buff.capRemaining(remaining); } else { remaining = 0;- buff.capRemaining(Integer.MAX_VALUE);+ buff.uncap(); } readPos = markPos; // readPos is used for progress emits markPos = -1;@@ -196,8 +232,33 @@ public class ControllableInputStream extends FilterInputStream { public void max(int newMax) { remaining += newMax - maxSize; // update remaining to reflect the difference in the new maxsize if (remaining < 0) remaining = 0;+ if (newMax == 0 || newMax > maxSize) truncated = false; maxSize = newMax;- buff.capRemaining(newMax == 0 ? Integer.MAX_VALUE : remaining);+ if (newMax == 0) buff.uncap();+ else buff.capRemaining(remaining);+ }++ /**+ Check if content remains beyond the configured cap.+ */+ public boolean checkTruncated() throws IOException {+ while (!truncated && maxSize != 0 && remaining <= 0) {+ if (expired()) throw new SocketTimeoutException("Read timeout");+ try {+ truncated = buff.hasMore();+ break;+ } catch (SocketTimeoutException e) {+ if (expired() || timeout == 0) throw e;+ }+ }+ return truncated;+ }++ /**+ Returns whether content was found beyond the configured cap.+ */+ public boolean isTruncated() {+ return truncated; } public void allowClose(boolean allowClose) {@@ -242,7 +303,7 @@ public class ControllableInputStream extends FilterInputStream { public BufferedInputStream inputStream() { // called via HttpConnection.Response.bodyStream(), needs an OG BufferedInputStream- return new BufferedInputStream(buff);+ return new BufferedInputStream(this); // this, not buff, so that return is constrained } private static class ProgressState<ProgressContext> {src/main/java/org/jsoup/internal/SimpleBufferedInput.javaindex daeeaed9..c0a41a60 100644@@ -17,13 +17,15 @@ import static org.jsoup.internal.SharedConstants.DefaultBufferSize; class SimpleBufferedInput extends FilterInputStream { static final int BufferSize = DefaultBufferSize; static final SoftPool<byte[]> BufferPool = new SoftPool<>(() -> new byte[BufferSize]);- private int capRemaining = Integer.MAX_VALUE; // how many bytes we are allowed to pull from the underlying stream+ private boolean capped; // whether pulls from the underlying stream are capped+ private int capRemaining; // how many bytes we are allowed to pull from the underlying stream+ private int lookahead = -1; // one byte retained when probing beyond a cap - private byte @Nullable [] byteBuf; // the byte buffer; recycled via SoftPool. Created in fill if required+ private byte @Nullable [] byteBuf; // the byte buffer; recycled via SoftPool. Created in fill if required private int bufPos; private int bufLength;- private int bufMark = -1; // mark set by ControllableInputStream; -1 when unset- private boolean inReadFully = false; // true when the underlying inputstream has been read fully+ private int bufMark = -1; // mark set by ControllableInputStream; -1 when unset+ private boolean inReadFully = false; // true when the underlying inputstream has been read fully SimpleBufferedInput(@Nullable InputStream in) { super(in);@@ -73,24 +75,31 @@ class SimpleBufferedInput extends FilterInputStream { compact(); bufLength = bufPos;- int toRead = Math.min(byteBuf.length - bufPos, capRemaining);+ if (lookahead >= 0) {+ if (capped && capRemaining <= 0) return;+ byteBuf[bufLength++] = (byte) lookahead;+ lookahead = -1;+ if (capped) capRemaining--;+ }++ int toRead = capped ? Math.min(byteBuf.length - bufLength, capRemaining) : byteBuf.length - bufLength; if (toRead <= 0) return;- int read = in.read(byteBuf, bufPos, toRead);+ int read = in.read(byteBuf, bufLength, toRead); if (read > 0) {- bufLength = read + bufPos;- capRemaining -= read;- while (byteBuf.length - bufLength > 0 && capRemaining > 0) { // read in more if we have space, without blocking+ bufLength += read;+ if (capped) capRemaining -= read;+ while (byteBuf.length - bufLength > 0 && (!capped || capRemaining > 0)) { // read in more if we have space, without blocking try { if (in.available() < 1) break; } catch (IOException e) { break; // available() is advisory; keep the bytes we've already buffered }- toRead = Math.min(byteBuf.length - bufLength, capRemaining);+ toRead = capped ? Math.min(byteBuf.length - bufLength, capRemaining) : byteBuf.length - bufLength; if (toRead <= 0) break; read = in.read(byteBuf, bufLength, toRead); if (read <= 0) break; bufLength += read;- capRemaining -= read;+ if (capped) capRemaining -= read; } } if (read == -1) inReadFully = true;@@ -117,16 +126,44 @@ class SimpleBufferedInput extends FilterInputStream { @Override public int available() throws IOException { int buffered = (byteBuf != null) ? (bufLength - bufPos) : 0;+ if (lookahead >= 0 && (!capped || capRemaining > 0)) buffered++; if (buffered > 0) { return buffered; // doesn't include those in.available(), but mostly used as a block test } return inReadFully ? 0 : in.available(); } + /**+ Caps how many more bytes may be pulled from the underlying stream.+ */ void capRemaining(int newRemaining) {+ capped = true; capRemaining = Math.max(0, newRemaining); } + /**+ Removes the cap on pulls from the underlying stream.+ */+ void uncap() {+ capped = false;+ }++ /**+ Look one byte beyond the current cap, retaining it in case the cap is subsequently raised.+ */+ boolean hasMore() throws IOException {+ if (bufPos < bufLength || lookahead >= 0) return true;+ if (inReadFully) return false;++ int next = in.read();+ if (next == -1) {+ inReadFully = true;+ return false;+ }+ lookahead = next;+ return true;+ }+ void setMark() { bufMark = bufPos; }src/test/java/org/jsoup/integration/ConnectIT.javaindex 663af855..ad5cca5b 100644@@ -3,6 +3,7 @@ package org.jsoup.integration; import org.jsoup.Connection; import org.jsoup.Jsoup; import org.jsoup.helper.DataUtil;+import org.jsoup.integration.routes.LargeGzipRoute; import org.jsoup.integration.routes.SlowRider; import org.jsoup.internal.SharedConstants; import org.jsoup.nodes.Document;@@ -14,11 +15,13 @@ import org.junit.jupiter.api.parallel.Execution; import java.io.BufferedInputStream; import java.io.IOException;+import java.io.InputStream; import java.io.UncheckedIOException; import java.net.SocketTimeoutException; import java.nio.ByteBuffer; import java.nio.charset.StandardCharsets; import java.util.concurrent.atomic.AtomicBoolean;+import java.util.concurrent.atomic.AtomicInteger; import static org.jsoup.integration.TestServer.origin; import static org.jsoup.integration.TestServer.start;@@ -167,6 +170,19 @@ public class ConnectIT { assertThrows(UncheckedIOException.class, res::body); } + @Test+ @Execution(CONCURRENT)+ void bodyStreamThrowsOnTimeout() throws IOException {+ Connection.Response res = slowRiderTimeout().timeout(TimeoutMillis).execute();+ try (BufferedInputStream stream = res.bodyStream()) {+ assertThrows(IOException.class, () -> {+ while (stream.read() != -1) {+ // consume until the delayed next chunk trips the timeout+ }+ });+ }+ }+ @Test @Execution(CONCURRENT) public void infiniteReadSupported() throws IOException {@@ -241,14 +257,14 @@ public class ConnectIT { } @Test- public void remainingAfterFirstRead() throws IOException {+ public void bodyStreamCapSurvivesReset() throws IOException { int bufferSize = 5 * 1024; int capSize = 100 * 1024; String url = origin().file.url("/htmltests/large.html"); // 280 K - try (BufferedInputStream stream = Jsoup.connect(url).maxBodySize(capSize)- .execute().bodyStream()) {+ Connection.Response res = Jsoup.connect(url).maxBodySize(capSize).execute();+ try (BufferedInputStream stream = res.bodyStream()) { // simulates parse which does a limited read first stream.mark(bufferSize);@@ -267,22 +283,20 @@ public class ConnectIT { ByteBuffer fullRead = DataUtil.readToByteBuffer(stream, 0); byte[] fullArray = fullRead.array(); - // bodyStream is not capped to body size - only for jsoup consumed stream- assertTrue(fullArray.length > capSize);-- assertEquals(LargeHtmlSize, fullRead.limit());+ assertEquals(capSize, fullRead.limit()); String fullText = new String(fullRead.array(), 0, fullRead.limit(), StandardCharsets.UTF_8); assertTrue(fullText.startsWith(firstText));- assertEquals(LargeHtmlSize, fullText.length());+ assertEquals(capSize, fullText.length()); }+ assertTrue(res.isTruncated()); } @Test- public void noLimitAfterFirstRead() throws IOException {+ public void unlimitedBodyStreamSurvivesReset() throws IOException { int firstMaxRead = 5 * 1024; String url = origin().file.url("/htmltests/large.html"); // 280 K- try (BufferedInputStream stream = Jsoup.connect(url).execute().bodyStream()) {+ try (BufferedInputStream stream = Jsoup.connect(url).maxBodySize(0).execute().bodyStream()) { // simulates parse which does a limited read first stream.mark(firstMaxRead); ByteBuffer firstBytes = DataUtil.readToByteBuffer(stream, firstMaxRead);@@ -302,7 +316,7 @@ public class ConnectIT { } @Test- public void bodyStreamConstrainedViaReadFully() throws IOException {+ public void readFullyCapCarriesIntoBodyStream() throws IOException { int cap = 5 * 1024; String url = origin().file.url("/htmltests/large.html"); // 280 K try (BufferedInputStream stream = Jsoup@@ -316,4 +330,65 @@ public class ConnectIT { assertEquals(cap, cappedRead.limit()); } }++ @Test+ void exactBodyStreamLimitIsNotTruncated() throws IOException {+ Connection.Response res = Jsoup.connect(origin().file.url("/htmltests/large.html"))+ .maxBodySize(LargeHtmlSize)+ .execute();++ try (BufferedInputStream stream = res.bodyStream()) {+ assertEquals(LargeHtmlSize, readCount(stream));+ }+ assertFalse(res.isTruncated());+ }++ @Test+ void bodyStreamReportsProgress() throws IOException {+ AtomicInteger progressCalls = new AtomicInteger();+ Connection.Response res = Jsoup.connect(origin().file.url("/htmltests/large.html"))+ .onResponseProgress((processed, total, percent, response) -> progressCalls.incrementAndGet())+ .execute();++ try (BufferedInputStream stream = res.bodyStream()) {+ assertEquals(LargeHtmlSize, readCount(stream));+ }+ assertTrue(progressCalls.get() > 1);+ assertFalse(res.isTruncated());+ }++ @Test+ void unlimitedBodyStreamReadsPastIntegerMax() throws IOException {+ Connection.Response res = Jsoup.connect(origin().largeGzip.url())+ .ignoreContentType(true)+ .maxBodySize(0)+ .execute();++ try (BufferedInputStream stream = res.bodyStream()) {+ assertEquals(LargeGzipRoute.DecodedBodySize, readCount(stream));+ }+ assertFalse(res.isTruncated());+ }++ @Test+ void bodyStreamCapsDecodedContent() throws IOException {+ int cap = 2 * 1024 * 1024;+ Connection.Response res = Jsoup.connect(origin().largeGzip.url())+ .ignoreContentType(true)+ .maxBodySize(cap)+ .execute();++ try (BufferedInputStream stream = res.bodyStream()) {+ assertEquals(cap, readCount(stream));+ }+ assertTrue(res.isTruncated());+ }++ private static long readCount(InputStream stream) throws IOException {+ long count = 0;+ byte[] buffer = new byte[1024 * 1024];+ int read;+ while ((read = stream.read(buffer)) != -1) count += read;+ return count;+ } }src/test/java/org/jsoup/integration/ConnectTest.javaindex f992768f..2ead3d09 100644@@ -1071,6 +1071,11 @@ public class ConnectTest { assertEquals(196577, mediumRes.parse().text().length()); assertEquals(actualDocText, largeRes.parse().text().length()); assertEquals(actualDocText, unlimitedRes.parse().text().length());+ assertFalse(defaultRes.isTruncated());+ assertTrue(smallRes.isTruncated());+ assertTrue(mediumRes.isTruncated());+ assertFalse(largeRes.isTruncated());+ assertFalse(unlimitedRes.isTruncated()); } @Test public void repeatable() throws IOException {@@ -1100,6 +1105,11 @@ public class ConnectTest { assertEquals(200 * 1024, mediumRes.body().length()); assertEquals(actualDocText, largeRes.body().length()); assertEquals(actualDocText, unlimitedRes.body().length());+ assertFalse(defaultRes.isTruncated());+ assertTrue(smallRes.isTruncated());+ assertTrue(mediumRes.isTruncated());+ assertFalse(largeRes.isTruncated());+ assertFalse(unlimitedRes.isTruncated()); assertEquals(actualDocText, defaultRes.readBody().length()); assertEquals(50 * 1024, smallRes.readBody().length());src/test/java/org/jsoup/integration/TestServer.javaindex 64f29563..39c40b75 100644@@ -85,8 +85,9 @@ public class TestServer { public final Endpoint deflate = new Endpoint("/Deflate", DeflateRoute::handle); public final Endpoint interrupted = new Endpoint("/Interrupted", InterruptedRoute::handle); public final Endpoint slowRider = new Endpoint("/SlowRider", SlowRider::handle);+ public final Endpoint largeGzip = new Endpoint("/LargeGzip", LargeGzipRoute::handle); private final List<Endpoint> endpoints = Collections.unmodifiableList(Arrays.asList(- hello, noContentType, echo, file, redirect, cookie, deflate, interrupted, slowRider));+ hello, noContentType, echo, file, redirect, cookie, deflate, interrupted, slowRider, largeGzip)); private Origin() { }src/test/java/org/jsoup/integration/routes/LargeGzipRoute.javanew file mode 100644index 00000000..aa6b5aec@@ -0,0 +1,50 @@+package org.jsoup.integration.routes;++import org.jsoup.integration.netty.TestRequest;+import org.jsoup.integration.netty.TestResponse;++import java.io.ByteArrayOutputStream;+import java.io.IOException;+import java.util.zip.GZIPOutputStream;++/**+ Serves a small gzip representation that expands beyond {@link Integer#MAX_VALUE}.+ */+public final class LargeGzipRoute {+ public static final long DecodedBodySize = (long) Integer.MAX_VALUE + 1;+ private static final int DecodedMemberSize = 1024 * 1024;+ private static final int GzipMemberCount = (int) (DecodedBodySize / DecodedMemberSize);+ private static final byte[] EncodedBody = encodeBody();++ private LargeGzipRoute() {+ }++ /**+ Serves a virtual large response without storing or transferring its decoded representation.+ */+ public static void handle(TestRequest request, TestResponse response) {+ response.setContentType("application/octet-stream");+ response.setHeader("Content-Encoding", "gzip");+ response.write(EncodedBody);+ }++ private static byte[] encodeBody() {+ try {+ byte[] member = encodeZeroFilledMember();+ ByteArrayOutputStream encoded = new ByteArrayOutputStream(member.length * GzipMemberCount);+ for (int i = 0; i < GzipMemberCount; i++)+ encoded.write(member, 0, member.length);+ return encoded.toByteArray();+ } catch (IOException e) {+ throw new IllegalStateException(e);+ }+ }++ private static byte[] encodeZeroFilledMember() throws IOException {+ ByteArrayOutputStream encoded = new ByteArrayOutputStream();+ try (GZIPOutputStream gzip = new GZIPOutputStream(encoded)) {+ gzip.write(new byte[DecodedMemberSize]);+ }+ return encoded.toByteArray();+ }+}src/test/java/org/jsoup/internal/ControllableInputStreamTest.javaindex fc90fe85..4819aa07 100644@@ -6,6 +6,8 @@ import java.io.ByteArrayInputStream; import java.io.FilterInputStream; import java.io.IOException; import java.io.InputStream;+import java.util.concurrent.atomic.AtomicBoolean;+import java.util.concurrent.atomic.AtomicInteger; import static org.junit.jupiter.api.Assertions.*; @@ -61,6 +63,118 @@ class ControllableInputStreamTest { in.close(); } + @Test+ void distinguishesExactLimitFromTruncation() throws IOException {+ CountingInputStream shorterSource = new CountingInputStream(new ByteArrayInputStream(new byte[4]));+ ControllableInputStream shorter = ControllableInputStream.wrap(shorterSource, 5);+ assertEquals(4, readCount(shorter.inputStream()));+ assertEquals(4, shorterSource.count);+ assertFalse(shorter.isTruncated());++ CountingInputStream exactSource = new CountingInputStream(new ByteArrayInputStream(new byte[5]));+ ControllableInputStream exact = ControllableInputStream.wrap(exactSource, 5);+ assertEquals(5, readCount(exact.inputStream()));+ assertEquals(5, exactSource.count);+ assertFalse(exact.isTruncated());++ CountingInputStream longerSource = new CountingInputStream(new ByteArrayInputStream(new byte[6]));+ ControllableInputStream longer = ControllableInputStream.wrap(longerSource, 5);+ assertEquals(5, readCount(longer.inputStream()));+ assertEquals(6, longerSource.count);+ assertTrue(longer.isTruncated());+ }++ @Test+ void retainsLookaheadWhenLimitIsRaised() throws IOException {+ byte[] data = "abcdef".getBytes();+ ControllableInputStream in = ControllableInputStream.wrap(new ByteArrayInputStream(data), 5);+ in.mark(data.length);+ assertEquals(5, readCount(in));+ assertTrue(in.isTruncated());++ in.reset();+ in.max(0);+ byte[] reread = new byte[data.length];+ assertEquals(5, in.read(reread));+ assertEquals(1, in.read(reread, 5, 1));+ assertArrayEquals(data, reread);+ assertFalse(in.isTruncated());+ in.close();+ }++ @Test+ void readAndSkipRespectLimit() throws IOException {+ ControllableInputStream single = ControllableInputStream.wrap(+ new ByteArrayInputStream(new byte[] {42, 43}), 1);+ assertEquals(42, single.read());+ assertEquals(-1, single.read());+ assertTrue(single.isTruncated());+ single.close();++ ControllableInputStream skipped = ControllableInputStream.wrap(new ByteArrayInputStream(new byte[200]), 100);+ try (InputStream stream = skipped.inputStream()) {+ assertEquals(100, stream.skip(1000));+ assertEquals(-1, stream.read());+ }+ assertTrue(skipped.isTruncated());+ }++ @Test+ void largeSkipSaturatesProgress() throws IOException {+ long size = (long) Integer.MAX_VALUE + 1;+ ControllableInputStream in = ControllableInputStream.wrap(new VirtualInputStream(size), 0);+ AtomicBoolean negative = new AtomicBoolean();+ AtomicInteger lastProcessed = new AtomicInteger();+ AtomicInteger lastTotal = new AtomicInteger();+ AtomicInteger lastPercent = new AtomicInteger();+ in.onProgress(-1, (processed, total, percent, context) -> {+ if (processed < 0) negative.set(true);+ lastProcessed.set(processed);+ lastTotal.set(total);+ lastPercent.set((int) percent);+ }, in);++ try (InputStream stream = in.inputStream()) {+ assertEquals(size, stream.skip(size));+ assertEquals(-1, stream.read());+ }+ assertFalse(negative.get());+ assertEquals(Integer.MAX_VALUE, lastProcessed.get());+ assertEquals(Integer.MAX_VALUE, lastTotal.get());+ assertEquals(100, lastPercent.get());+ }++ private static long readCount(InputStream in) throws IOException {+ long count = 0;+ byte[] buffer = new byte[1024 * 1024];+ int read;+ while ((read = in.read(buffer)) != -1) count += read;+ return count;+ }++ private static final class VirtualInputStream extends InputStream {+ private long remaining;++ VirtualInputStream(long size) {+ remaining = size;+ }++ @Override+ public int read() {+ if (remaining == 0) return -1;+ remaining--;+ return 0;+ }++ @Override+ public int read(byte[] bytes, int offset, int length) {+ if (remaining == 0) return -1;+ int read = (int) Math.min(remaining, length);+ remaining -= read;+ return read;+ }+ }+ private static final class CountingInputStream extends FilterInputStream { int count = 0;