Backtestsscrapyrun-20261025-211112#1705

Issue #1705 Merged fix PR #8198 Run 25 Oct 2026 · run-20261025-211112 scrapy/scrapy

Image pipeline should allow to decide whether to auto-convert to jpg or not

Better

Our patch was judged better than the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

Both patches add an opt-out setting and both pass the independent tests, so the difference lies in correctness at the edges, compatibility, and craft. Patch A changes the signature of `convert_image` to require a new keyword-only `image_format` argument with no default or deprecation path; any subclass that overrides `convert_image` with the old signature will raise TypeError when the base pipeline calls it, and A had to rewrite existing tests to match (the 5 removed test lines). Patch B leaves `convert_image` untouched and routes the no-convert case through a separate helper, so existing overrides keep working. On behaviour, A stores preserved-format files with no extension at all, while B reuses the FilesPipeline URL-extension logic (`.png`, `.gif`, etc.), which is more useful and consistent with the sibling pipeline. B also handles the MPO-opened-JPEG case, wraps unsupported-format thumbnail saves in ImageException with a clear message, and falls back to application/octet-stream for unknown MIME types; A has an unguarded `assert orig_image.format is not None` and would surface raw Pillow errors for formats that can't be saved. B's extra change to `_key_for_pipe` is scope beyond the issue, but it is directly motivated: without it a class-prefixed `*_IMAGES_CONVERT_TO_JPEG = False` would be silently ignored, and it is covered by a dedicated test; the risk is small since callers only use the returned key with a default. Tests: A covers file_path, convert_image, and image_downloaded for the new mode; B covers those plus the default content-type, min-size check in the new path, the custom-settings matrix, and the class-attr vs setting override, with end-to-end on-disk verification. Docs are comparable; B adds a news entry and versionadded. Overall B resolves the issue without breaking the existing API and with better edge handling.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

7m 36s

Wall-clock time in the container

Full test suite

No new failures

107 passed · 20 skipped · 127 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

347vs 183 merged

Changed lines, ours vs merged

Not 100% it's accurate, but the issue got reported here:

https://news.ycombinator.com/item?id=10941213

Image pipeline converts to jpg automatically. Cannot be disabled without writing all the image code yourself

The two patches

Our patch+316 −31
docs/news.rstindex 3581c4240..186efacff 100644@@ -24,6 +24,22 @@ Backward-incompatible changes      (:gh:`TBD`) +New features+~~~~~~~~~~~~++-   Added the :setting:`IMAGES_CONVERT_TO_JPEG` setting, which can be set to+    ``False`` to make the :ref:`Images Pipeline <images-pipeline>` store+    images in their original format instead of converting them to JPEG.+    (:gh:`TBD`)++Bug fixes+~~~~~~~~~++-   Class-specific :ref:`media pipeline <topics-media-pipeline>` settings+    (e.g. ``MYPIPELINE_IMAGES_CONVERT_TO_JPEG``) set to a false value, such as+    ``False`` or ``0``, are no longer ignored in favor of the generic setting.+    (:gh:`TBD`)+ .. _release-2.19.0:  Scrapy 2.19.0 (2026-09-10)docs/topics/media-pipeline.rstindex 362480515..01263edb7 100644@@ -20,7 +20,8 @@ Both pipelines implement these features:  The Images Pipeline has a few extra functions for processing images: -* Convert all downloaded images to a common format (JPEG) and mode (RGB)+* Convert all downloaded images to a common format (JPEG) and mode (RGB),+  unless :setting:`IMAGES_CONVERT_TO_JPEG` is ``False`` * Thumbnail generation * Check images' width/height to make sure they meet a minimum constraint @@ -521,6 +522,40 @@ Example of image files stored using ``small`` and ``big`` thumbnail names::  The first one is the full image, as downloaded from the site. +.. _topics-images-format:++Keeping the original image format+---------------------------------++.. setting:: IMAGES_CONVERT_TO_JPEG++.. versionadded:: VERSION++By default, the Images Pipeline converts all downloaded images, and their+thumbnails, to JPEG images in RGB mode, and stores them with a ``.jpg``+extension.++To store images exactly as downloaded instead, set+:setting:`IMAGES_CONVERT_TO_JPEG` to ``False``:++.. code-block:: python++   IMAGES_CONVERT_TO_JPEG = False++When this setting is ``False``:++* The full image is stored with the same bytes it was downloaded with.++* Thumbnails are saved in the format of the original image (e.g. PNG+  thumbnails for PNG images).++* The file extension of both the full image and its thumbnails is determined+  from the image URL, the same way as for the :ref:`Files Pipeline+  <topics-file-naming>` (it may be empty if it cannot be determined).++* The ``Content-Type`` sent to storage backends that support it (e.g. Amazon S3+  or Google Cloud Storage) matches the format of the original image.+ Filtering out small images -------------------------- scrapy/pipelines/files.pyindex 8e49f52e9..86573b0b8 100644@@ -64,6 +64,27 @@ def _to_string(path: str | PathLike[str]) -> str:     return str(path)  # convert a Path object to string  +def _get_media_extension(request: Request) -> str:+    """Return the file extension (including the leading dot) for the media+    file of *request*, guessed from its URL, or an empty string."""+    # clean it up and look at the path first+    parsed_url = urlparse_cached(request)+    media_ext = Path(parsed_url.path).suffix++    # if path has no extension look at the raw  URL+    if media_ext not in mimetypes.types_map:+        media_ext = Path(request.url).suffix++    # Handles empty and wild extensions by trying to guess the+    # mime type then extension or default to empty string otherwise+    if media_ext not in mimetypes.types_map:+        media_ext = ""+        media_type = mimetypes.guess_type(request.url)[0]+        if media_type:+            media_ext = cast("str", mimetypes.guess_extension(media_type))+    return media_ext++ def _md5sum(file: IO[bytes]) -> str:     """Calculate the md5 checksum of a file-like object without reading its     whole content in memory.@@ -750,22 +771,7 @@ class FilesPipeline(MediaPipeline):         item: Any = None,     ) -> str:         media_guid = hashlib.sha1(to_bytes(request.url)).hexdigest()  # noqa: S324--        # clean it up and look at the path first-        parsed_url = urlparse_cached(request)-        media_ext = Path(parsed_url.path).suffix--        # if path has no extension look at the raw  URL-        if media_ext not in mimetypes.types_map:-            media_ext = Path(request.url).suffix--        # Handles empty and wild extensions by trying to guess the-        # mime type then extension or default to empty string otherwise-        if media_ext not in mimetypes.types_map:-            media_ext = ""-            media_type = mimetypes.guess_type(request.url)[0]-            if media_type:-                media_ext = cast("str", mimetypes.guess_extension(media_type))+        media_ext = _get_media_extension(request)         return f"full/{media_guid}{media_ext}"  scrapy/pipelines/images.pyindex 7186fc8de..dbe163310 100644@@ -18,7 +18,13 @@ from itemadapter import ItemAdapter from scrapy.exceptions import NotConfigured, ScrapyDeprecationWarning from scrapy.http import Request, Response from scrapy.http.request import NO_CALLBACK-from scrapy.pipelines.files import FilesPipeline, GCSFilesStore, S3FilesStore, _md5sum+from scrapy.pipelines.files import (+    FilesPipeline,+    GCSFilesStore,+    S3FilesStore,+    _get_media_extension,+    _md5sum,+) from scrapy.pipelines.media import FileException from scrapy.utils.defer import ensure_awaitable from scrapy.utils.python import to_bytes@@ -52,6 +58,7 @@ class ImagesPipeline(FilesPipeline):     MIN_HEIGHT: int = 0     EXPIRES: int = 90     THUMBS: ClassVar[dict[str, tuple[int, int]]] = {}+    CONVERT_TO_JPEG: bool = True     DEFAULT_IMAGES_URLS_FIELD = "image_urls"     DEFAULT_IMAGES_RESULT_FIELD = "images" @@ -110,6 +117,9 @@ class ImagesPipeline(FilesPipeline):         self.thumbs: dict[str, tuple[int, int]] = settings.get(             resolve("IMAGES_THUMBS"), self.THUMBS         )+        self.convert_to_jpeg: bool = settings.getbool(+            resolve("IMAGES_CONVERT_TO_JPEG"), self.CONVERT_TO_JPEG+        )      @classmethod     def from_crawler(cls, crawler: Crawler) -> Self:@@ -150,6 +160,9 @@ class ImagesPipeline(FilesPipeline):         *,         item: Any = None,     ) -> str:+        content_type = "image/jpeg"+        if not self.convert_to_jpeg:+            content_type = self._get_content_type(response)         checksum: str | None = None         for path, image, buf in self.get_images(response, request, info, item=item):             if checksum is None:@@ -162,7 +175,7 @@ class ImagesPipeline(FilesPipeline):                     buf,                     info,                     meta={"width": width, "height": height},-                    headers={"Content-Type": "image/jpeg"},+                    headers={"Content-Type": content_type},                 )             )         assert checksum is not None@@ -188,6 +201,18 @@ class ImagesPipeline(FilesPipeline):                 f"{self.min_width}x{self.min_height})"             ) +        if not self.convert_to_jpeg:+            yield from self._get_unconverted_images(+                orig_image.format,+                transposed_image,+                path,+                response,+                request,+                info,+                item=item,+            )+            return+         image, buf = self.convert_image(             transposed_image, response_body=BytesIO(response.body)         )@@ -200,6 +225,60 @@ class ImagesPipeline(FilesPipeline):             thumb_image, thumb_buf = self.convert_image(image, size, response_body=buf)             yield thumb_path, thumb_image, thumb_buf +    def _get_unconverted_images(+        self,+        image_format: str | None,+        image: Image.Image,+        path: str,+        response: Response,+        request: Request,+        info: MediaPipeline.SpiderInfo,+        *,+        item: Any = None,+    ) -> Iterator[tuple[str, Image.Image, BytesIO]]:+        """Yield the original image, stored as downloaded, and its thumbnails,+        saved in the format of the original image."""+        yield path, image, BytesIO(response.body)++        # Pillow opens some JPEG files (e.g. from phone cameras) as MPO, which+        # older Pillow versions cannot save.+        if image_format == "MPO":+            image_format = "JPEG"+        for thumb_id, size in self.thumbs.items():+            thumb_path = self.thumb_path(+                request, thumb_id, response=response, info=info, item=item+            )+            thumb_image = self._resize_image(image, size)+            thumb_buf = BytesIO()+            try:+                thumb_image.save(thumb_buf, image_format)+            except (KeyError, OSError, ValueError) as e:+                raise ImageException(+                    f"Cannot save a {size[0]}x{size[1]} thumbnail of the image "+                    f"in its original format ({image_format}): {e}"+                ) from e+            yield thumb_path, thumb_image, thumb_buf++    def _get_content_type(self, response: Response) -> str:+        """Return the MIME type of the image in *response*."""+        image_format = self._Image.open(BytesIO(response.body)).format+        if image_format == "MPO":+            image_format = "JPEG"+        mime_type = self._Image.MIME.get(image_format) if image_format else None+        return mime_type or "application/octet-stream"++    def _resize_image(self, image: Image.Image, size: tuple[int, int]) -> Image.Image:+        image = image.copy()+        try:+            # Image.Resampling.LANCZOS was added in Pillow 9.1.0+            # remove this try except block,+            # when updating the minimum requirements for Pillow.+            resampling_filter = self._Image.Resampling.LANCZOS+        except AttributeError:+            resampling_filter = self._Image.ANTIALIAS  # type: ignore[attr-defined]+        image.thumbnail(size, resampling_filter)+        return image+     def convert_image(         self,         image: Image.Image,@@ -220,15 +299,7 @@ class ImagesPipeline(FilesPipeline):             image = image.convert("RGB")          if size:-            image = image.copy()-            try:-                # Image.Resampling.LANCZOS was added in Pillow 9.1.0-                # remove this try except block,-                # when updating the minimum requirements for Pillow.-                resampling_filter = self._Image.Resampling.LANCZOS-            except AttributeError:-                resampling_filter = self._Image.ANTIALIAS  # type: ignore[attr-defined]-            image.thumbnail(size, resampling_filter)+            image = self._resize_image(image, size)         elif image.format == "JPEG":             return image, response_body @@ -262,7 +333,7 @@ class ImagesPipeline(FilesPipeline):         item: Any = None,     ) -> str:         image_guid = hashlib.sha1(to_bytes(request.url)).hexdigest()  # noqa: S324-        return f"full/{image_guid}.jpg"+        return f"full/{image_guid}{self._get_image_extension(request)}"      def thumb_path(         self,@@ -274,4 +345,9 @@ class ImagesPipeline(FilesPipeline):         item: Any = None,     ) -> str:         thumb_guid = hashlib.sha1(to_bytes(request.url)).hexdigest()  # noqa: S324-        return f"thumbs/{thumb_id}/{thumb_guid}.jpg"+        return f"thumbs/{thumb_id}/{thumb_guid}{self._get_image_extension(request)}"++    def _get_image_extension(self, request: Request) -> str:+        if self.convert_to_jpeg:+            return ".jpg"+        return _get_media_extension(request)scrapy/pipelines/media.pyindex 41b2e17c9..3f35752bb 100644@@ -128,7 +128,7 @@ class MediaPipeline(ABC):         if (             not base_class_name             or class_name == base_class_name-            or (settings and not settings.get(formatted_key))+            or (settings is not None and settings.get(formatted_key) is None)         ):             return key         return formatted_keytests/test_pipeline_images.pyindex 74b4f3a9b..a98eeb161 100644@@ -273,6 +273,124 @@ class TestImagesPipeline:         assert converted.mode == "RGB"         assert converted.getcolors() == [(10000, (205, 230, 255))] +    def _no_convert_pipeline(self) -> ImagesPipeline:+        crawler = get_crawler(+            None, {"IMAGES_STORE": self.tempdir, "IMAGES_CONVERT_TO_JPEG": False}+        )+        return build_from_crawler(ImagesPipeline, crawler)++    def test_convert_to_jpeg_default(self):+        assert self.pipeline.convert_to_jpeg is True++    def test_file_path_no_convert(self):+        pipeline = self._no_convert_pipeline()+        guid = "3fd165099d8e71b8a48b2683946e64dbfad8b52d"+        req = Request("https://dev.mydeco.com/mydeco.gif")+        assert pipeline.file_path(req) == f"full/{guid}.gif"+        assert pipeline.thumb_path(req, "small") == f"thumbs/small/{guid}.gif"+        assert pipeline.file_path(+            Request("https://example.com/image.png?width=100")+        ).endswith(".png")+        # No extension can be determined from the URL.+        assert "." not in pipeline.file_path(Request("https://example.com/image"))++    def test_get_images_no_convert(self):+        pipeline = self._no_convert_pipeline()+        pipeline.thumbs = {"small": (20, 20)}++        _, buf = _create_image("PNG", "RGBA", (50, 50), (0, 127, 255, 50))+        url = "https://dev.mydeco.com/mydeco.png"+        resp = Response(url=url, body=buf.getvalue())+        req = Request(url=url)++        images = list(+            pipeline.get_images(response=resp, request=req, info=DUMMY_SPIDER_INFO)+        )+        assert len(images) == 2++        path, image, image_buf = images[0]+        assert path == pipeline.file_path(req)+        assert path.startswith("full/")+        assert path.endswith(".png")+        assert image.mode == "RGBA"+        assert image_buf.getvalue() == buf.getvalue()++        thumb_path, thumb_image, thumb_buf = images[1]+        assert thumb_path == pipeline.thumb_path(req, "small")+        assert thumb_path.endswith(".png")+        assert thumb_image.size == (20, 20)+        thumb_buf.seek(0)+        stored_thumb = Image.open(thumb_buf)+        assert stored_thumb.format == "PNG"+        assert stored_thumb.mode == "RGBA"+        pixel = stored_thumb.getpixel((0, 0))+        assert isinstance(pixel, tuple)+        assert pixel[3] == 50++    def test_get_images_no_convert_min_size(self):+        pipeline = self._no_convert_pipeline()+        pipeline.min_width = 100+        _, buf = _create_image("PNG", "RGB", (50, 50), (0, 0, 0))+        url = "https://dev.mydeco.com/mydeco.png"+        resp = Response(url=url, body=buf.getvalue())+        with pytest.raises(ImageException, match="Image too small"):+            next(+                pipeline.get_images(+                    response=resp, request=Request(url), info=DUMMY_SPIDER_INFO+                )+            )++    @coroutine_test+    async def test_image_downloaded_no_convert(self) -> None:+        """With IMAGES_CONVERT_TO_JPEG = False, the original image bytes and+        PNG thumbnails are stored with the original Content-Type."""+        pipeline = self._no_convert_pipeline()+        pipeline.thumbs = {"small": (20, 20)}+        _, buf = _create_image("PNG", "RGBA", (50, 50), (0, 127, 255, 50))+        url = "https://dev.mydeco.com/mydeco.png"+        response = Response(url=url, body=buf.getvalue())+        request = Request(url=url)++        persisted: list[tuple[str, dict[str, str]]] = []+        persist_file = pipeline.store.persist_file++        def record_persist_file(path, buf, info, meta=None, headers=None):+            persisted.append((path, headers))+            return persist_file(path, buf, info, meta=meta, headers=headers)++        pipeline.store.persist_file = record_persist_file  # type: ignore[method-assign]++        checksum = await pipeline.image_downloaded(response, request, DUMMY_SPIDER_INFO)++        buf.seek(0)+        assert checksum == _md5sum(buf)+        full_path = pipeline.file_path(request)+        thumb_path = pipeline.thumb_path(request, "small")+        assert full_path.endswith(".png")+        assert Path(self.tempdir, full_path).read_bytes() == buf.getvalue()+        thumb = Image.open(Path(self.tempdir, thumb_path))+        assert thumb.format == "PNG"+        assert thumb.mode == "RGBA"+        assert persisted == [+            (full_path, {"Content-Type": "image/png"}),+            (thumb_path, {"Content-Type": "image/png"}),+        ]++    @coroutine_test+    async def test_image_downloaded_default_content_type(self) -> None:+        _, buf = _create_image("PNG", "RGB", (50, 50), (0, 0, 0))+        url = "https://dev.mydeco.com/mydeco.png"+        persisted: list[dict[str, str]] = []++        def record_persist_file(path, buf, info, meta=None, headers=None):+            persisted.append(headers)++        self.pipeline.store.persist_file = record_persist_file  # type: ignore[method-assign]+        await self.pipeline.image_downloaded(+            Response(url=url, body=buf.getvalue()), Request(url), DUMMY_SPIDER_INFO+        )+        assert persisted == [{"Content-Type": "image/jpeg"}]+     def test_convert_image_legacy_resampling_filter(         self, monkeypatch: pytest.MonkeyPatch     ) -> None:@@ -412,6 +530,7 @@ class TestImagesPipelineCustomSettings:         ("IMAGES_URLS_FIELD", "IMAGES_URLS_FIELD"),         ("IMAGES_RESULT_FIELD", "IMAGES_RESULT_FIELD"),         ("THUMBS", "IMAGES_THUMBS"),+        ("CONVERT_TO_JPEG", "IMAGES_CONVERT_TO_JPEG"),     ]      # This should match what is defined in ImagesPipeline.@@ -422,6 +541,7 @@ class TestImagesPipelineCustomSettings:         "THUMBS": {},         "IMAGES_URLS_FIELD": "image_urls",         "IMAGES_RESULT_FIELD": "images",+        "CONVERT_TO_JPEG": True,     }      def _generate_fake_settings(@@ -446,6 +566,7 @@ class TestImagesPipelineCustomSettings:                 "small": (random.randint(1, 1000), random.randint(1, 1000)),                 "big": (random.randint(1, 1000), random.randint(1, 1000)),             },+            "IMAGES_CONVERT_TO_JPEG": False,         }         if not prefix:             return settings@@ -471,6 +592,7 @@ class TestImagesPipelineCustomSettings:             EXPIRES = random.randint(1000, 2000)             IMAGES_URLS_FIELD = "field_one"             IMAGES_RESULT_FIELD = "field_two"+            CONVERT_TO_JPEG = False          return UserDefinedImagePipeline @@ -610,6 +732,21 @@ class TestImagesPipelineCustomSettings:             expected_value = settings.get(settings_attr)             assert getattr(pipeline_cls, pipe_attr.lower()) == expected_value +    def test_convert_to_jpeg_setting_overrides_class_attr(self, tmp_path):+        class UserDefinedImagePipeline(ImagesPipeline):+            CONVERT_TO_JPEG = False++        pipeline = build_from_crawler(+            UserDefinedImagePipeline,+            get_crawler(+                None, {"IMAGES_STORE": tmp_path, "IMAGES_CONVERT_TO_JPEG": True}+            ),+        )+        assert pipeline.convert_to_jpeg is True+        assert pipeline.file_path(Request("https://example.com/a.png")).endswith(+            ".jpg"+        )+     def test_images_store_s3_acl_setting_used(self, tmp_path):         old_policy = S3FilesStore.POLICY tests/test_pipeline_media.pyindex fd377fb7f..3cc5d63de 100644@@ -520,6 +520,21 @@ class TestMediaPipelineAllowRedirectSettings:             UserDefinedPipeline, {"USERDEFINEDPIPELINE_MEDIA_ALLOW_REDIRECTS": True}         ) +    def test_subclass_specific_false_setting_overrides_standard_setting(self):+        pipe = UserDefinedPipeline(+            crawler=get_crawler(+                None,+                {+                    "MEDIA_ALLOW_REDIRECTS": True,+                    "USERDEFINEDPIPELINE_MEDIA_ALLOW_REDIRECTS": False,+                },+            )+        )+        request = Request("http://url")+        pipe._modify_media_request(request)+        assert pipe.allow_redirects is False+        assert "handle_httpstatus_list" not in request.meta+  class TestBuildFromCrawler:     def setup_method(self): 
Merged fix · PR #8198+160 −23
docs/topics/media-pipeline.rstindex 362480515..ca2bfae64 100644@@ -521,6 +521,26 @@ Example of image files stored using ``small`` and ``big`` thumbnail names::  The first one is the full image, as downloaded from the site. +.. _preserving-image-format:++Preserving the original image format+------------------------------------++.. setting:: IMAGES_PRESERVE_FORMAT++By default, the Images Pipeline converts every image to JPEG, flattening+transparency onto a white background. Set :setting:`IMAGES_PRESERVE_FORMAT`+to ``True`` to keep each image, and its thumbnails, in the format it was+downloaded in instead:++.. code-block:: python++    IMAGES_PRESERVE_FORMAT = True++Stored files then get no extension, since the actual format is only known+once the image has been downloaded; see :ref:`file-naming-response` to+derive one from the response instead.+ Filtering out small images -------------------------- scrapy/pipelines/images.pyindex 7186fc8de..c97f119ca 100644@@ -52,6 +52,7 @@ class ImagesPipeline(FilesPipeline):     MIN_HEIGHT: int = 0     EXPIRES: int = 90     THUMBS: ClassVar[dict[str, tuple[int, int]]] = {}+    PRESERVE_FORMAT: bool = False     DEFAULT_IMAGES_URLS_FIELD = "image_urls"     DEFAULT_IMAGES_RESULT_FIELD = "images" @@ -110,6 +111,9 @@ class ImagesPipeline(FilesPipeline):         self.thumbs: dict[str, tuple[int, int]] = settings.get(             resolve("IMAGES_THUMBS"), self.THUMBS         )+        self.preserve_format: bool = settings.getbool(+            resolve("IMAGES_PRESERVE_FORMAT"), self.PRESERVE_FORMAT+        )      @classmethod     def from_crawler(cls, crawler: Crawler) -> Self:@@ -156,13 +160,14 @@ class ImagesPipeline(FilesPipeline):                 buf.seek(0)                 checksum = _md5sum(buf)             width, height = image.size+            content_type = self._Image.MIME.get(image.format or "JPEG", "image/jpeg")             await ensure_awaitable(                 self.store.persist_file(                     path,                     buf,                     info,                     meta={"width": width, "height": height},-                    headers={"Content-Type": "image/jpeg"},+                    headers={"Content-Type": content_type},                 )             )         assert checksum is not None@@ -178,6 +183,8 @@ class ImagesPipeline(FilesPipeline):     ) -> Iterator[tuple[str, Image.Image, BytesIO]]:         path = self.file_path(request, response=response, info=info, item=item)         orig_image = self._Image.open(BytesIO(response.body))+        assert orig_image.format is not None+        image_format = orig_image.format         transposed_image = self._ImageOps.exif_transpose(orig_image)          width, height = transposed_image.size@@ -189,7 +196,9 @@ class ImagesPipeline(FilesPipeline):             )          image, buf = self.convert_image(-            transposed_image, response_body=BytesIO(response.body)+            transposed_image,+            image_format=image_format,+            response_body=BytesIO(response.body),         )         yield path, image, buf @@ -197,7 +206,9 @@ class ImagesPipeline(FilesPipeline):             thumb_path = self.thumb_path(                 request, thumb_id, response=response, info=info, item=item             )-            thumb_image, thumb_buf = self.convert_image(image, size, response_body=buf)+            thumb_image, thumb_buf = self.convert_image(+                image, size, image_format=image_format, response_body=buf+            )             yield thumb_path, thumb_image, thumb_buf      def convert_image(@@ -205,19 +216,22 @@ class ImagesPipeline(FilesPipeline):         image: Image.Image,         size: tuple[int, int] | None = None,         *,+        image_format: str,         response_body: BytesIO,     ) -> tuple[Image.Image, BytesIO]:-        if image.format in {"PNG", "WEBP"} and image.mode == "RGBA":-            background = self._Image.new("RGBA", image.size, (255, 255, 255))-            background.paste(image, image)-            image = background.convert("RGB")-        elif image.mode == "P":-            image = image.convert("RGBA")-            background = self._Image.new("RGBA", image.size, (255, 255, 255))-            background.paste(image, image)-            image = background.convert("RGB")-        elif image.mode != "RGB":-            image = image.convert("RGB")+        target_format = image_format if self.preserve_format else "JPEG"+        if target_format == "JPEG":+            if image_format in {"PNG", "WEBP"} and image.mode == "RGBA":+                background = self._Image.new("RGBA", image.size, (255, 255, 255))+                background.paste(image, image)+                image = background.convert("RGB")+            elif image.mode == "P":+                image = image.convert("RGBA")+                background = self._Image.new("RGBA", image.size, (255, 255, 255))+                background.paste(image, image)+                image = background.convert("RGB")+            elif image.mode != "RGB":+                image = image.convert("RGB")          if size:             image = image.copy()@@ -229,11 +243,13 @@ class ImagesPipeline(FilesPipeline):             except AttributeError:                 resampling_filter = self._Image.ANTIALIAS  # type: ignore[attr-defined]             image.thumbnail(size, resampling_filter)-        elif image.format == "JPEG":+        elif target_format == image_format:+            image.format = target_format             return image, response_body          buf = BytesIO()-        image.save(buf, "JPEG")+        image.save(buf, target_format)+        image.format = target_format         return image, buf      def get_media_requests(@@ -262,7 +278,8 @@ class ImagesPipeline(FilesPipeline):         item: Any = None,     ) -> str:         image_guid = hashlib.sha1(to_bytes(request.url)).hexdigest()  # noqa: S324-        return f"full/{image_guid}.jpg"+        ext = "" if self.preserve_format else ".jpg"+        return f"full/{image_guid}{ext}"      def thumb_path(         self,@@ -274,4 +291,5 @@ class ImagesPipeline(FilesPipeline):         item: Any = None,     ) -> str:         thumb_guid = hashlib.sha1(to_bytes(request.url)).hexdigest()  # noqa: S324-        return f"thumbs/{thumb_id}/{thumb_guid}.jpg"+        ext = "" if self.preserve_format else ".jpg"+        return f"thumbs/{thumb_id}/{thumb_guid}{ext}"tests/test_pipeline_images.pyindex 74b4f3a9b..04c946f57 100644@@ -97,6 +97,22 @@ class TestImagesPipeline:             == "full/244e0dd7d96a3b7b01f54eded250c9e272577aa1.jpg"         ) +    def test_file_path_preserve_format(self):+        pipeline = build_from_crawler(+            ImagesPipeline,+            get_crawler(+                None, {"IMAGES_STORE": self.tempdir, "IMAGES_PRESERVE_FORMAT": True}+            ),+        )+        assert (+            pipeline.file_path(Request("https://dev.mydeco.com/mydeco.gif"))+            == "full/3fd165099d8e71b8a48b2683946e64dbfad8b52d"+        )+        assert (+            pipeline.thumb_path(Request("https://dev.mydeco.com/mydeco.gif"), "50")+            == "thumbs/50/3fd165099d8e71b8a48b2683946e64dbfad8b52d"+        )+     def test_thumbnail_name(self):         thumb_path = self.pipeline.thumb_path         name = "50"@@ -240,12 +256,81 @@ class TestImagesPipeline:         assert Path(self.tempdir, "full", name).read_bytes() == buf.getvalue()         assert Path(self.tempdir, "thumbs", "small", name).exists() +    def test_convert_image_preserve_format(self):+        pipeline = build_from_crawler(+            ImagesPipeline,+            get_crawler(+                None, {"IMAGES_STORE": self.tempdir, "IMAGES_PRESERVE_FORMAT": True}+            ),+        )++        SIZE = (100, 100)+        COLOUR = (0, 127, 255, 50)+        im, buf = _create_image("PNG", "RGBA", SIZE, COLOUR)++        converted, converted_buf = pipeline.convert_image(+            im, image_format="PNG", response_body=buf+        )+        assert converted.mode == "RGBA"+        assert converted.format == "PNG"+        # the original bytes are reused, no re-encoding happens+        assert converted_buf == buf++        thumbnail, thumbnail_buf = pipeline.convert_image(+            converted,+            size=(10, 25),+            image_format="PNG",+            response_body=converted_buf,+        )+        assert thumbnail.mode == "RGBA"+        assert thumbnail.format == "PNG"+        assert thumbnail.size == (10, 10)+        assert Image.open(thumbnail_buf).format == "PNG"++    @coroutine_test+    async def test_image_downloaded_preserve_format(+        self, monkeypatch: pytest.MonkeyPatch+    ) -> None:+        pipeline = build_from_crawler(+            ImagesPipeline,+            get_crawler(+                None, {"IMAGES_STORE": self.tempdir, "IMAGES_PRESERVE_FORMAT": True}+            ),+        )+        pipeline.thumbs = {"small": (20, 20)}+        _, buf = _create_image("PNG", "RGBA", (50, 50), (0, 127, 255, 50))+        url = "https://dev.mydeco.com/mydeco.gif"+        response = Response(url=url, body=buf.getvalue())++        persisted = []+        monkeypatch.setattr(+            pipeline.store,+            "persist_file",+            lambda path, b, info, meta=None, headers=None: persisted.append(+                (path, b.getvalue(), headers)+            ),+        )++        await pipeline.image_downloaded(response, Request(url=url), DUMMY_SPIDER_INFO)++        assert len(persisted) == 2+        full_path, full_bytes, full_headers = persisted[0]+        assert full_path == "full/3fd165099d8e71b8a48b2683946e64dbfad8b52d"+        assert full_bytes == buf.getvalue()+        assert full_headers == {"Content-Type": "image/png"}++        _, thumb_bytes, thumb_headers = persisted[1]+        assert Image.open(io.BytesIO(thumb_bytes)).format == "PNG"+        assert thumb_headers == {"Content-Type": "image/png"}+     def test_convert_image(self):         SIZE = (100, 100)         # straight forward case: RGB and JPEG         COLOUR: tuple[int, ...] = (0, 127, 255)         im, buf = _create_image("JPEG", "RGB", SIZE, COLOUR)-        converted, converted_buf = self.pipeline.convert_image(im, response_body=buf)+        converted, converted_buf = self.pipeline.convert_image(+            im, image_format="JPEG", response_body=buf+        )         assert converted.mode == "RGB"         assert converted.getcolors() == [(10000, COLOUR)]         # check that we don't convert JPEGs again@@ -253,7 +338,7 @@ class TestImagesPipeline:          # check that thumbnail keep image ratio         thumbnail, _ = self.pipeline.convert_image(-            converted, size=(10, 25), response_body=converted_buf+            converted, size=(10, 25), image_format="JPEG", response_body=converted_buf         )         assert thumbnail.mode == "RGB"         assert thumbnail.size == (10, 10)@@ -261,7 +346,9 @@ class TestImagesPipeline:         # transparency case: RGBA and PNG         COLOUR = (0, 127, 255, 50)         im, buf = _create_image("PNG", "RGBA", SIZE, COLOUR)-        converted, _ = self.pipeline.convert_image(im, response_body=buf)+        converted, _ = self.pipeline.convert_image(+            im, image_format="PNG", response_body=buf+        )         assert converted.mode == "RGB"         assert converted.getcolors() == [(10000, (205, 230, 255))] @@ -269,10 +356,20 @@ class TestImagesPipeline:         COLOUR = (0, 127, 255, 50)         im, buf = _create_image("PNG", "RGBA", SIZE, COLOUR)         im = im.convert("P")-        converted, _ = self.pipeline.convert_image(im, response_body=buf)+        converted, _ = self.pipeline.convert_image(+            im, image_format="PNG", response_body=buf+        )         assert converted.mode == "RGB"         assert converted.getcolors() == [(10000, (205, 230, 255))] +        # grayscale case: L and PNG+        im, buf = _create_image("PNG", "L", SIZE, 128)+        converted, _ = self.pipeline.convert_image(+            im, image_format="PNG", response_body=buf+        )+        assert converted.mode == "RGB"+        assert converted.getcolors() == [(10000, (128, 128, 128))]+     def test_convert_image_legacy_resampling_filter(         self, monkeypatch: pytest.MonkeyPatch     ) -> None:@@ -287,7 +384,9 @@ class TestImagesPipeline:         )         im, buf = _create_image("JPEG", "RGB", (100, 100), (0, 127, 255)) -        thumbnail, _ = self.pipeline.convert_image(im, size=(10, 25), response_body=buf)+        thumbnail, _ = self.pipeline.convert_image(+            im, size=(10, 25), image_format="JPEG", response_body=buf+        )          assert thumbnail.size == (10, 10)