Fixes jazzband-roadies/help#182

Some notable changes:

• Deprecate explicit support for Python 3.4 and 3.5 in order to simplify the test matrix.
• Clean up the test runner setup with tox and Travis

Hello,

There is a serious performance issue in current 12.2 branch. Problem is that we have ~2M thumbnails, so if execution goes here (marked line):

# /sorl/thumbnail/base.py:101 (get_thumbnail)
        # We have to check exists() because the Storage backend does not
        # overwrite in some implementations.
        if not thumbnail.exists(): # <---- This is the root of the problem!
            try:
                source_image = default.engine.get_image(source)
            except IOError:
                if settings.THUMBNAIL_DUMMY:
                    return DummyImageFile(geometry_string)
                else:
                    # if S3Storage says file doesn't exist remotely, don't try to
                    # create it and exit early.
                    # Will return working empty image type; 404'd image
                    logger.warn(text_type('Remote file [%s] at [%s] does not exist'),
                                file_, geometry_string)

                    return thumbnail

            # We might as well set the size since we have the image in memory
            image_info = default.engine.get_image_info(source_image)
            options['image_info'] = image_info
            size = default.engine.get_image_size(source_image)
            source.set_size(size)

            try:
                self._create_thumbnail(source_image, geometry_string, options,
                                       thumbnail)
                self._create_alternative_resolutions(source_image, geometry_string,
                                                     options, thumbnail.name)
            finally:
                default.engine.cleanup(source_image)

it is actually asks boto to return LIST of all stored thumbnails (without even using prefix), so appliction hangs with 100% CPU and high memory usage (well, not a surprise actually).

Wouldn't it be better to provide a prefix for lookup (constructed with the same function as used to store thumbnail) ?

In a mean time we've had to revert to 11.12.1b which works better.

This is related to a fix introduced in #92

Hello I am experiencing problems with sorl and django 1.7 (testing environment):

(zenbframework)vagrant@precise64:/python-projects/zenbframework/zenframework$ pip freeze
Django==1.7b1
Pillow==2.4.0
argparse==1.2.1
django-braces==1.4.0
django-crispy-forms==1.4.0
django-mptt==0.6.0
redis==2.9.1
six==1.6.1
sorl-thumbnail==11.12
unicode-slugify==0.1.1
wsgiref==0.1.2

My settings:

INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',

    'sorl.thumbnail',
    'posts',
    'news',
)

Trying to run migrate (replaces syncdb):

(zenbframework)vagrant@precise64:/python-projects/zenbframework/zenframework$ ./manage.py migrate
Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/vagrant/.virtualenvs/zenbframework/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 427, in execute_from_command_line
    utility.execute()
  File "/home/vagrant/.virtualenvs/zenbframework/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 391, in execute
    django.setup()
  File "/home/vagrant/.virtualenvs/zenbframework/local/lib/python2.7/site-packages/django/__init__.py", line 21, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/home/vagrant/.virtualenvs/zenbframework/local/lib/python2.7/site-packages/django/apps/registry.py", line 85, in populate
    app_config = AppConfig.create(entry)
  File "/home/vagrant/.virtualenvs/zenbframework/local/lib/python2.7/site-packages/django/apps/config.py", line 120, in create
    "cannot import name '%s' from '%s'" % (cls_name, mod_path))
ImportError: cannot import name 'thumbnail' from 'sorl'

Hi, how are you?

Here have a new issue, i think.

My png file haven't background color, have transparency, and, when i upload using sorl-thumbnail's ImageField the image have black background color, see:

How can we fix this bug?

The TEMPLATE_DEBUG setting which was deprecated in Django 1.8 and 1.9, and is missing from 1.10's docs (so I assume it's now obsolete) is required by sorl-thumbnail.

Without a TEMPLATE_DEBUG setting I get this when I try and use the {% thumbnail ... %} tag in a template:

'Settings' object has no attribute 'TEMPLATE_DEBUG'

If I add it to my settings, the template tag works.

I'm trying to use sorl with pretty basic configuration (only change being THUMBNAIL_PRESERVE_FORMAT = True). Unfortunately, for some images (for example https://dl.dropboxusercontent.com/u/35354297/4.png), it fails with this traceback:

Traceback (most recent call last):
  File "/Users/xaralis/Workspace/adventura/adv/app/models/photos.py", line 54, in get_thumbnail_in_format
    return get_thumbnail(img, **f['opts'])
  File "/Users/xaralis/.virtualenvs/adventura/src/sorl-thumbnail/sorl/thumbnail/shortcuts.py", line 8, in get_thumbnail
    return default.backend.get_thumbnail(file_, geometry_string, **options)
  File "/Users/xaralis/.virtualenvs/adventura/src/sorl-thumbnail/sorl/thumbnail/base.py", line 118, in get_thumbnail
    thumbnail)
  File "/Users/xaralis/.virtualenvs/adventura/src/sorl-thumbnail/sorl/thumbnail/base.py", line 151, in _create_thumbnail
    default.engine.write(image, options, thumbnail)
  File "/Users/xaralis/.virtualenvs/adventura/src/sorl-thumbnail/sorl/thumbnail/engines/base.py", line 142, in write
    progressive=progressive
  File "/Users/xaralis/.virtualenvs/adventura/src/sorl-thumbnail/sorl/thumbnail/engines/pil_engine.py", line 227, in _get_raw_data
    image.save(bf, **params)
  File "/Users/xaralis/.virtualenvs/adventura/lib/python2.7/site-packages/PIL/Image.py", line 1468, in save
    save_handler(self, fp, filename)
  File "/Users/xaralis/.virtualenvs/adventura/lib/python2.7/site-packages/PIL/PngImagePlugin.py", line 584, in _save
    raise IOError("cannot use transparency for this mode")
IOError: cannot use transparency for this mode

Which actually leads here: https://github.com/python-imaging/Pillow/blob/master/PIL/PngImagePlugin.py#L584. It looks like the sorl is giving some bad image data to PIL.

Options passed to get_thumbnail are following:

{'quality': 80, 'geometry_string': '1600x1200'}

We are looking forward to using sorl-thumbnail with Django 3.2 on edX.

Required PR - #674 is already merged into the master branch. Do there any chance a new version will be bumped soon?

If any blockers with the release I would be happy to help.

Thank you in advance.

If the path is prefixed with MEDIA_URL it will be replaced with MEDIA_ROOT (updated in 1b0ba0d)

Related to #251

~~The SuspiciousOperation is raised here if the read of the image gets permission denied. In many cases where this is wrong it is because the path given is /media/... which may refer to MEDIA_ROOT not /media.~~

I'm running the latest version (34e1ffa2) and I'm (still) trying to optimize pages with a lot of thumbnails. This time, I'm running into troubles with our Amazon S3 storage. Getting thumbnails stored on there can take over a second per thumbnail unless it's cached, which does not always to seem the case. I hoped the replacement of exists() mentioned in #92 would fix my issue, but alas, it's still taking a very long time. A page with about 15 thumbnails on it can take up to 10-15 seconds to load because of this. I hope you guys can give me any pointers.

I've used the same line profiling decorator as I mentioned in #232 which clearly adds some loading time (over 2 seconds per thumbnail), but it shows a notable difference between a cached thumbnail and one which has to be retrieved from S3:

Example of a cached thumbnail:

File: sorl/thumbnail/base.py
Function: get_thumbnail at line 61
Total time: 0.00121 s

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    61                                               @profile_each_line
    62                                               def get_thumbnail(self, file_, geometry_string, **options):
    63                                                   """
    64                                                   Returns thumbnail as an ImageFile instance for file with geometry and
    65                                                   options given. First it will try to get it from the key value store,
    66                                                   secondly it will create it.
    67                                                   """
    68         1            4      4.0      0.3          logger.debug(text_type('Getting thumbnail for file [%s] at [%s]'), file_,
    69         1            9      9.0      0.7                       geometry_string)
    70         1            1      1.0      0.1          if file_:
    71         1           14     14.0      1.2              source = ImageFile(file_)
    72                                                   elif settings.THUMBNAIL_DUMMY:
    73                                                       return DummyImageFile(geometry_string)
    74                                                   else:
    75                                                       return None
    76
    77                                                   #preserve image filetype
    78         1            6      6.0      0.5          if settings.THUMBNAIL_PRESERVE_FORMAT:
    79                                                       options.setdefault('format', self._get_format(file_))
    80
    81        10           13      1.3      1.1          for key, value in self.default_options.items():
    82         9           12      1.3      1.0              options.setdefault(key, value)
    83
    84
    85                                                   # For the future I think it is better to add options only if they
    86                                                   # differ from the default settings as below. This will ensure the same
    87                                                   # filenames being generated for new options at default.
    88         4            6      1.5      0.5          for key, attr in self.extra_options:
    89         3           10      3.3      0.8              value = getattr(settings, attr)
    90         3            4      1.3      0.3              if value != getattr(default_settings, attr):
    91                                                           options.setdefault(key, value)
    92         1          327    327.0     27.0          name = self._get_thumbnail_filename(source, geometry_string, options)
    93         1           13     13.0      1.1          thumbnail = ImageFile(name, default.storage)
    94         1          787    787.0     65.0          cached = default.kvstore.get(thumbnail)
    95         1            3      3.0      0.2          if cached:
    96         1            1      1.0      0.1              return cached

Example of a thumbnail retrieved from S3:

File: sorl/thumbnail/base.py
Function: get_thumbnail at line 61
Total time: 2.30476 s

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    61                                               @profile_each_line
    62                                               def get_thumbnail(self, file_, geometry_string, **options):
    63                                                   """
    64                                                   Returns thumbnail as an ImageFile instance for file with geometry and
    65                                                   options given. First it will try to get it from the key value store,
    66                                                   secondly it will create it.
    67                                                   """
    68         1            4      4.0      0.0          logger.debug(text_type('Getting thumbnail for file [%s] at [%s]'), file_,
    69         1           14     14.0      0.0                       geometry_string)
    70         1            2      2.0      0.0          if file_:
    71         1           16     16.0      0.0              source = ImageFile(file_)
    72                                                   elif settings.THUMBNAIL_DUMMY:
    73                                                       return DummyImageFile(geometry_string)
    74                                                   else:
    75                                                       return None
    76
    77                                                   #preserve image filetype
    78         1            8      8.0      0.0          if settings.THUMBNAIL_PRESERVE_FORMAT:
    79                                                       options.setdefault('format', self._get_format(file_))
    80
    81        10           16      1.6      0.0          for key, value in self.default_options.items():
    82         9           14      1.6      0.0              options.setdefault(key, value)
    83
    84
    85                                                   # For the future I think it is better to add options only if they
    86                                                   # differ from the default settings as below. This will ensure the same
    87                                                   # filenames being generated for new options at default.
    88         4            7      1.8      0.0          for key, attr in self.extra_options:
    89         3           18      6.0      0.0              value = getattr(settings, attr)
    90         3            6      2.0      0.0              if value != getattr(default_settings, attr):
    91                                                           options.setdefault(key, value)
    92         1          493    493.0      0.0          name = self._get_thumbnail_filename(source, geometry_string, options)
    93         1           20     20.0      0.0          thumbnail = ImageFile(name, default.storage)
    94         1        29609  29609.0      1.3          cached = default.kvstore.get(thumbnail)
    95         1            2      2.0      0.0          if cached:
    96                                                       return cached
    97                                                   else:
    98                                                       # We have to check exists() because the Storage backend does not
    99                                                       # overwrite in some implementations.
   100                                                       # so we make the assumption that if the thumbnail is not cached, it doesn't exist
   101         1           19     19.0      0.0              print 'No cached thumbnail'
   102         1            2      2.0      0.0              try:
   103         1      1078407 1078407.0     46.8                  source_image = default.engine.get_image(source)
   104                                                       except IOError:
   105                                                           if settings.THUMBNAIL_DUMMY:
   106                                                               return DummyImageFile(geometry_string)
   107                                                           else:
   108                                                               # if S3Storage says file doesn't exist remotely, don't try to
   109                                                               # create it and exit early.
   110                                                               # Will return working empty image type; 404'd image
   111                                                               logger.warn(text_type('Remote file [%s] at [%s] does not exist'), file_, geometry_string)
   112                                                               return thumbnail
   113
   114                                                       # We might as well set the size since we have the image in memory
   115         1           15     15.0      0.0              image_info = default.engine.get_image_info(source_image)
   116         1            2      2.0      0.0              options['image_info'] = image_info
   117         1            8      8.0      0.0              size = default.engine.get_image_size(source_image)
   118         1            7      7.0      0.0              source.set_size(size)
   119         1            2      2.0      0.0              try:
   120         1            3      3.0      0.0                  self._create_thumbnail(source_image, geometry_string, options,
   121         1       998498 998498.0     43.3                                         thumbnail)
   122         1           11     11.0      0.0                  self._create_alternative_resolutions(source_image, geometry_string,
   123         1          223    223.0      0.0                                                       options, thumbnail.name)
   124                                                       finally:
   125         1           13     13.0      0.0                  default.engine.cleanup(source_image)
   126
   127                                                   # If the thumbnail exists we don't create it, the other option is
   128                                                   # to delete and write but this could lead to race conditions so I
   129                                                   # will just leave that out for now.
   130         1         1139   1139.0      0.0          default.kvstore.get_or_set(source)
   131         1       196181 196181.0      8.5          default.kvstore.set(thumbnail, source)
   132         1            3      3.0      0.0          return thumbnail

sorl-thumbnail has been a great asset in our Django based web project however the implementation of cropping left a little to be desired (or at least the implementation didn't make a lot of sense to me) so I made some modifications.

I modified the cropping behavior in the following ways:

1. The crop parameter now takes x,y coordinates of the top left corner as well as width/height of the crop region.
2. Cropping occurs before scalling.

Feel free to merge this into the main project if you also feel that this is an improvement. If you have any additional notes I will be happy to make further modifications if this isn't exactly what you had in mind.

Hi,

I would like to see a new version of sorl-thumbnail on PyPI:

• Officially support Django 4.0, 4.1 and Python 3.9, 3.10 on PyPI.
• Remove DeprecationWarning with Pillow up to version 9.2.0 [https://github.com/jazzband/sorl-thumbnail/issues/695].

Commit https://github.com/jazzband/sorl-thumbnail/commit/8d7bd407be95f3dafdc2ad800b51194cada3599e should be included in the new version.

Currently, Pillow 9.0.1 is the last version supported by sorl-thumbnail without DeprecationWarning.

Doc URL: https://sorl-thumbnail.readthedocs.io/en/latest/operation.html

Targeted URL: https://docs.google.com/drawings/edit?id=1wlE4LkQpzXd2a2Nxfjt6_j5NG7889dzMyf0V-xPAJSE&hl=en

Error:

Django 4.1 added a system check to find duplicate template tags at startup. If easy-thumbnails and sorl-thumbnails are installed in the same project, it will detect the possible clash and will exit with an error. This happens even if the tags are not used in any template.

SystemCheckError: System check identified some issues:

ERRORS:
?: (templates.E003) 'thumbnail' is used for multiple template tag modules: 'easy_thumbnails.templatetags.thumbnail', 'sorl.thumbnail.templatetags.thumbnail'

I've added a minimal reproduction here: https://github.com/MarcoGlauser/template_tag_conflict Simply run a manage.py command like makemigrations

Related: https://github.com/SmileyChris/easy-thumbnails/issues/609

When setting the storage, sorl does not allow for a class instance being deconstructible, which is necessary to be able to use backends with arguments.

sorl.thumbnails.images.py

cls = self.storage._wrapped.__class__

For a deconstructible class this can use self.storage.deconstruct (checking whether the class is deconstructible first to ensure backwards compatibility).

deserialize could check whether the storage is a list (deconstructible) or a single value (old method)

Hi,

When developing locally without the media files available, I keep getting FileNotFoundError printed to the console. I'm calling sorl.thumbnail.get_thumbnail directly (i.e, without using the templatetags.

backend_1   |   File "/opt/venv/lib/python3.9/site-packages/sorl/thumbnail/base.py", line 104, in get_thumbnail
backend_1   |     source_image = default.engine.get_image(source)
backend_1   |   File "/opt/venv/lib/python3.9/site-packages/sorl/thumbnail/engines/pil_engine.py", line 72, in get_image
backend_1   |     buffer = BytesIO(source.read())
backend_1   |   File "/opt/venv/lib/python3.9/site-packages/sorl/thumbnail/images.py", line 162, in read
backend_1   |     f = self.storage.open(self.name)
backend_1   |   File "/opt/venv/lib/python3.9/site-packages/django/core/files/storage.py", line 38, in open
backend_1   |     return self._open(name, mode)
backend_1   |   File "/opt/venv/lib/python3.9/site-packages/django/core/files/storage.py", line 243, in _open
backend_1   |     return File(open(self.path(name), mode))
backend_1   | FileNotFoundError: [Errno 2] No such file or directory: '/workspace/media/posts/image-72.png'

I tried to use the DUMMY config, but the error continue.

I believe that before when I was on django==2.2.2 and sorl-thumbnail==12.5.0 this error was getting suppressed somehow.

Using:

         'sorl.thumbnail': {
             'handlers': ['null'],
         },

Works, but I don't want to suppress every sorl-thumbnail error.

Looking at https://github.com/jazzband/sorl-thumbnail/blob/master/sorl/thumbnail/base.py#L106, and comparing with the 12.5.0 version, I couldn't find any difference, so it must be a new Django default that I'm missing.

Does anyone have any ideas here? Thanks!

I think that the configurable KVStore adds complexity to the project without real added value, all the more that now Django also includes a Redis cache backend in core. What about only keeping THUMBNAIL_CACHE to point to some Django-configured cache and tell users to do the cache configuration at the Django project level?

Are there still use cases to implement/specify custom KVStores that are not possible as Django cache backends?

@camilonova, would love to get your input on this idea.