Pointers on using celery with sorl-thumbnails with remote storages?
amazon-s3, celery, django, python, sorl-thumbnail
Solution
I think what you want to do is set THUMBNAIL_BACKEND to a custom class that overrides the _create_thumbnail method. Instead of generating the thumbnail in that function, kick of a celery task that calls _create_thumbnail with the same arguments as given to the function. The thumbnail won't be available during the request, but it will get generated in the background.
Problem
I'm surprised I don't see anything but "use celery" when searching for how to use celery tasks with sorl-thumbnails and S3. The problem: using remote storages causes massive delays when generating thumbnails (think 100s+ for a page with many thumbnails) while the thumbnail engine downloads originals from remote storage, crunches them, then uploads back to s3. Where is a good place to set up the celery task within sorl, and what should I call? Any of your experiences / ideas would be greatly appreciated. I will start digging around Sorl internals to find a more useful place to delay this task, but there are a few more things I'm curious about if this has been solved before. What image is returned immediately? Sorl must be told somehow that the image returned is not the real thumbnail. The cache must be invalidated when celery finishes the task. Handle multiple thumbnail generation requests cleanly (only need the first one for a given cache key) For now, I've temporarily solved this by using an nginx reverse proxy cache that can serve hits while the backend spends time generating expensive pages (resizing huge PNGs on a huge product grid) but it's a very manual process.