CarrierWave with plupload direct to s3 - how to avoid redundant download/upload?

carrierwave, ruby-on-rails

Solution

So I have found a solution. Thanks to the contributors for getting it started. I want to make sure credit is given where it is due but the answer was a bit more involved than what was proposed.

Here is what worked....

1) After plupload finishes uploading to S3, I assigned the file attribute (my mounted CarrierWave uploader) to the returned key from S3. This is accomplished in plupload by using the FileUploaded event to post the response to another action:

init : {
    FileUploaded: function(up, file, info) {
        $.post('#{media_path}', {s3_response: info.response});
    }
}

2) The action picks up that response and assigns the file attribute(or whatever the name of your mounted uploader is) via a method on the model:

def create
  med = Medium.new_from_s3_response(params[:s3_response])
end

def self.new_from_s3_response(response)
  med = Medium.new

  # Grab filename from response use libxml-ruby gem
  xml = XML::Parser.string(response).parse
  node = xml.find_first('//PostResponse/Key')
  med['file'] = node.content
  med.save

  # Queue up a job to have the file downloaded and processed
  # Note that I am using Mongoid so the handy serialization by DelayedJob doesn't work
  Delayed::Job.enqueue MediaJob.new(med.id)
end

3) The job fires and calls process_file which downloads the original and creates/uploads the various versions. Note that the original does not get uploaded. Mission accomplished!

class MediaJob < Struct.new(:medium_id)
  def perform
    medium = Medium.find(medium_id)
    medium.process_file

  end
end  

def process_file
  # Download from S3
  file.cache_stored_file!
  # Convert to sanitized file for creating versions
  file.retrieve_from_cache!(file.cache_name)

  # Loop through versions and create them....
  file.versions.values.each do |uploader|
    uploader.store!
  end
end

I got the code for creating the versions from a wiki entry on the CarrierWave GitHub page.

Problem

I have recently finished setting up plupload to upload files directly to S3. This is done using S3s posting method in combination with plupload. After the upload is finished, I queue a delayed job that loads up a model and uses the remote_[uploader]_url method of CarrierWave to assign that uploaded resource to the model. I would like to use CarrierWave because of the simplicity of managing multiple versions. Here is my job (slightly edited, I realize I could use send_later for this simple example): ``` class MediaJob < Struct.new(:medium) def perform medium.process_file medium.save end end ``` Here is the process_file method: ``` def process_file self.remote_file_url = s3_original_url end ``` The result is that the file ends up moving in and out of S3 three times: 1) Uploaded to S3 with plupload 2) Downloaded by CarrierWave for processing 3) Re-uploaded by CarrierWave when the model is saved This is all well and good for a small file but gets very expensive when dealing with larger files. There must be a way to eliminate #3. I don't want CarrierWave to upload the original file when it saves since the original file is already there. I do want the versions to be uploaded though.

Original source