How to measure how much memory each gem requires at initialization?

bundler, memory, ruby, ruby-on-rails, rubygems

Solution

We had a problem in which our basic Rails app, with no traffic or requests, had a footprint of ~140MB on startup.

We used the following approach to trace the memory requirement of each gem specified in the Gemfile of our app, without having to try to patch bundler.

- using `rails new myappname` generate a new empty rails app

- Copy the Gemfile from the main project to this new rails project

- run `bundle install` and then `rails server` to ensure it is possible to boot up the rails server and that any basic configurations required are loaded

- Open up the Gemfile and with the exception of the specification for the rails gem, append `require: false` at the end of each line. Ensure that any other gems that are specified with one name but required using :require => 'othergemname' are using the older ruby Hash notation so that the pattern match below will catch it.

- Run `bundle install` again to regenerate the Gemfile.lock

Create the following script which will use manually require each gem specified in the Gemfile and log the system memory consumed by the rails process before and after.

# require_and_profile.rb
def require_and_profile(gemname = nil)
  unless gemname
    puts "%-20s: %10s | %10s" % ['gem','increment','total']
    return
  end
  # This is how to get memory of calling process in OS X, check host OS for variants
  memory_usage = `ps -o rss= -p #{Process.pid}`.to_i / 1024.0
  require gemname
  puts "%-20s: %10.2f | %10.2f" % [ gemname, (`ps -o rss= -p #{Process.pid}`.to_i / 1024.0 - memory_usage), (`ps -o rss= -p #{Process.pid}`.to_i / 1024.0)]
end

pattern = /^[^#]*gem[ ]*['"]([^,'"]*)['"][ ,~>0-9\.'"]*(:require[ => ]*['"]([^'"]*)['"][, ])?/

require_and_profile
File.open('Gemfile').each do |line|
  if line.match(pattern)
  if line.match(pattern)[3]
    require_and_profile line.match(pattern)[3]
  else
      require_and_profile line.match(pattern)[1]
    end
  end
end

Run `rails c`

- `load 'require_and_profile.rb'`

- The output shows how much (in MB) each gem adds to the base app footprint (increment) and what the total footprint is after inclusion of the gem (total).

This helped us identify for example, that we'd been requiring asset-sync in our boot when we only needed it in the :asset group. We do find that on different boot-ups the memory footprint of each gem is not exactly the same, but running it a few times does show you the patterns of which ones are the memory-hungry gems.

Problem

Original source