Stop asking for password when installing gems
bundler, ruby-on-rails, rubygems, rvm
Solution
Absolutely has to do with the system ruby, not RVM, unless you installed RVM using the Multi-User installation type. If you did that and its still asking you for your password, then you installed as root, strictly against what the listed documentation states, and your general user was not added to the 'rvm' group the installer creates. (NOTE: This is based on the idea that you want a multi-user install, not a single user one. If you want the single user install, not not prefix with sudo when you run the installer.)
Rip out RVM, log out then back in (to ensure a completely fresh reinitialization of the environment), and then rerun the installer command as your regular user, not as root, prefixing with 'sudo' as the documentation instructs.
If you do not have RVM installed, then follow the documentation at https://rvm.io to install either as a single user install, or as a multi-user install. In this case, without RVM installed, what Billy Chan described above is your fix, though I would suggest tightening the rules a bit by figuring out which exact set of commands (gem bin names) you need to run on a regular basis and adding entries for those in the sudoers file (visudo).
Right now the problem reads you trying to use the system ruby which *RVM does NOT control (it simply allows you access to it by setting the proper GEM_PATH, RUBY_* environment variables etc), or your RVM multi-user install was done incorrectly.
Problem
Whenever I bundle my rails 3.2 gems, it asks me for my password: Enter your password to install the bundled RubyGems to your system This gets really annoying, especially when bundling several times in one project. However, when I set the gem directory to world-readable, it always gives me a warning when executing any (!) rails command. This is even more annoying, of course. How can I turn this off?