chef-solo: Installing build-essential recipe before postgresql
chef-infra, postgresql, ruby
Solution
The simplest solution is to update your local postgresql cookbook. The issue you observe is already fixed in recent versions of the cookbook (all versions >= 1.0.0).
Now how did they solve it? At first, you'll notice that the client recipe was split up into `client.rb` and `ruby.rb`. The former just installs the OS packages, the latter the Ruby gem for usage in the subsequent Chef run. As you observed, the installation of that gem requires the build-essentials to be installed before. This is controlled by a new attribute used by the build-essential cookbook called `node['build-essential']['compile_time']`. This makes sure that the required packages are installed during compile time, i.e., directly when the build-essential recipe is included.
To understand why this is important, you have to understand a bit about the Anatomy of a Chef Run, namely the difference between the compile time when the source code of the recipes is executed and the resource collection for later conversion is build and the following convergence where the resources are executed to move the system towards the intended state.
Generally, most actual changes are done in the convergence phase. Sometimes it is however necessary to build something before, in the compile phase, to be able to use it to decide which resources to define exactly as is the case here. As we install to `pg` gem this early, we can use it to talk to database early and e.g. decide whether to create a database or not.
Problem
I'm trying to install postgresql on my server using chef-solo. I've had this working by doing the bootstrap myself and including the build-esential package manually in a script. Now I want to use knife-solo to handle the bootstrapping and I'm having a problem defining the order in which recipes are executed. I want to install the build-essential package before postgresql. Here is my json file for the server I'm buidling ``` { "dbuser": "myuser", "dbpass": "mypassword", "postgresql":{"password":{"postgres":"mypassword"}}, "passenger":{"version":"3.0.14", "max_pool_size":"2"}, "default_attributes":{"build_essential":{"compiletime": "true"}}, "run_list": [ "recipe[build-essential]", "role[web_server]", "recipe[sqlite]", "recipe[postgresql::server]", "recipe[pope::default]" ] } ``` When I issue the command ``` knife cook <user>@<my server's ip> ``` I get this error ``` WARNING: No knife configuration file found Checking cookbook syntax... Checking Chef version Enter the password for test@<my server's ip>: [2012-10-29T16:14:44+11:00] INFO: *** Chef 10.16.2 *** [2012-10-29T16:14:45+11:00] INFO: Setting the run_list to ["recipe[build-essential]", "role[web_server]", "recipe[sqlite]", "recipe[postgresql::server]", "recipe[mysettings::default]"] from JSON [2012-10-29T16:14:45+11:00] INFO: Run List is [recipe[build-essential], role[web_server], recipe[sqlite], recipe[postgresql::server], recipe[pope::default]] [2012-10-29T16:14:45+11:00] INFO: Run List expands to [build-essential, apache2, passenger_apache2, sqlite, postgresql::server, mysettings::default] [2012-10-29T16:14:45+11:00] INFO: Starting Chef Run for test-VirtualBox [2012-10-29T16:14:45+11:00] INFO: Running start handlers [2012-10-29T16:14:45+11:00] INFO: Start handlers complete. [2012-10-29T16:14:46+11:00] INFO: Processing package[postgresql-client] action install (postgresql::client line 37) [2012-10-29T16:14:47+11:00] INFO: Processing package[libpq-dev] action install (postgresql::client line 37) [2012-10-29T16:14:47+11:00] INFO: Processing package[make] action install (postgresql::client line 37) [2012-10-29T16:14:47+11:00] INFO: Processing gem_package[pg] action install (postgresql::client line 42) ================================================================================ Error executing action `install` on resource 'gem_package[pg]' ================================================================================ Mixlib::ShellOut::ShellCommandFailed ------------------------------------ Expected process to exit with [0], but received '1' ---- Begin output of /opt/chef/embedded/bin/gem install pg -q --no-rdoc --no-ri -v "0.14.1" ---- STDOUT: Building native extensions. This could take a while... STDERR: ERROR: Error installing pg: ERROR: Failed to build gem native extension. /opt/chef/embedded/bin/ruby extconf.rb checking for pg_config... yes Using config values from /usr/bin/pg_config checking for libpq-fe.h... yes checking for libpq/libpq-fs.h... yes checking for pg_config_manual.h... yes checking for PQconnectdb() in -lpq... no checking for PQconnectdb() in -llibpq... no checking for PQconnectdb() in -lms/libpq... no Can't find the PostgreSQL client library (libpq) *** extconf.rb failed *** Could not create Makefile due to some reason, probably lack of necessary libraries and/or headers. Check the mkmf.log file for more details. You may need configuration options. Provided configuration options: --with-opt-dir --with-opt-include --without-opt-include=${opt-dir}/include --with-opt-lib --without-opt-lib=${opt-dir}/lib --with-make-prog --without-make-prog --srcdir=. --curdir --ruby=/opt/chef/embedded/bin/ruby --with-pg --without-pg --with-pg-dir --without-pg-dir --with-pg-include --without-pg-include=${pg-dir}/include --with-pg-lib --without-pg-lib=${pg-dir}/lib --with-pg-config --without-pg-config --with-pg_config --without-pg_config --with-pqlib --without-pqlib --with-libpqlib --without-libpqlib --with-ms/libpqlib --without-ms/libpqlib Gem files will remain installed in /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/pg-0.14.1 for inspection. Results logged to /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/pg-0.14.1/ext/gem_make.out ---- End output of /opt/chef/embedded/bin/gem install pg -q --no-rdoc --no-ri -v "0.14.1" ---- Ran /opt/chef/embedded/bin/gem install pg -q --no-rdoc --no-ri -v "0.14.1" returned 1 Cookbook Trace: --------------- /tmp/chef-solo/cookbooks/postgresql/recipes/client.rb:44:in `from_file' /tmp/chef-solo/cookbooks/postgresql/recipes/server.rb:24:in `from_file' Resource Declaration: --------------------- # In /tmp/chef-solo/cookbooks/postgresql/recipes/client.rb 42: gem_package "pg" do 43: action :nothing 44: end.run_action(:install) Compiled Resource: ------------------ # Declared in /tmp/chef-solo/cookbooks/postgresql/recipes/client.rb:42:in `from_file' gem_package("pg") do provider Chef::Provider::Package::Rubygems action [:nothing] retries 0 retry_delay 2 package_name "pg" version "0.14.1" cookbook_name :postgresql recipe_name "client" gem_binary "/opt/chef/embedded/bin/gem" end ================================================================================ Recipe Compile Error in /tmp/chef-solo/cookbooks/postgresql/recipes/server.rb ================================================================================ Mixlib::ShellOut::ShellCommandFailed ------------------------------------ gem_package[pg] (postgresql::client line 42) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1' ---- Begin output of /opt/chef/embedded/bin/gem install pg -q --no-rdoc --no-ri -v "0.14.1" ---- STDOUT: Building native extensions. This could take a while... STDERR: ERROR: Error installing pg: ERROR: Failed to build gem native extension. /opt/chef/embedded/bin/ruby extconf.rb checking for pg_config... yes Using config values from /usr/bin/pg_config checking for libpq-fe.h... yes checking for libpq/libpq-fs.h... yes checking for pg_config_manual.h... yes checking for PQconnectdb() in -lpq... no checking for PQconnectdb() in -llibpq... no checking for PQconnectdb() in -lms/libpq... no Can't find the PostgreSQL client library (libpq) *** extconf.rb failed *** Could not create Makefile due to some reason, probably lack of necessary libraries and/or headers. Check the mkmf.log file for more details. You may need configuration options. Provided configuration options: --with-opt-dir --with-opt-include --without-opt-include=${opt-dir}/include --with-opt-lib --without-opt-lib=${opt-dir}/lib --with-make-prog --without-make-prog --srcdir=. --curdir --ruby=/opt/chef/embedded/bin/ruby --with-pg --without-pg --with-pg-dir --without-pg-dir --with-pg-include --without-pg-include=${pg-dir}/include --with-pg-lib --without-pg-lib=${pg-dir}/lib --with-pg-config --without-pg-config --with-pg_config --without-pg_config --with-pqlib --without-pqlib --with-libpqlib --without-libpqlib --with-ms/libpqlib --without-ms/libpqlib Gem files will remain installed in /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/pg-0.14.1 for inspection. Results logged to /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/pg-0.14.1/ext/gem_make.out ---- End output of /opt/chef/embedded/bin/gem install pg -q --no-rdoc --no-ri -v "0.14.1" ---- Ran /opt/chef/embedded/bin/gem install pg -q --no-rdoc --no-ri -v "0.14.1" returned 1 Cookbook Trace: --------------- /tmp/chef-solo/cookbooks/postgresql/recipes/client.rb:44:in `from_file' /tmp/chef-solo/cookbooks/postgresql/recipes/server.rb:24:in `from_file' Relevant File Content: ---------------------- /tmp/chef-solo/cookbooks/postgresql/recipes/client.rb: 37: package pg_pack do 38: action :nothing 39: end.run_action(:install) 40: end 41: 42: gem_package "pg" do 43: action :nothing 44>> end.run_action(:install) 45: [2012-10-29T16:15:03+11:00] ERROR: Running exception handlers [2012-10-29T16:15:03+11:00] ERROR: Exception handlers complete [2012-10-29T16:15:03+11:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out [2012-10-29T16:15:03+11:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: gem_package[pg] (postgresql::client line 42) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1' ---- Begin output of /opt/chef/embedded/bin/gem install pg -q --no-rdoc --no-ri -v "0.14.1" ---- STDOUT: Building native extensions. This could take a while... STDERR: ERROR: Error installing pg: ERROR: Failed to build gem native extension. /opt/chef/embedded/bin/ruby extconf.rb checking for pg_config... yes Using config values from /usr/bin/pg_config checking for libpq-fe.h... yes checking for libpq/libpq-fs.h... yes checking for pg_config_manual.h... yes checking for PQconnectdb() in -lpq... no checking for PQconnectdb() in -llibpq... no checking for PQconnectdb() in -lms/libpq... no Can't find the PostgreSQL client library (libpq) *** extconf.rb failed *** Could not create Makefile due to some reason, probably lack of necessary libraries and/or headers. Check the mkmf.log file for more details. You may need configuration options. Provided configuration options: --with-opt-dir --with-opt-include --without-opt-include=${opt-dir}/include --with-opt-lib --without-opt-lib=${opt-dir}/lib --with-make-prog --without-make-prog --srcdir=. --curdir --ruby=/opt/chef/embedded/bin/ruby --with-pg --without-pg --with-pg-dir --without-pg-dir --with-pg-include --without-pg-include=${pg-dir}/include --with-pg-lib --without-pg-lib=${pg-dir}/lib --with-pg-config --without-pg-config --with-pg_config --without-pg_config --with-pqlib --without-pqlib --with-libpqlib --without-libpqlib --with-ms/libpqlib --without-ms/libpqlib Gem files will remain installed in /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/pg-0.14.1 for inspection. Results logged to /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/pg-0.14.1/ext/gem_make.out ---- End output of /opt/chef/embedded/bin/gem install pg -q --no-rdoc --no-ri -v "0.14.1" ---- Ran /opt/chef/embedded/bin/gem install pg -q --no-rdoc --no-ri -v "0.14.1" returned 1 ```