Why didn't my rails app on heroku restart itself?
heroku, ruby-on-rails
Solution
Heroku's support answer suggests restarting your app manually with `heroku restart`. They're fixing the issue right now.
Hi, A process management error on our side caused some crashed apps only running 1 web dyno to be reported as "idle" even though they were actually crashed. This means that the crashed dyno was never restarted, causing subsequent requests to fail. We've identified this problem and are implementing a fix. If your app is still unresponsive, please try restarting it with the heroku restart command. Please let us know if you need more help. Thanks, Heroku Support
Problem
I manage a rails app for a client of mine and recently it went down. The site was down for 9 hours before I noticed. I checked the logs and every request for the past 9 hours is prepended with the following code: ``` at=error code=H10 desc="App crashed" ``` Before that, I see the following logs: ``` 2012-11-16T00:55:46+00:00 heroku[web.1]: Idling 2012-11-16T00:55:50+00:00 heroku[web.1]: Stopping all processes with SIGTERM 2012-11-16T00:55:51+00:00 app[web.1]: [2012-11-16 00:55:51] ERROR SignalException: SIGTERM 2012-11-16T00:55:51+00:00 app[web.1]: /usr/local/lib/ruby/1.9.1/webrick/server.rb:90:in `select' 2012-11-16T00:56:00+00:00 heroku[web.1]: Error R12 (Exit timeout) -> At least one process failed to exit within 10 seconds of SIGTERM 2012-11-16T00:56:00+00:00 heroku[web.1]: Stopping remaining processes with SIGKILL 2012-11-16T00:56:02+00:00 heroku[web.1]: State changed from up to down 2012-11-16T00:56:02+00:00 heroku[web.1]: Process exited with status 137 2012-11-16T01:03:55+00:00 heroku[web.1]: Unidling 2012-11-16T01:03:55+00:00 heroku[web.1]: State changed from down to starting 2012-11-16T01:03:59+00:00 heroku[web.1]: Starting process with command `bundle exec rails server -p 4303` 2012-11-16T01:04:00+00:00 heroku[nginx]: 98.139.241.251 - - [16/Nov/2012:01:04:00 +0000] "GET / HTTP/1.1" 499 0 "-" "YahooCacheSystem" domain.com 2012-11-16T01:04:22+00:00 app[web.1]: => Ctrl-C to shutdown server 2012-11-16T01:04:22+00:00 app[web.1]: ** [NewRelic][11/16/12 01:04:21 +0000 b8af98a1-2246-4b34-9dfe-61b9d4b747bc (2)] INFO : Dispatcher: webrick 2012-11-16T01:04:22+00:00 app[web.1]: ** [NewRelic][11/16/12 01:04:21 +0000 b8af98a1-2246-4b34-9dfe-61b9d4b747bc (2)] INFO : Application: acsolar 2012-11-16T01:04:22+00:00 app[web.1]: ** [NewRelic][11/16/12 01:04:21 +0000 b8af98a1-2246-4b34-9dfe-61b9d4b747bc (2)] INFO : New Relic Ruby Agent 3.4.0.1 Initialized: pid = 2 2012-11-16T01:04:22+00:00 app[web.1]: => Booting WEBrick 2012-11-16T01:04:22+00:00 app[web.1]: => Rails 3.1.1 application starting in production on http://0.0.0.0:4303 2012-11-16T01:04:22+00:00 app[web.1]: => Call with -d to detach 2012-11-16T01:04:25+00:00 app[web.1]: [DEPRECATION] Your applications public directory contains an assets/products and/or assets/taxons subdirectory. 2012-11-16T01:04:25+00:00 app[web.1]: Run `rake spree:assets:relocate_images` to relocate the images. 2012-11-16T01:04:34+00:00 app[web.1]: ** [NewRelic][11/16/12 01:04:32 +0000 b8af98a1-2246-4b34-9dfe-61b9d4b747bc (2)] INFO : Reporting performance data every 60 seconds. 2012-11-16T01:04:34+00:00 app[web.1]: Connected to NewRelic Service at collector-5.newrelic.com 2012-11-16T01:05:00+00:00 heroku[web.1]: Error R10 (Boot timeout) -> Web process failed to bind to $PORT within 60 seconds of launch 2012-11-16T01:05:00+00:00 heroku[web.1]: Stopping process with SIGKILL 2012-11-16T01:05:02+00:00 heroku[web.1]: Process exited with status 137 2012-11-16T01:05:02+00:00 heroku[web.1]: State changed from crashed to down 2012-11-16T01:05:02+00:00 heroku[web.1]: State changed from starting to crashed ``` I'm guessing that it may have spun down and had an error booting back up, but how come it stayed in the crashed state without restarting itself? Is there anything I can do to have it automatically restart if this happens again in the future? I've got NewRelic running on this too and it didn't notify me at all, but that's another problem I'll have to investigate.