Jenkins Build Never Finishing
bash, jenkins, oracle
Solution
This turned out to be something horrible :-)
After finishing the work, Jenkins tries to kill all processes it spawned. To identify them, it goes through all processes in the OS, reads from `/proc/<pid>/environ` (this is a Linux box) which contains the process’ environment variables and compares them with the environment it sets to Jenkins processes.
Problem was there was one particular Oracle process running on our db server where if you tried to read from `/proc/pid/environ` for it, it would just hang forever – which is where the Jenkins code would get stuck.
I have no idea why it was getting stuck like this and nor did our DBA. We restarted it and now it works.
Problem
I have a Jenkins master/slave set up which has been working quite happily, running Oracle imports on some Linux boxes. I have just added a new slave node and tried to run our existing database import job on this new node. This job consists of three subprojects; the first one runs some execute shells, copying files and changing permissions and this currently completes successfully, the second runs an execute shell which ends with an Oracle impdp. The impdp completes (the db exists and `ps -ef` no longer shows impdp running) but the Jenkins subproject never finishes. The UI just sits there with the clock whirring around. I've tried adding an echo after the impdp, and this also executes correctly, but the subproject still never finishes. If I add a Post-Build email notification, it is not sent. The third subproject is never reached. What could be the cause of this and how do I debug what is happening?