Possible memory issue crashing Hbase Thrift Server

cloudera, crash, hbase, memory, thrift

Solution

Using /etc/hbase/conf/hbase-env.sh, I increased my heap size, and this addressed the crashing issue.

# The maximum amount of heap to use, in MB. Default is 1000.
export HBASE_HEAPSIZE=8000

Thanks to Harsh J on the CDH Users mailing list for helping me figure out. As he pointed out, my lack of log messages indicates a `kill -9` is probably taking place:

Indeed if a shutdown handler message is missing in the log tail pre-crash, there may have been a kill -9 passed to the process via the OOM handler.

Problem

I'm running Cloudera CDH4 with Hbase and Hbase Thrift Server. Several times a day, the Thrift Server crashes. In /var/log/hbase/hbase-hbase-thrift-myserver.out, there is this: ``` # # java.lang.OutOfMemoryError: Java heap space # -XX:OnOutOfMemoryError="kill -9 %p" # Executing /bin/sh -c "kill -9 8151"... ``` In /var/log/hbase/hbase-hbase-thrift-myserver.log, there are no error messages at the end of the file. There are only a lot of DEBUG messages stating that one of the nodes is caching a particular file. I can't figure out any configuration options for the Hbase Thrift Server. There are no obvious files in /etc/. Just /etc/hbase/conf and its Hbase files. Any ideas on debugging?

Original source