WARN snappy.LoadSnappy: Snappy native library not loaded

Question

whatever I do, I can't get rid of this error. I know snappy is a fast and therefore a preferable compression/decompression library over the other options. I would like to use this library for my processing. As far as I know Google uses this internally for their BigTables, MapReduce (basically for all their killer applications). I did research on my own. People suggest not to use it, or java-snappy as an option, but I want to stick with the hadoop snappy. I have the corresponding library in my setup. (I mean under lib) Could someone fix this error? I see that jobs are done successfully regardless this error. ``` ****hdfs://localhost:54310/user/hduser/gutenberg 12/06/01 18:18:54 INFO input.FileInputFormat: Total input paths to process : 3 12/06/01 18:18:54 INFO util.NativeCodeLoader: Loaded the native-hadoop library 12/06/01 18:18:54 WARN snappy.LoadSnappy: Snappy native library not loaded 12/06/01 18:18:54 INFO mapred.JobClient: Running job: job_201206011229_0008 12/06/01 18:18:55 INFO…

Accepted Answer

You'll see this error message if the shared library (.so) for snappy is not located on the `LD_LIBARAY_PATH` / `java.library.path`. If you have the libraries installed in the correct location then you shouldn't see the above error messages.

If you do have the .so installed in the same folder as the hadoop native lib (`libhadoop.so`), then the above 'error' could be related to the node you are submitting your jobs from (like you say, your job doesn't error and this looks like a message on the client side).

Can you share some details of your job configuration (where you configure your output format, and the associated compression options).

WARN snappy.LoadSnappy: Snappy native library not loaded

Solution

Problem