AM Container is running beyond virtual memory limits

cloudera, hadoop, hadoop-yarn

Solution

Ok, found out. Increase the master memry parameter to more tan 750MB and you will succeed running the YARN app.

Problem

I was playing with distributed shell application (hadoop-2.0.0-cdh4.1.2). This is the error I'm receiving at the moment. ``` 13/01/01 17:09:09 INFO distributedshell.Client: Got application report from ASM for, appId=5, clientToken=null, appDiagnostics=Application application_1357039792045_0005 failed 1 times due to AM Container for appattempt_1357039792045_0005_000001 exited with exitCode: 143 due to: Container [pid=24845,containerID=container_1357039792045_0005_01_000001] is running beyond virtual memory limits. Current usage: 77.8mb of 512.0mb physical memory used; 1.1gb of 1.0gb virtual memory used. Killing container. Dump of the process-tree for container_1357039792045_0005_01_000001 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 24849 24845 24845 24845 (java) 165 12 1048494080 19590 /usr/java/bin/java -Xmx512m org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster --container_memory 128 --num_containers 1 --priority 0 --shell_command ping --shell_args localhost --debug |- 24845 23394 24845 24845 (bash) 0 0 108654592 315 /bin/bash -c /usr/java/bin/java -Xmx512m org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster --container_memory 128 --num_containers 1 --priority 0 --shell_command ping --shell_args localhost --debug 1>/tmp/logs/application_1357039792045_0005/container_1357039792045_0005_01_000001/AppMaster.stdout 2>/tmp/logs/application_1357039792045_0005/container_1357039792045_0005_01_000001/AppMaster.stderr ``` The interesting part is that, there seems to be no problem with the setup, since a simple `ls` or `uname` command completed successfully and the output was available in the container2 stdout. Regarding the setup, `yarn.nodenamager.vmem-pmem-ratio` is `3` and the total physical memory available is 2GB, which I thinks is more than enough for example to run. For the command in question, the "ping localhost" generated two replies, as it can be seen from the `containerlogs/container_1357039792045_0005_01_000002/721917/stdout/?start=-4096.` So, what could be the problem?

Original source