Find whether file exists or not in HDFS using shell script

bash, hadoop, hdfs, linux, shell

Solution

You can try `-test` option to achieve the same.

hdfs dfs -test -[defszrw] HDFS_PATH

- `-d`: if the path is a directory, return 0.

- `-e`: if the path exists, return 0.

Since 2.7.0

- `-f`: if the path is a file, return 0.

- `-s`: if the path is not empty, return 0.

- `-r`: if the path exists and read permission is granted, return 0.

since 2.8.0

- `-w`: if the path exists and write permission is granted, return 0.

- `-z`: if the file is zero-length, return 0.

Example:

if hdfs dfs -test -e $HDFS_PATH; then
    echo "[$HDFS_PATH] exists on HDFS"
    hdfs dfs -ls $HDFS_PATH
fi

Reference: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html#test

Problem

I have a `shell script` like below. This script prints the path of a file located in `HDFS` ``` #!/bin/bash TIMESTAMP=`date "+%Y-%m-%d"` path=/user/$USER/logs/${TIMESTAMP}.fail_log path1=/user/$USER/logs/`date -d "-1 days" '+%Y-%m-%d'`.fail_log echo filePath=$path echo filePath1=$path1 ``` In the script the `paths` provided are `hdfs` locations In this script I am getting the filepath. Now I want to know whether the file actually exists or not in `HDFS`. If the file exists then only print the filepath or else do nothing. How can I do that?

Original source