Get HDFS file path in PySpark for files in sequence file format

Question

My data on HDFS is in Sequence file format. I am using PySpark (Spark 1.6) and trying to achieve 2 things: Data path contains a timestamp in yyyy/mm/dd/hh format that I would like to bring into the data itself. I tried SparkContext.wholeTextFiles but I think that might not support Sequence file format. How do I deal with the point above if I want to crunch data for a day and want to bring in the date into the data? In this case I would be loading data like yyyy/mm/dd/* format. Appreciate any pointers.

Accepted Answer

If stored types are compatible with SQL types and you use Spark 2.0 it is quite simple. Import `input_file_name`:

```
from pyspark.sql.functions import input_file_name 
```

Read file and convert to a `DataFrame`:

```
df = sc.sequenceFile("/tmp/foo/").toDF()
```

Add file name:

```
df.withColumn("input", input_file_name())
```

If this solution is not applicable in your case then universal one is to list files directly (for HDFS you can use `hdfs3` library):

```
files = ...
```

read one by one adding file name:

```
def read(f):
    """Just to avoid problems with late binding"""
    return sc.sequenceFile(f).map(lambda x: (f, x))

rdds = [read(f) for f in files]
```

and union:

```
sc.union(rdds)
```

Get HDFS file path in PySpark for files in sequence file format

Solution

Problem