PySpark Drop Rows
apache-spark, pyspark, python
Solution
AFAIK there's no 'easy' way to do this.
This should do the trick, though:
val header = data.first
val rows = data.filter(line => line != header)
Problem
how do you drop rows from an RDD in PySpark? Particularly the first row, since that tends to contain column names in my datasets. From perusing the API, I can't seem to find an easy way to do this. Of course I could do this via Bash / HDFS, but I just want to know if this can be done from within PySpark.