How to bundle many files in S3 using Spark
amazon-s3, apache-spark, hadoop, scala
Solution
have you tried something along the lines of sc.wholeTextFiles?
It creates an RDD where the key is the filename and the value is the byte array of the whole file. You can then map this so the key is the file date, and then groupByKey?
http://spark.apache.org/docs/latest/programming-guide.html
Problem
I have 20 million files in S3 spanning roughly 8000 days. The files are organized by timestamps in UTC, like this: `s3://mybucket/path/txt/YYYY/MM/DD/filename.txt.gz`. Each file is UTF-8 text containing between 0 (empty) and 100KB of text (95th percentile, although there are a few files that are up to several MBs). Using Spark and Scala (I'm new to both and want to learn), I would like to save "daily bundles" (8000 of them), each containing whatever number of files were found for that day. Ideally I would like to store the original filenames as well as their content. The output should reside in S3 as well and be compressed, in some format that is suitable for input in further Spark steps and experiments. One idea was to store bundles as a bunch of JSON objects (one per line and `'\n'`-separated), e.g. ``` {id:"doc0001", meta:{x:"blah", y:"foo", ...}, content:"some long string here"} {id:"doc0002", meta:{x:"foo", y:"bar", ...}, content: "another long string"} ``` Alternatively, I could try the Hadoop SequenceFile, but again I'm not sure how to set that up elegantly. Using the Spark shell for example, I saw that it was very easy to read the files, for example: ``` val textFile = sc.textFile("s3n://mybucket/path/txt/1996/04/09/*.txt.gz") // or even val textFile = sc.textFile("s3n://mybucket/path/txt/*/*/*/*.txt.gz") // which will take for ever ``` But how do I "intercept" the reader to provide the file name? Or perhaps I should get an RDD of all the files, split by day, and in a reduce step write out `K=filename, V=fileContent`?