Writing a sparkdataframe to a .csv file in S3 and choose a name in pyspark

Question

I have a dataframe and a i am going to write it an a .csv file in S3 i use the following code: ``` df.coalesce(1).write.csv("dbfs:/mnt/mount1/2016//product_profit_weekly",mode='overwrite',header=True) ``` it puts a .csv file in product_profit_weekly folder , at the moment .csv file has a weired name in S3 , is it possible to choose a file name when i am going to write it?

Accepted Answer

All spark dataframe writers (df.write.___) don't write to a single file, but write one chunk per partition. I imagine what you get is a directory called

```
df.coalesce(1).write.csv("dbfs:/mnt/mount1/2016//product_profit_weekly
```

and one file inside called

```
part-00000
```

In this case, you are doing something that could be quite inefficient and not very "sparky" -- you are coalescing all dataframe partitions to one, meaning that your task isn't actually executed in parallel!

Here's a different model. To take advantage of all spark parallelization, which means DON'T coalesce, and write in parallel to some directory.

If you have 100 partitions, you will get:

```
part-00000
part-00001
...
part-00099
```

If you need everything in one flat file, write a little function to merge it after the fact. You could either do this in scala, or in bash with:

```
cat ${dir}.part-* > $flatFilePath
```

Writing a sparkdataframe to a .csv file in S3 and choose a name in pyspark

Solution

Problem