Duplicate rows in a Pyspark Dataframe

Question

Let's say I have a dataframe: ``` df = sqlContext.createDataFrame( [(1, 10, 21.0, 0), (3, 14, -23.0, 1)], ("x1", "x2", "x3", "x4")) df.show() ## +---+---+-----+---+ ## | x1| x2| x3| x4| ## +---+---+-----+---+ ## | 1| 10| 23.0| 5| ## | 3| 14|-23.0| 0| ## +---+---+-----+---+ ``` What would be an efficient way to "duplicate" rows and setting `x4=1` in those duplicates and have: ``` ## +---+---+-----+---+ ## | x1| x2| x3| x4| ## +---+---+-----+---+ ## | 1| 10| 23.0| 5| ## | 1| 10| 23.0| 1| ## | 3| 14|-23.0| 0| ## | 3| 14|-23.0| 1| ## +---+---+-----+---+ ``` In Apache PIG the analog would be simple: do a foreach and generate: ``` FLATTEN(TOBAG(1, x4)) AS x4 ``` Thank you all

Accepted Answer

Import required functions from `pyspark.sql.functions`:

```
from pyspark.sql.functions import array, explode, lit
```

and replace existing column:

```
df.withColumn("x4", explode(array(lit(1), df["x4"])))
```

Duplicate rows in a Pyspark Dataframe

Solution

Problem