How to create new DataFrame with dict

pyspark

Solution

Thanks everyone for some suggestions, I figured out the other way to resolve my problem with pivot, the code is:

cMap = {"k1" : "v1", "k2" : "v1", "k3" : "v2", "k4" : "v2"}
a_cMap = [(k,)+(v,) for k,v in cMap.items()] 
data = spark.createDataFrame(a_cMap, ['key','val'])

from pyspark.sql.functions import count
data = data.groupBy('key').pivot('val').agg(count('val'))
data.show()

+---+----+----+
|key|  v1|  v2|
+---+----+----+
| k2|   1|null|
| k4|null|   1|
| k1|   1|null|
| k3|null|   1|
+---+----+----+

data = data.na.fill(0)
data.show()

+---+---+---+
|key| v1| v2|
+---+---+---+
| k2|  1|  0|
| k4|  0|  1|
| k1|  1|  0|
| k3|  0|  1|
+---+---+---+

keys = spark.createDataFrame([('k1','2'),('k2','3'),('k3','4'),('k4','5'),('k5','6')], ["key",'temp'])

newDF = keys.join(data,'key')
newDF.show()
+---+----+---+---+
|key|temp| v1| v2|
+---+----+---+---+
| k2|   3|  1|  0|
| k4|   5|  0|  1|
| k1|   2|  1|  0|
| k3|   4|  0|  1|
+---+----+---+---+

But, I can't convert 1 to true, 0 to false.

Problem

I had one `dict`, like: ``` cMap = {"k1" : "v1", "k2" : "v1", "k3" : "v2", "k4" : "v2"} ``` and one DataFrame `A`, like: ``` +---+ |key| +---- | k1| | k2| | k3| | k4| +---+ ``` to create the DataFame above with code: ``` data = [('k1'), ('k2'), ('k3'), ('k4')] A = spark.createDataFrame(data, ['key']) ``` I want to get the new DataFrame, like: ``` +---+----------+----------+ |key| v1 | v2 | +---+----------+----------+ | k1|true |false | | k2|true |false | | k3|false |true | | k4|false |true | +---+----------+----------+ ``` I wish to get some suggestions, thanks!

Original source