I have a dataset in the following way:

FieldA FieldB ArrayField 1 A {1,2,3} 2 B {3,5}

I would like to explode the data on ArrayField so the output will look in the following way:

FieldA FieldB ExplodedField 1 A 1 1 A 2 1 A 3 2 B 3 2 B 5

I mean I want to generate an output line for each item in the array the in ArrayField while keeping the values of the other fields.

How would you implement it in Spark. Notice that the input dataset is very large.

4

3 Answers

The explode function should get that done.

pyspark version:

>>> df = spark.createDataFrame([(1, "A", [1,2,3]), (2, "B", [3,5])],["col1", "col2", "col3"]) >>> from pyspark.sql.functions import explode >>> df.withColumn("col3", explode(df.col3)).show() +----+----+----+ |col1|col2|col3| +----+----+----+ | 1| A| 1| | 1| A| 2| | 1| A| 3| | 2| B| 3| | 2| B| 5| +----+----+----+

Scala version

scala> val df = Seq((1, "A", Seq(1,2,3)), (2, "B", Seq(3,5))).toDF("col1", "col2", "col3") df: org.apache.spark.sql.DataFrame = [col1: int, col2: string ... 1 more field] scala> df.withColumn("col3", explode($"col3")).show() +----+----+----+ |col1|col2|col3| +----+----+----+ | 1| A| 1| | 1| A| 2| | 1| A| 3| | 2| B| 3| | 2| B| 5| +----+----+----+
1

You can use explode function Below is the simple example for your case 

import org.apache.spark.sql.functions._ import spark.implicits._ val data = spark.sparkContext.parallelize(Seq( (1, "A", List(1,2,3)), (2, "B", List(3, 5)) )).toDF("FieldA", "FieldB", "FieldC") data.withColumn("ExplodedField", explode($"FieldC")).drop("FieldC")

Hope this helps!

0

explode does exactly what you want. Docs:

Also, here is an example from a different question using it:

0