I am trying to run some code, but getting error:
'DataFrame' object has no attribute '_get_object_id'
The code:
items = [(1,12),(1,float('Nan')),(1,14),(1,10),(2,22),(2,20),(2,float('Nan')),(3,300), (3,float('Nan'))] sc = spark.sparkContext rdd = sc.parallelize(items) df = rdd.toDF(["id", "col1"]) import pyspark.sql.functions as func means = df.groupby("id").agg(func.mean("col1")) # The error is thrown at this line df = df.withColumn("col1", func.when((df["col1"].isNull()), means.where(func.col("id")==df["id"])).otherwise(func.col("col1"))) 2 Answers
You can't reference a second spark DataFrame inside a function, unless you're using a join. IIUC, you can do the following to achieve your desired result.
Suppose that means is the following:
#means.show() #+---+---------+ #| id|avg(col1)| #+---+---------+ #| 1| 12.0| #| 3| 300.0| #| 2| 21.0| #+---+---------+ Join df and means on the id column, then apply your when condition
from pyspark.sql.functions import when df.join(means, on="id")\ .withColumn( "col1", when( (df["col1"].isNull()), means["avg(col1)"] ).otherwise(df["col1"]) )\ .select(*df.columns)\ .show() #+---+-----+ #| id| col1| #+---+-----+ #| 1| 12.0| #| 1| 12.0| #| 1| 14.0| #| 1| 10.0| #| 3|300.0| #| 3|300.0| #| 2| 21.0| #| 2| 22.0| #| 2| 20.0| #+---+-----+ But in this case, I'd actually recommend using a Window with pyspark.sql.functions.mean:
from pyspark.sql import Window from pyspark.sql.functions import col, mean df.withColumn( "col1", when( col("col1").isNull(), mean("col1").over(Window.partitionBy("id")) ).otherwise(col("col1")) ).show() #+---+-----+ #| id| col1| #+---+-----+ #| 1| 12.0| #| 1| 10.0| #| 1| 12.0| #| 1| 14.0| #| 3|300.0| #| 3|300.0| #| 2| 22.0| #| 2| 20.0| #| 2| 21.0| #+---+-----+ I think you are using Scala API, in which you use (). In PySpark, use [] instead.
0
