NameError: name 'SparkSession' is not defined

I'm new to cask cdap and Hadoop environment.

I'm creating a pipeline and I want to use a PySpark Program. I have all the script of the spark program and it works when I test it by command like, insted it doesn't if I try to copy- paste it in a cdap pipeline.

It gives me an error in the logs:

NameError: name 'SparkSession' is not defined 

My script starts in this way:

from pyspark.sql import * spark = SparkSession.builder.getOrCreate() from pyspark.sql.functions import trim, to_date, year, month sc= SparkContext() 

How can I fix it?

1 Answer

Spark connects with the local running spark cluster through SparkContext. A better explanation can be found here .

To initialise a SparkSession, a SparkContext has to be initialized. One way to do that is to write a function that initializes all your contexts and a spark session.

def init_spark(app_name, master_config): """ :params app_name: Name of the app :params master_config: eg. local[4] :returns SparkContext, SQLContext, SparkSession: """ conf = (SparkConf().setAppName(app_name).setMaster(master_config)) sc = SparkContext(conf=conf) sc.setLogLevel("ERROR") sql_ctx = SQLContext(sc) spark = SparkSession(sc) return (sc, sql_ctx, spark) 

This can then be called as

sc, sql_ctx, spark = init_spark("App_name", "local[4]")