No module name pyspark error

This is the exact code from a tutorial I'm following. My classmate didn't get this error with the same code:

ImportError Traceback (most recent call last) <ipython-input-1-c6e1bed850ab> in <module>() ----> 1 from pyspark import SparkContext 2 sc = SparkContext('local', 'Exam_3') 3 4 from pyspark.sql import SQLContext 5 sqlContext = SQLContext(sc) ImportError: No module named pyspark 

This is the code:

from pyspark import SparkContext sc = SparkContext('local', 'Exam_3') from pyspark.sql import SQLContext sqlContext = SQLContext(sc) data = sc.textFile("exam3") parsedData = data.map(lambda line: [float(x) for x in line.split(',')]) retail = sqlContext.createDataFrame(parsedData, ['category_name','product_id', 'product_name', 'product_price']) retail.registerTempTable("exam3") print parsedData.take(3) 

7 Answers

You don't have pyspark installed in a place available to the python installation you're using. To confirm this, on your command line terminal, with your virtualenv activated, enter your REPL (python) and type import pyspark:

$ python Python 3.5.0 (default, Dec 3 2015, 09:58:14) [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import pyspark Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named 'pyspark' 

If you see the No module name 'pyspark' ImportError you need to install that library. Quit the REPL and type:

pip install pyspark 

Then re-enter the repl to confirm it works:

$ python Python 3.5.0 (default, Dec 3 2015, 09:58:14) [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import pyspark >>> 

As a note, it is critical your virtual environment is activated. When in the directory of your virtual environment:

$ source bin/activate 

These instructions are for a unix-based machine, and will vary for Windows.

Just use:

import findspark findspark.init() import pyspark # only run after findspark.init() 

If you don't have findspark module install it with:

python -m pip install findspark 

You can use findspark to make spark accessible at run time. Typically findspark will find the directory where you have installed spark, but if it is installed in a non-standard location, you can point it to the correct directory. Once you have installed findspark, if spark is installed at /path/to/spark_home just put

import findspark findspark.init('/path/to/spark_home') 

at the very top of your script/notebook and you should now be able to access the pyspark module.

Here is the latest solution that is worked for me FOR MAC users only. I've installed pyspark through pip install pyspark. But, it didn't work when I execute pyspark in terminal or even in python import pyspark. I checked that pyspark already installed in my laptop.

At the end, I found the solution. You just need to add into the bash profile file.

Follow steps:

1) Type the following in a terminal window to go to your home folder.

cd ~

2) Then the following to create a .bash_profile. (You may skip if it already exists.)

touch .bash_profile

3) open -e .bash_profile

Then add the following variables.

export SPARK_VERSION=`ls /usr/local/Cellar/apache-spark/ | sort | tail -1` export SPARK_HOME="/usr/local/Cellar/apache-spark/$SPARK_VERSION/libexec" export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.10.7-src.zip:$PYTHONPATH 

YOU NEED TO CHANGE py4j-x.x.x-src.zip version number in LAST LINE

4) Once all these variables are assigned, save and close .bash_profile. Then the type following command to reload the file.

. .bash_profile

Make sure first to install pyspark using conda :

conda install pyspark 

import findspark findspark.init()

Traceback (most recent call last): File "", line 1, in ImportError: No module named 'findspark'

$ pip install findspark

it will work

my solution to this problem was to

• open up a brand new .ipynb on local machine by:

$jupyter-lab 

• next in the very first cell running this:

$pip install pyspark

my output is:

Collecting pyspark Using cached pyspark-3.2.0.tar.gz (281.3 MB) Preparing metadata (setup.py) ... done Collecting py4j==0.10.9.2 Using cached py4j-0.10.9.2-py2.py3-none-any.whl (198 kB) Building wheels for collected packages: pyspark Building wheel for pyspark (setup.py) ... done Created wheel for pyspark: filename=pyspark-3.2.0-py2.py3-none-any.whl size=281805913 sha256=26e539058858454dbbb48158111968d67e663c7b53e64c4fd91e38d92ac1cd80 Stored in directory: /Users/user/Library/Caches/pip/wheels/2f/f8/95/2ad14a4614b4a9f645ee928fbbd057b1b254c67adb494c9a58 Successfully built pyspark Installing collected packages: py4j, pyspark Successfully installed py4j-0.10.9.2 pyspark-3.2.0 Note: you may need to restart the kernel to use updated packages. 

• then import pyspark:

$import pyspark

You may as well want to try running the pip command directly in the lab env.