what is difference between spacy.load('en_core_web_sm') and spacy.load('en')? This link explains different model sizes. But i am still not clear how spacy.load('en_core_web_sm') and spacy.load('en') differ

spacy.load('en') runs fine for me. But the spacy.load('en_core_web_sm') throws error

i have installed spacyas below. when i go to jupyter notebook and run command nlp = spacy.load('en_core_web_sm') I get the below error 

--------------------------------------------------------------------------- OSError Traceback (most recent call last) <ipython-input-4-b472bef03043> in <module>() 1 # Import spaCy and load the language library 2 import spacy ----> 3 nlp = spacy.load('en_core_web_sm') 4 5 # Create a Doc object C:\Users\nikhizzz\AppData\Local\conda\conda\envs\tensorflowspyder\lib\site-packages\spacy\__init__.py in load(name, **overrides) 13 if depr_path not in (True, False, None): 14 deprecation_warning(Warnings.W001.format(path=depr_path)) ---> 15 return util.load_model(name, **overrides) 16 17 C:\Users\nikhizzz\AppData\Local\conda\conda\envs\tensorflowspyder\lib\site-packages\spacy\util.py in load_model(name, **overrides) 117 elif hasattr(name, 'exists'): # Path or Path-like to model data 118 return load_model_from_path(name, **overrides) --> 119 raise IOError(Errors.E050.format(name=name)) 120 121 OSError: [E050] Can't find model 'en_core_web_sm'. It doesn't seem to be a shortcut link, a Python package or a valid path to a data directory.

how I installed Spacy ---

(C:\Users\nikhizzz\AppData\Local\conda\conda\envs\tensorflowspyder) C:\Users\nikhizzz>conda install -c conda-forge spacy Fetching package metadata ............. Solving package specifications: . Package plan for installation in environment C:\Users\nikhizzz\AppData\Local\conda\conda\envs\tensorflowspyder: The following NEW packages will be INSTALLED: blas: 1.0-mkl cymem: 1.31.2-py35h6538335_0 conda-forge dill: 0.2.8.2-py35_0 conda-forge msgpack-numpy: 0.4.4.2-py_0 conda-forge murmurhash: 0.28.0-py35h6538335_1000 conda-forge plac: 0.9.6-py_1 conda-forge preshed: 1.0.0-py35h6538335_0 conda-forge pyreadline: 2.1-py35_1000 conda-forge regex: 2017.11.09-py35_0 conda-forge spacy: 2.0.12-py35h830ac7b_0 conda-forge termcolor: 1.1.0-py_2 conda-forge thinc: 6.10.3-py35h830ac7b_2 conda-forge tqdm: 4.29.1-py_0 conda-forge ujson: 1.35-py35hfa6e2cd_1001 conda-forge The following packages will be UPDATED: msgpack-python: 0.4.8-py35_0 --> 0.5.6-py35he980bc4_3 conda-forge The following packages will be DOWNGRADED: freetype: 2.7-vc14_2 conda-forge --> 2.5.5-vc14_2 Proceed ([y]/n)? y blas-1.0-mkl.t 100% |###############################| Time: 0:00:00 0.00 B/s cymem-1.31.2-p 100% |###############################| Time: 0:00:00 1.65 MB/s msgpack-python 100% |###############################| Time: 0:00:00 5.37 MB/s murmurhash-0.2 100% |###############################| Time: 0:00:00 1.49 MB/s plac-0.9.6-py_ 100% |###############################| Time: 0:00:00 0.00 B/s pyreadline-2.1 100% |###############################| Time: 0:00:00 4.62 MB/s regex-2017.11. 100% |###############################| Time: 0:00:00 3.31 MB/s termcolor-1.1. 100% |###############################| Time: 0:00:00 187.81 kB/s tqdm-4.29.1-py 100% |###############################| Time: 0:00:00 2.51 MB/s ujson-1.35-py3 100% |###############################| Time: 0:00:00 1.66 MB/s dill-0.2.8.2-p 100% |###############################| Time: 0:00:00 4.34 MB/s msgpack-numpy- 100% |###############################| Time: 0:00:00 0.00 B/s preshed-1.0.0- 100% |###############################| Time: 0:00:00 0.00 B/s thinc-6.10.3-p 100% |###############################| Time: 0:00:00 5.49 MB/s spacy-2.0.12-p 100% |###############################| Time: 0:00:10 7.42 MB/s (C:\Users\nikhizzz\AppData\Local\conda\conda\envs\tensorflowspyder) C:\Users\nikhizzz>python -V Python 3.5.3 :: Anaconda custom (64-bit) (C:\Users\nikhizzz\AppData\Local\conda\conda\envs\tensorflowspyder) C:\Users\nikhizzz>python -m spacy download en Collecting en_core_web_sm==2.0.0 from Downloading (37.4MB) 100% |################################| 37.4MB ... Installing collected packages: en-core-web-sm Running setup.py install for en-core-web-sm ... done Successfully installed en-core-web-sm-2.0.0 Linking successful C:\Users\nikhizzz\AppData\Local\conda\conda\envs\tensorflowspyder\lib\site-packages\en_core_web_sm --> C:\Users\nikhizzz\AppData\Local\conda\conda\envs\tensorflowspyder\lib\site-packages\spacy\data\en You can now load the model via spacy.load('en') (C:\Users\nikhizzz\AppData\Local\conda\conda\envs\tensorflowspyder) C:\Users\nikhizzz>
3

31 Answers

1 2

Initially I downloaded two en packages using following statements in anaconda prompt.

python -m spacy download en_core_web_lg python -m spacy download en_core_web_sm

But, I kept on getting linkage error and finally running below command helped me to establish link and solved error.

python -m spacy download en

Also make sure you to restart your runtime if working with Jupyter. -PS : If you get linkage error try giving admin previlages.

3

The answer to your misunderstanding is a Unix concept, softlinks which we could say that in Windows are similar to shortcuts. Let's explain this.

When you spacy download en, spaCy tries to find the best small model that matches your spaCy distribution. The small model that I am talking about defaults to en_core_web_sm which can be found in different variations which correspond to the different spaCy versions (for example spacy, spacy-nightly have en_core_web_sm of different sizes).

When spaCy finds the best model for you, it downloads it and then links the name en to the package it downloaded, e.g. en_core_web_sm. That basically means that whenever you refer to en you will be referring to en_core_web_sm. In other words, en after linking is not a "real" package, is just a name for en_core_web_sm.

However, it doesn't work the other way. You can't refer directly to en_core_web_sm because your system doesn't know you have it installed. When you did spacy download en you basically did a pip install. So pip knows that you have a package named en installed for your python distribution, but knows nothing about the package en_core_web_sm. This package is just replacing package en when you import it, which means that package en is just a softlink to en_core_web_sm.

Of course, you can directly download en_core_web_sm, using the command: python -m spacy download en_core_web_sm, or you can even link the name en to other models as well. For example, you could do python -m spacy download en_core_web_lg and then python -m spacy link en_core_web_lg en. That would make en a name for en_core_web_lg, which is a large spaCy model for the English language.

Hope it is clear now :)

1

The below worked for me :

import en_core_web_sm nlp = en_core_web_sm.load()
2

For those who are still facing problems even after installing it as administrator from Anaconda prompt, here's a quick fix:

  1. Got to the path where it is downloaded. For e.g.

    C:\Users\name\AppData\Local\Continuum\anaconda3\Lib\site-packages\en_core_web_sm\en_core_web_sm-2.2.0

  2. Copy the path.

  3. Paste it in:

    nlp = spacy.load(r'C:\Users\name\AppData\Local\Continuum\anaconda3\Lib\site-packages\en_core_web_sm\en_core_web_sm-2.2.0')

  4. Works like a charm :)

PS: Check for spacy version

1

Using the Spacy language model in Colab requires only the following two steps:

  1. Download the model (change the name according to the size of the model)
!python -m spacy download en_core_web_lg
  1. Restart the colab runtime! Perform shortcut key： Ctrl + M + .

Test

import spacy nlp = spacy.load("en_core_web_lg")

successful!!!

Try this method as this worked like a charm to me:

In your Anaconda Prompt, run the command:

!python -m spacy download en

After running the above command, you should be able to execute the below in your jupyter notebook:

spacy.load('en_core_web_sm')

First of all, install spacy using the following command for jupyter notebook pip install -U spacy

Then write the following code:

import en_core_web_sm nlp = en_core_web_sm.load()

I am running Jupyter Notebook on Windows.

Finally, its a version issue, Need to execute below commands in conda cmd prompt( open as admin)

  • pip install spacy==2.3.5

  • python -m spacy download en_core_web_sm

  • python -m spacy download en

from chatterbot import ChatBot import spacy import en_core_web_sm nlp = en_core_web_sm.load() ChatBot("hello")

Output - enter image description here

import spacy nlp = spacy.load('/opt/anaconda3/envs/NLPENV/lib/python3.7/site-packages/en_core_web_sm/en_core_web_sm-2.3.1')

Try giving the absolute path of the package with the version as shown in the image.

It works perfectly fine.

Steps to load up modules based on different versions of spacy

download the best-matching version of a specific model for your spaCy installation

python -m spacy download en_core_web_sm pip install .tar.gz archive from path or URL pip install /Users/you/en_core_web_sm-2.2.0.tar.gz

or

pip install

Add to your requirements file or environment yaml file. Theres range of version that one spacy version is comptable with you can view more under

if your not sure running below code

nlp = spacy.load('en_core_web_sm')

will give off a warning telling what version model will be compatible with your installed spacy verion

enironment.yml example

name: root channels: - defaults - conda-forge - anaconda dependencies: - python=3.8.3 - pip - spacy=2.3.2 - scikit-learn=0.23.2 - pip: -

a simple solution for this which I saw on spacy.io

from spacy.lang.en import English nlp=English()

2

As for Windows based Anaconda,

  1. Open Anaconda Prompt

  2. Activate your environment. Ex: active myspacyenv

  3. pip install

  4. python -m spacy download en_core_web_sm

  5. Open Jupyter Notebook ex: active myspacyenv and then jupyter notebook on Anaconda Promt

import spacy spacy.load('en_core_web_sm')

and it will run peacefully!

1

Don't run !python -m spacy download en_core_web_lg from inside jupyter. Do this instead:

import spacy.cli spacy.cli.download("en_core_web_lg")

You may need to restart the kernel before running the above two commands for it to work.

Open Anaconda Navigator. Click on any IDE. Run the code: 

!pip install -U spacy download en_core_web_sm !pip install -U spacy download en_core_web_sm

It will work. If you are open IDE directly close it and follow this procedure once.

Loading the module using the different syntax worked for me.

import en_core_web_sm nlp = en_core_web_sm.load()

Anaconda Users

  1. If you're using a conda virtual environment, be sure that its the same version of Python as that in your base environment. To verify this, run python --version in each environment. If not the same, create a new virtual environment with that version of Python (Ex. conda create --name myenv python=x.x.x).

  2. Activate the virtual environment (conda activate myenv)

  3. conda install -c conda-forge spacy
  4. python -m spacy download en_core_web_sm

I just ran into this issue, and the above worked for me. This addresses the issue of the download occurring in an area that is not accessible to your current virtual environment.

You should then be able to run the following:

import spacy nlp = spacy.load("en_core_web_sm")

Open command prompt or terminal and execute the below code:

pip3 install

Execute the below chunk in your Jupiter notebook.

import spacy

nlp = spacy.load('en_core_web_sm')

Hope the above code works for all:)

I had also same issue as I couldnt load module using '''spacy.load()''' You can follow below steps to solve this on windows:

  1. download using !python -m spacy download en_core_web_sm
  2. import en_core_web_sm as import en_core_web_sm
  3. load using en_core_web_sm.load() to some variable

Complete code will be:

python -m spacy download en_core_web_sm import en_core_web_sm nlp = en_core_web_sm.load()

This works with colab:

!python -m spacy download en import en_core_web_sm nlp = en_core_web_sm.load()

Or for the medium:

import en_core_web_md nlp = en_core_web_md.load()
0

Instead of any of the above, this solved my error.

conda install -c conda-forge spacy-model-en_core_web_sm

If you are an anaconda user, this is the solution.

I'm running PyCharm on MacOS and while none of the above answers completely worked for me, they did provide enough clues and I was finally able to everything working. I am connecting to an ec2 instance and have configured PyCharm such that I can edit on my Mac and it automatically updates the files on my ec2 instance. Thus, the problem was on the ec2 side where it was not finding Spacy even though I installed it several different times and ways. If I ran my python script from the command line, everything worked fine. However, from within PyCharm, it was initially not finding Spacy and the models. I eventually fixed the "finding" spacy issue using the above recommendation of adding a "requirements.txt" file. But the models were still not recognized.

My solution: download the models manually and place them in the file system on the ec2 instance and explicitly point to them when loaded. I downloaded the files from here:

After downloading, I dropped moved them to my ec2 instance, decompressed and untared them in my filesystem, e.g. /path_to_models/en_core_web_lg-3.0.0/

I then load a model using the explicit path and it worked from within PyCharm (note the path used goes all the way to en_core_web_lg-3.0.0; you will get an error if you do not use the folder with the config.cfg file):

nlpObject = spacy.load('/path_to_models/en_core_web_lg-3.0.0/en_core_web_lg/en_core_web_lg-3.0.0')
1

Check installed version of spacy pip show spacy You will get something like this:

Name: spacy Version: 3.1.3 Summary: Industrial-strength Natural Language Processing (NLP) in Python

Install the relevant version of the model using: !pip install -U

I tried all the above answers but could not succeed. Below worked for me :

(Specific to WINDOWS os)

  1. Run anaconda command prompt with admin privilege(Important)
  2. Then run below commands:
 pip install -U --user spacy python -m spacy download en
  1. Try below command for verification:
import spacy spacy.load('en')
  1. It might work for others versions as well: enter image description here

If you have already downloaded spacy and the language model (E.g., en_core_web_sm or en_core_web_md), then you can follow these steps:

  1. Open Anaconda prompt as admin

  2. Then type : python -m spacy link [package name or path] [shortcut]

    For E.g., python -m spacy link /Users/you/model en

This will create a symlink to the your language model. Now you can load the model using spacy.load("en") in your notebooks or scripts

This is what I did:

  1. Went to the virtual environment where I was working on Anaconda Prompt / Command Line

  2. Ran this: python -m spacy download en_core_web_sm

And was done

TRY THIS :-
!python -m spacy download en_core_web_md

Even I faced similar issue. How I resolved it

  1. start anaconda prompt in admin mode.
  2. installed both python -m spacy download en and python -m spacy download en_core_web_sm after above steps only I started jupyter notebook where I am accessing this package. Now I can access both import spacy nlp = spacy.load('en_core_web_sm') or nlp = spacy.load('en') Both are working for me.

I faced a similar issue. I installed spacy and en_core_web_sm from a specific conda environment. However, I got two(02) differents issues as following:

[Errno 2] No such file or directory: '....\en_core_web_sm\en_core_web_sm-2.3.1\vocab\lexemes.bin' or OSError: [E050] Can't find model 'en_core_web_sm'.... It doesn't seem to be a shortcut link, a Python package or a valid path to a data directory.

I did the following:

  1. Open Command Prompt as Administrator
  2. Go to c:>
  3. Activate my Conda environment (If you work in a specific conda environment):
c:\>activate <conda environment name>
  1. (conda environment name)c:\>python -m spacy download en
  2. Return to Jupyter Notebook and you can load the language library:
nlp = en_core_web_sm.load()

For me, it works :)

Download en_core_web_sm tar file

Open terminal from anaconda or open anaconda evn. Run this:

pip3 install /Users/yourpath/Downloads/en_core_web_sm-3.1.0.tar.gz;

or

pip install /Users/yourpath/Downloads/en_core_web_sm-3.1.0.tar.gz;

Restart jupyter, it will work.

Run this in os console:

python -m spacy download en python -m spacy link en_core_web_sm en_core_web_sm

Then run this in python console or on your python IDE:

import spacy spacy.load('en_core_web_sm')

1 2