Update: Almost since one year after writing this article, spaCy now has been upgraded to version 1.2, and new data and new features are added in it. I fix some problems in this article for spacy install and test, especially the load data method. Just enjoy it. 2016.11.13

I met spaCy by google “nltk cython”, and found it very interesting:

spaCy.io | Build Tomorrow’s Language Technologies

spaCy is a library for industrial-strength natural language processing in Python and Cython. It features state-of-the-art speed and accuracy, a concise API, and great documentation. If you’re a small company doing NLP, we want spaCy to seem like a minor miracle.

Now let’s get started with spacy。

Installing spaCy
Installing spacy is very easy, I have installed spacy on my mac and ubuntu vps, both using the pip install methods:

$ sudo pip install -U spacy

And you should download the data and models from spacy, here we downlaod the English data:

$ sudo python -m spacy.en.download all

Note the download data is about 1G，and it split by two parts: parser and glove word2vec modes, and you can also download them one by one:

$ python -m spacy.en.download parser

$ python -m spacy.en.download glove

The data is in the spacy install dir, like this:

textminer@textminer:/usr/local/lib/python2.7/dist-packages/spacy/data$ du -sh *
776M en-1.1.0
774M en_glove_cc_300_1m_vectors-1.0.0

and in the parser dir”en-1.1.0″, like following:

textminer@textminer:/usr/local/lib/python2.7/dist-packages/spacy/data/en-1.1.0$ du -sh *
424M deps
8.0K meta.json
35M ner
12M pos
84K tokenizer
300M vocab
6.3M wordnet

Test spaCy
After installing spaCy, you can test it by the Python or iPython interpreter:

First, load and initialize the nlp data and text processor， this took about one minute on my macbook pro:

In [1]: import spacy

In [2]: nlp = spacy.load('en')

Word Tokenize Test:

In [14]: doc1 = nlp(u"this's spacy tokenize test")

In [15]: print doc1
this's spacy tokenize test

In [16]: for token in doc1:
....: print token
....:
this
's
spacy
tokenize
test

Sentence Tokenize Test or Sentence Segmentation Test:

In [23]: doc2 = nlp(u"this is spacy sentence tokenize test. this is second sent! is this the third sent? final test.")

In [24]: for sent in doc2.sents:
print sent
....:
this is spacy sentence tokenize test.
this is second sent!
is this the third sent?
final test.

Lemmatize Test:
In [35]: doc3 = nlp(u"this is spacy lemmatize testing. programming books are more better than others")

In [36]: for token in doc3:
....: print token, token.lemma, token.lemma_
....:
this 496 this
is 488 be
spacy 173779 spacy
lemmatize 1510965 lemmatize
testing 2900 testing
. 419 .
programming 3408 programming
books 1011 book
are 488 be
more 529 more
better 615 better
than 555 than
others 871 others

Pos Tagging Test:
In [40]: doc4 = nlp(u"This is pos tagger test for spacy pos tagger")

In [41]: for token in doc4:
print token, token.pos, token.pos_
....:
This 87 DET
is 97 VERB
pos 82 ADJ
tagger 89 NOUN
test 89 NOUN
for 83 ADP
spacy 89 NOUN
pos 89 NOUN
tagger 89 NOUN

Named Entity Recognizer (NER) Test:
Here I used the same example from Stanford NER Python Example:

In [51]: doc5 = nlp(u"Rami Eid is studying at Stony Brook University in New York")

In [52]: for ent in doc5.ents:
print ent, ent.label, ent.label_
....:
Rami Eid 346 PERSON
Stony Brook University 349 ORG
New York 350 GPE

Noun Chunk Test:
In [58]: doc6 = nlp(u"Natural language processing (NLP) deals with the application of computational models to text or speech data.")

In [59]: for np in doc6.noun_chunks:
....: print np
....:
Natural language processing
the application
computational models
text
speech data

Word Vectors Test:
This example is copied from the spacy official example:
In [68]: doc7 = nlp(u"Apples and oranges are similar. Boots and hippos aren't.")
In [69]: apples = doc7[0]

In [70]: oranges = doc7[2]

In [71]: boots = doc7[6]

In [72]: hippos = doc7[8]

In [73]: apples.similarity(oranges)
Out[73]: 0.7857989796519943

In [74]: boots.similarity(hippos)
Out[74]: 0.41468512503416094


Now it’s your time to explore the spaCy.

Post by TextMiner