Install scrapy

How to install Scrapy on Ubuntu 16.04

Today I’m going to show you how to install Scrapy framework in your system in less than 5 minutes. In ubuntu it’s very easy to install packages so it should be the same now.

Install Scrapy On Ubuntu

The very first step is to make sure that each dependency of Scrapy is installed locally in your system.

Scrapy dependencies

You will need these packages installed already:

  • python-dev
  • python-pip
  • libxml2-dev
  • libxslt1-dev
  • zlib1g-dev
  • libffi-dev
  • libssl-dev

If you’re not sure these packages are already installed or you want to make sure they are, just open a terminal with Ctrl+alt and run this command:

sudo apt-get install python-dev python-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev

This command will install every dependency you will need.

Install Scrapy with pip

Now you can easily install scrapy with pip, run this command:

pip install Scrapy

You can test if Scrapy installed properly by running this:

scrapy version

Now you should see which Scrapy version is installed now.

Scrapy: command not found

In case you see something like “scrapy: command not found” or Scrapy just doesn’t work then follow this guide:

Install lxml and gcc with these commands:

sudo apt-get install python-dev
sudo apt-get install libevent-dev
sudo apt-get install libxml2 libxml2-dev
sudo apt-get install libxml2-dev libxslt-dev
sudo apt-get install python-lxml

Install twisted:

sudo apt-get install python-twisted python-libxml2 python-simplejson

Install pyOpenSSL:

wget https://pypi.python.org/packages/source/p/pyOpenSSL/pyOpenSSL-0.13.tar.gz
tar -zxvf pyOpenSSL-0.13.tar.gz
cd pyOpenSSL-0.13
sudo python setup.py install

Install pycrypto:

wget http://pypi.python.org/packages/source/p/pycrypto/pycrypto-2.5.tar.gz
tar -zxvf pycrypto-2.5.tar.gz
cd pycrypto-2.5
sudo python setup.py install

Install easy_install:

wget http://peak.telecommunity.com/dist/ez_setup.py
python ez_setup.py

Finally, install Scrapy:

sudo easy_install Scrapy

You’re done! Again, you can test that Scrapy is installed correctly with this command:

scrapy version

If you cannot install scrapy for some reason let me know in the comments and I will help you!

So now you’ve just setup Scrapy on your system. If you want to learn how to build awesome scrapers with Scrapy just fill in the form below and get started!