Today I’m going to show you how to install Scrapy framework in your system in less than 5 minutes. In ubuntu it’s very easy to install packages so it should be the same now.
Install Scrapy On Ubuntu
The very first step is to make sure that each dependency of Scrapy is installed locally in your system.
You will need these packages installed already:
If you’re not sure these packages are already installed or you want to make sure they are, just open a terminal with Ctrl+alt and run this command:
sudo apt-get install python-dev python-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev
This command will install every dependency you will need.
Install Scrapy with pip
Now you can easily install scrapy with pip, run this command:
pip install Scrapy
You can test if Scrapy installed properly by running this:
Now you should see which Scrapy version is installed now.
Scrapy: command not found
In case you see something like “scrapy: command not found” or Scrapy just doesn’t work then follow this guide:
Install lxml and gcc with these commands:
sudo apt-get install python-dev sudo apt-get install libevent-dev sudo apt-get install libxml2 libxml2-dev sudo apt-get install libxml2-dev libxslt-dev sudo apt-get install python-lxml
sudo apt-get install python-twisted python-libxml2 python-simplejson
wget https://pypi.python.org/packages/source/p/pyOpenSSL/pyOpenSSL-0.13.tar.gz tar -zxvf pyOpenSSL-0.13.tar.gz cd pyOpenSSL-0.13 sudo python setup.py install
wget http://pypi.python.org/packages/source/p/pycrypto/pycrypto-2.5.tar.gz tar -zxvf pycrypto-2.5.tar.gz cd pycrypto-2.5 sudo python setup.py install
wget http://peak.telecommunity.com/dist/ez_setup.py python ez_setup.py
Finally, install Scrapy:
sudo easy_install Scrapy
You’re done! Again, you can test that Scrapy is installed correctly with this command:
If you cannot install scrapy for some reason let me know in the comments and I will help you!
So now you’ve just setup Scrapy on your system. If you want to learn how to build awesome scrapers with Scrapy just fill in the form below and get started!