Python爬虫之Scrapy的安装

一.Scrapy的介绍

Scrapy是用Python开发的一个快速,高层次的屏幕抓取和web抓取框架，用于抓取web站点并从页面中提取结构化的数据。Scrapy用途广泛，可以用于数据挖掘、监测和自动化测试。它有个萌出血的外号叫小抓抓。

下面主要介绍下Scrapy的安装步骤以及遇到的一些坑。

二，安装步骤

首先安装蟒蛇包工具点子，它相当于iOS的开发中的的CocoaPods。

1. $ sudo easy_install pip

安装成功后执行下面命令

2. $ sudo pip安装Scrapy

DEPRECATION: Uninstalling a distutils installed project (six) has been deprecatedandwill be removedina future version. Thisisdue to the fact that uninstalling a distutils project will only partially uninstall the project.Uninstalling six-1.4.1:Exception:Traceback (most recent call last):File"/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/basecommand.py", line215,inmainstatus = self.run(options, args)File"/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/commands/install.py", line342,inrunprefix=options.prefix_path,File"/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_set.py", line778,ininstallrequirement.uninstall(auto_confirm=True)File"/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_install.py", line754,inuninstallpaths_to_remove.remove(auto_confirm)File"/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_uninstall.py", line115,inremoverenames(path, new_path)File"/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/utils/__init__.py", line267,inrenamesshutil.move(old, new)File"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line302,inmovecopy2(src, real_dst)File"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line131,incopy2copystat(src, dst)File"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line103,incopystatos.chflags(dst, st.st_flags)OSError: [Errno1] Operationnotpermitted:'/tmp/pip-QfQY7O-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info'

安装提示Mac自带的python 2.7版本有误，所以我们需要重新安装python

3. $ brew安装python

（如果此前没有安装过brew，请先安装，命令行如下）

$ ruby -e"$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

python安装成功后提示：

==> **Summary**🍺  /usr/local/Cellar/python/2.7.13: 6,337 files, 86.7M, builtin3 minutes 19 seconds

4. $ sudo pip安装Scrapy

出现下面提示则代表lxml安装出错，

Command"/usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/tmp/pip-build-keKznw/lxml/setup.py';f=getattr(tokenize,'open', open)(__file__);code=f.read().replace('\r\n','\n');f.close();exec(compile(code,__file__,'exec'))" install --record /tmp/pip-FEkrEy-record/install-record.txt --single-version-externally-managed --compile"failed with error code1in /private/tmp/pip-build-keKznw/lxml/

按照下面三步重新安装LXML

1.xcode-select --install2.C_INCLUDE_PATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include/libxml2:/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include/libxml2//libxml:/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include3.sudo pip install lxml

5.成功后再次执行$ sudo pip install Scrapy

Successfully built Twisted PyDispatcher lxml pycparserInstalling collected packages: cssselect, six, w3lib, queuelib, zope.interface, constantly, incremental, Twisted, PyDispatcher, lxml, enum34, ipaddress, idna, pycparser, cffi, cryptography, pyOpenSSL, parsel, ScrapySuccessfully installed PyDispatcher-2.0.5Scrapy-1.3.0Twisted-16.6.0cffi-1.9.1constantly-15.1.0cryptography-1.7.1cssselect-1.0.0enum34-1.1.6idna-2.2incremental-16.10.1ipaddress-1.0.17lxml-3.7.1parsel-1.1.0pyOpenSSL-16.2.0pycparser-2.17queuelib-1.4.2six-1.10.0w3lib-1.16.0zope.interface-4.3.3You are using pip version8.1.2, however version9.0.1is available.You should consider upgrading via the'pip install --upgrade pip'command.

到这里就已经安装成功了。（因为Scrapy安装源被墙，所以安装过程最好使用VPN）