Python爬虫入门之爬万本书籍！

2019-04-10 本文已影响0人 14e61d025165

前言

Python现在非常火，语法简单而且功能强大，很多同学都想学Python！所以小的给各位看官们准备了高价值Python学习视频教程及相关电子版书籍，都放在了文章结尾，欢迎前来领取！

<tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1554879160815 ql-align-center" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; text-align: left; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;">

image

在这里还是要推荐下我自己建的Python开发学习群:683380553，群里都是学Python开发的，如果你正在学习Python ，小编欢迎你加入，大家都是软件开发党，不定期分享干货（只有Python软件开发相关的），包括我自己整理的一份2018最新的Python进阶资料和高级开发教程，欢迎进阶中和进想深入Python的小伙伴

最近闲的无聊，想爬点书看看。于是我选择了这个网站雨枫轩（http://www.rain8.com/）

STEP1.分析网站

一开始我想通过一篇文章引用的链接，将书爬完，后来发现并不需要这样做。比如我们可以打开人生哲学这个栏目。

<tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1554879160821 ql-align-center" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; text-align: left; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;">

image

如图1所示，会把页面数全列出来。

并且这个url

也非常的有规律。

可以看出是由

组成的。

知道了这点后，我们就能轻松的把网站爬完了。

STEP2.初始化

首先我们来理清思路。

<tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1554879160827 ql-align-center" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; text-align: left; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;">

image

这是我们需要的库

其实这个项目用urllib2也能完成。

我们首先来初始化,在self.urls里，我们定义一些需要fetch的栏目。(比如言情，恐怖仙侠之类的书就可以跳过)。然后写一些正则表达式，供其他的method使用。

<pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">class fetchBook:
def init(self):
self.header = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0'}
self.urls = [
'http://txt.rain8.com/txtgw/',
'http://txt.rain8.com/txtzj/',
'http://txt.rain8.com/txtzx/',
'http://txt.rain8.com/txtsh/'
]
self.rePageIndex = re.compile('list_\d+_\d+.html')#得到栏目编号
self.rePageCount = re.compile('<strong>\d+</strong>')#得到页面数目
self.reDownloadGet1 = re.compile('href=.http://txt.rain8.com/plus/xiazai_yfx.php?[^>]+')#得到下载链接
self.reGetTitle = re.compile('<title>.+</title>')#得到标题
self.reGetAuthor = re.compile("</small><span>[^>]+")#得到作者名称
self.reBookGetNew = re.compile('')#得到书籍链接
self.reBookGetOld = re.compile('')
self.cnt = 0
</pre>

STEP3.获取所有的页面

我们可以查看人生哲学源代码。可以找到总共的页数。

<tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1554879160834 ql-align-center" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; text-align: left; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;">

image

然后我们就能得到每一页的数据。

代码如下

<pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;"> def viewAllPage(self,url):
"""
函数功能为把该栏目下所有页面全过一遍。
"""
req = requests.get(url,headers = self.header)
pageIndex = self.rePageIndex.findall(req.text)[0][5:7]
pageCount = int(self.rePageCount.findall(req.text)[0][8:-9])
urlToFetch = [url,'list_',pageIndex,'_','1','.html']
foldname = self.reGetTitle.findall(req.text)[0][7:]
foldname = foldname.encode('unicode_escape').decode('string_escape')
foldname = foldname.split('|')[0]
self.createDir(foldname)
for page in range(1,pageCount+1):
urlToFetch[4] = str(page)
url_to_get = ''.join(urlToFetch)#得到所有页面的url
</pre>

STEP4.得到下载链接

然后我们用上面写好的正则表达式，来匹配她的下载链接。

<pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;"> def fetchDownloadUrl(self,bookurl):
req = requests.get(bookurl,headers = self.header)
result = self.reDownloadGet1.findall(req.text)
result = result[0][6:-17]
authorname = self.reGetAuthor.findall(req.text)[0][14:-6].encode('unicode_escape').decode('string_escape')
req = requests.get(result,headers = self.header)
bookname = self.reGetTitle.findall(req.text)[0][7:-24]
downloadurl = self.reDownloadGet1.findall(req.text)[0][6:-17]
return downloadurl,bookname,authorname
</pre>

STEP5.下载

下载的时候，由于该网站下载的书都是rar格式的。所以我们只要用二进制的方式写就可以了。

我们调用

然后

就可以写进去了。

STEP6.编码问题

之前一直被python的编码问题搞的头大，这次又遇见了。于是找了点资料看了看。

因为python工作使用的编码是unicode，如果要在编码间进行转化，推荐要先decode成unicode，然后再encode成别的编码。

然后我在爬内容的过程中碰到了这个问题：

\xb9这样的很明显应该是gbk编码。而python却在字符串前加了个u。然后我对这串字符encode还是decode都会报错。或者打印出来乱码。

经过查阅资料，我找到了一种解决办法

经过这样处理后

'\xb9\xc5\xca\xab\xb4\xca\xc3\xfb\xd6\xf8'
</pre>

很明显就恢复了正常。这时我们只要decode('gb2312')就能转换成unicode编码了。

STEP7.结果展示

<tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1554879160851 ql-align-center" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; text-align: left; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;">

image <input class="pgc-img-caption-ipt" placeholder="图片描述(最多50字)" value="" style="box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1) 0s;"></tt-image> <tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1554879160854" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;">

image

Python爬虫入门之爬万本书籍！

猜你喜欢

热点阅读