代码改变世界程序员@IT·互联网

如何爬取豆瓣阅读的所有出版社?

2017-05-19  本文已影响0人  陈昱熹

#1.导入相关模块
import urllib.request
import re
import sqlite3
#2.读取网页信息并解码
url = "https://read.douban.com/provider/all"
data = urllib.request.urlopen(url).read().decode('utf-8','ignore')
#3.采用正则表达式提取目标内容
pat1 = '<div class="name">(.*?)</div>'
pat2 = '<div class="works-num">(.*?) 部作品在售</div>'
rst1 = re.compile(pat1).findall(data)
rst2 = re.compile(pat2).findall(data)
rst = zip(rst1,rst2)
#4.存储txt文件
with open('demo1.txt','w') as fh:
    for i,j in rst:
        line = i + ' ,' + j +'\n'
        fh.write(line)
#5.存储sqlite数据库
conn = sqlite3.connect('demo1.db')
curs = conn.cursor()
curs.execute('''
CREATE TABLE publish (
    publisher TEXT,
    num INTEGER
)
''')
query = 'INSERT INTO publish VALUES (?,?)'
for line in open('demo1.txt'):
    line = line.split(',')
    curs.execute(query,line)
#curs.executemany(query,rst) **为什么不能直接这样做?
#curs.executemany(query,rst) **为什么不能直接这样做?
#curs.executemany(query,rst) **为什么不能直接这样做?
conn.commit()
conn.close()
上一篇 下一篇

猜你喜欢

热点阅读