趁热打铁的爬阳光电影网
2017-04-09 本文已影响53人
谁占了我的一年的称号
女朋友这几天晚上总是在看电影，╮(╯▽╰)╭，让哥哥一个人自己玩。哼，不就是电影嘛？我给你一个库！
说干就干，前几天在程老哥的指导下，终于理解了多层网页爬取的时候，数据是怎么传递的。今天选的阳光电影网也是这种的结构：http://www.ygdy8.com/ 选择最喜欢的欧美类
嘿嘿嘿
起始网页之这样：http://www.ygdy8.com/html/gndy/oumei/list_7_1.html
Paste_Image.png
不废话，先上代码：
Paste_Image.png
# -*- coding: utf-8 -*-
import scrapy
from yangguang.items import YangguangItem
from scrapy.spiders import CrawlSpider
class Ygdy8ComSpider(CrawlSpider):
name = "ygdy8.com"
allowed_domains = ["ygdy8.com"]
start_urls = ['http://www.ygdy8.com/html/gndy/oumei/list_7_1.html']
def parse(self, response):
items=[]
print(response.url)
infos= response.xpath('//table[@border="0"]/tr[2]/td[2]/b/a[2]')
for info in infos:
item = YangguangItem()
next_page_link = info.xpath('@href')[0].extract()
next_page_name = info.xpath('text()')[0].extract()
full_page_link= 'http://www.ygdy8.com'+next_page_link#这里一定要加http：// 不然会报错
item['next_page_name'] = next_page_name
item['full_page_link'] = full_page_link
items.append(item)
for item in items:
yield scrapy.Request(url=item['full_page_link'],meta={'item_1':item},callback=self.parse_page) #老规矩，这里把下一页的网址传递给下一页的解析函数
for i in range(2,164): #构造循环函数
url= 'http://www.ygdy8.com/html/gndy/oumei/list_7_%s.html'%i
yield scrapy.Request(url,callback=self.parse)
def parse_page(self,response): #解析传递下来的网址的函数
items= response.meta['item_1']
item = YangguangItem()
dy_link= response.xpath('//table[@border="0"]/tbody/tr/td/a/@href').extract()
item['dy_link']=dy_link
item['next_page_name']=items['next_page_name']
item['full_page_link']= items['full_page_link']
print(item)
yield item
items
import scrapy
class YangguangItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
full_page_link= scrapy.Field()
dy_link = scrapy.Field()
next_page_name= scrapy.Field()
pipline
import pymysql
def dbHandle():
conn = pymysql.connect(
host="localhost",
user="root",
passwd="密码",
charset="utf8",
use_unicode=False
)
return conn
class YangguangPipeline(object):
def process_item(self, item, spider):
dbObject = dbHandle()
cursor = dbObject.cursor()
sql = "insert into ygdy.dy(dy_link,next_page_name,full_page_link) value (%s,%s,%s)"
try:
cursor.execute(sql, (item['dy_link'], item['next_page_name'], item['full_page_link']))
cursor.connection.commit()
except BaseException as e:
print("错误在这里>>>>", e, "<<<<<<错误在这里")
dbObject.rollback()
return item
setting
之前一直存不到数据库，后来问了程老哥，
ITEM_PIPELINES = {
'yangguang.pipelines.YangguangPipeline': 300,
}```
这句话默认是不打开的，所以要在setting里把他打开。
最后看一下存下来的东西
老婆，来找我要电影吧。。
