京东商品爬取

这次的练习主要是对京东的ipad商品页面进行爬取，主页如下：

image.png

items.py

对名字、商铺、价格和营销方式进行抓取

    name = scrapy.Field()
    shop = scrapy.Field()
    icon = scrapy.Field()
    price = scrapy.Field()

jd_spider.py

此处对url是自己观察规律进行构造的，发现url只有page进行了改变，并且是以2的间隔增长

class JdSpiderSpider(scrapy.Spider):
    name = 'jd_spider'
    allowed_domains = ['www.jd.com']
    start_urls = ['https://search.jd.com/Search?keyword=ipad&enc=utf-8&page={}'.format(str(i) for i in range(1,101, 2 ))]


    def parse(self, response):
        lists = response.xpath('//li[@class="gl-item"]/div')
        for list in lists:
            item = JingdongItem()
            item['name'] = list.xpath('.//div[@class="p-name p-name-type-2"]/a/em/text()').extract_first()
            item['shop'] = list.xpath('.//div[@class="p-shop"]/span/a/text()').extract_first()
            item['icon'] = list.xpath('.//div[@class="p-icons"]/i[@class="goods-icons J-picon-tips J-picon-fix"]/text()').extract_first()
            item['price'] = list.xpath('.//div[@class="p-price"]/strong/i/text()').extract_first()
            yield item

•    其他爬虫代码可参考github