python爬虫小试
问题来源:最近想在一品威客上寻找兼职，但是发现一品威客的兼职信息不支持按任务或者投标人数进行排序，因而想通过爬虫将兼职信息爬取下来，然后在本地进行排序查找。
一.搭建scrapy环境
1.安装python3.6
ps：这个网上教程很多
2.安装pywin32
ps：利用pip进行安装，在cmd命令窗口下输入命令：python -m pip install pywin32
3安装Twisted
ps：这个安装可以参https://blog.csdn.net/sinat_35637319/article/details/78940415
4.安装scrapy
利用命令pip install Scrapy 就OK了
二.scrapy的基本命令
1.建立工程命令：scrapy startproject xxx
2.运行工程命令：scrapy crawl xxx
三.在Pycharm进行爬虫开发
scrapy 需要通过命令窗口执行命令scrapy startproject xxx 生成工程（不支持在pycharm上进行工程建立），然后可以利用pycharm进行代码编辑（直接在pycharm上打开这个工程）
若想在pycharm上进行调试，需要如图所示的配置。
pycharm上的环境配置
到此为止，就可以在pycharm上进行开发了。
三.如何爬数据
1.scrapy运行原理
scrapy框架原理图
pycharm工程架构图
scrapy框架数据流与代码执行过程
a. spider->scrapy->Internet
在工程中只需要关心spiders的weike.py的实现
b.Internet->scrapy->spider->pipeline
spiders的weike.py -----(items.py)----->pipelines.py
weike.py :负责提供页面请求的uri以及网页响应得到的数据解析
pipelines.py：负责保存网页响应的数据
items.py：负责保存的载体，是一个字典对象，类似于数据model,
setting.py:负责配置工程的重要参数
2.weike.py的编写
# -*- coding: utf-8 -*-
import re
import scrapy
from weike.items import WeikeItem
class weike(scrapy.Spider):
# 爬虫名
name = "weike"
# 爬虫作用范围
allowed_domains = ["epwk.com"]
url = "http://www.epwk.com/soft/task/"
offset = 1
string="page%d.html" % offset
# 起始url
start_urls = [url + string]
def parse(self, response):
cotain=response.xpath("//div[@class='task_class_list_li'] ")
for each in cotain.xpath("//div[@class='task_class_list_li_box'] "):
# 初始化模型对象
item = WeikeItem()
price = each.xpath("./div[1]/h3/b/text()").extract()[0]
item['projectPrice']=re.split(u'\xa0',price)[1]
projectName= each.xpath("./div[1]/h3/a/text()").extract()[0]
item['projectName']=str(projectName).strip()
canjiaCount = each.xpath("./div[1]/samp/text()").extract()[0]
item['canjiaCount']=re.sub("\D", "", canjiaCount)
item['viewCount'] = each.xpath("./div[1]/samp/font/text()").extract()[0]
try:
item['time1'] = each.xpath("./div[2]/span/span[1]/text() ").extract()[0]
item['time2'] = each.xpath("./div[2]/span/span[2]/text() ").extract()[0]
except:
pass
finally:
pass
yield item
if self.offset < 10:
self.offset += 1
string = "page%d.html" % self.offset
# 每次处理完一页的数据之后，重新发送下一页页面请求
# self.offset自增1，同时拼接为新的url，并调用回调函数self.parse处理Response
yield scrapy.Request(self.url + str(string), callback = self.parse)
3.pipelines.py的编写
# -*- coding: utf-8 -*-
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html
import json
class WeikePipeline(object):
"""
功能：保存item数据
"""
def __init__(self):
self.filename = open("weike.json", "w",encoding='utf-8')
def process_item(self, item, spider):
text= json.dumps(dict(item), ensure_ascii=False) + ",\n"
text.replace(u'\xa0', u' ')
self.filename.write(bytes.decode(text.encode("utf-8")))
return item
def close_spider(self, spider):
self.filename.close()
4.items.py的编写
# -*- coding: utf-8 -*-
# Define here the models for your scraped items
#
# See documentation in:
# https://doc.scrapy.org/en/latest/topics/items.html
import scrapy
class WeikeItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
# 金额
projectPrice = scrapy.Field()
# 名称
projectName = scrapy.Field()
# 投标人数
canjiaCount = scrapy.Field()
# 浏览人数
viewCount = scrapy.Field()
# 截止日期
time1 = scrapy.Field()
time2 = scrapy.Field()
5.setting.py的编写
# -*- coding: utf-8 -*-
BOT_NAME = 'weike'
SPIDER_MODULES = ['weike.spiders']
NEWSPIDER_MODULE = 'weike.spiders'
# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'Tencent (+http://www.yourdomain.com)'
# Obey robots.txt rules
ROBOTSTXT_OBEY = True
# 设置请求头部，添加url
DEFAULT_REQUEST_HEADERS = {
"User-Agent" : "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0;",
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
}
# 设置item——pipelines
ITEM_PIPELINES = {
'weike.pipelines.WeikePipeline': 300,
}
6.运行结果
点击pycharm中的运行可以得到如下结果weike.json，数据截图如下：
输出结果截图
现在得到的json结果，也可以保存成excel进行分析
总结
