石头的作业~第一周第三节练习项目：爬取租房信息

实战计划0430-石头的练习作业

练习的要求

课程要求抓取的内容：

实现效果如下

运行结果

相关代码

__author__ = 'daijielei'

'''
11课时练习，从小猪短租处抓取一堆租房详情
'''
from bs4 import BeautifulSoup   #解析html文档用
import requests                 #抓取html页面用
import time

#urls用来制定抓取范围，header该处无用处
header = {
    'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36',
    'Host':'bj.xiaozhu.com'
}


#getlist用来抓取对应网页上的每个列表的跳转url
#测试用url = "http://bj.xiaozhu.com/search-duanzufang-p1-0/"
def getlist(url):
    webData = requests.get(url,headers=header)
    soup = BeautifulSoup(webData.text,'lxml')

    houselists = soup.select('ul > li > a')

    for houselist in houselists:
        listUrl = str(houselist.get('href'))
        if(listUrl.find('http://bj.xiaozhu.com/fangzi/')!=-1):#确认链接内容为详情页，则进行详情页信息抓取
            #print(listUrl)
            getInfoPage(listUrl)


#getInfoPage用来抓取详情页里的相关信息，如标题、地址、金额、房屋图片、拥有人图片等等
#测试用url = "http://bj.xiaozhu.com/fangzi/525041101.html"
def getInfoPage(url):
    webData = requests.get(url,headers=header)
    soup = BeautifulSoup(webData.text,'lxml')
    time.sleep(2)

    titles = soup.select('body > div.wrap.clearfix.con_bg > div.con_l > div.pho_info > h4 > em')
    address = soup.select('body > div.wrap.clearfix.con_bg > div.con_l > div.pho_info > p > span.pr5')
    pays = soup.select('#pricePart > div.day_l > span')
    houseimages = soup.select('#curBigImage')
    ownerimages = soup.select('#floatRightBox > div.js_box.clearfix > div.member_pic > a > img')
    ownernames = soup.select('#floatRightBox > div.js_box.clearfix > div.w_240 > h6 > a')
    ownerSexs = soup.select('#floatRightBox > div.js_box.clearfix > div.w_240 > h6 > span')

    for title,addres,pay,houseimage,ownerimage,ownername,ownerSex in zip(titles,address,pays,houseimages,ownerimages,ownernames,ownerSexs):
        data = {
            'title':title.get_text(),
            'addres':addres.get_text(),
            'pay':pay.get_text(),
            'houseimage':houseimage.get('src'),
            'ownerimage':ownerimage.get('src'),
            'ownername':ownername.get_text(),
            'ownerSex':'female'
        }
        if('member_boy_ico' in str(ownerSex['class'])):#该处用来判断是男性还是女性，男性是通过标签来区分
            data['ownerSex'] = "male"
        print(data)

def startCatch():
    urls = ["http://bj.xiaozhu.com/search-duanzufang-p{}-0/".format(str(i)) for i in range(1,3,1)]
    for url in urls:
        getlist(url)

#入口，该语句只有直接执行时会被调用
if __name__ == '__main__':
    startCatch()

笔记、想法、总结

1、这个代码是之前写的了，所以显得比较乱，尤其是抓取的部分现在写的话我就不会用这种写法了

2、这里面主要有3个函数，功能还是比较清晰，先爬取列表，再进列表里去爬取数据：

def startCatch():    #用来便利列表，会调用getlist
def getlist(url):      #用来抓取列表的数据，会调用getInfoPage
def getInfoPage(url):#抓取详情页的数据

3、比较难抓的数据只有性别信息，因为不是直接能抓取的值，而是要做一下判断：
在html中为如下,是用span里的class来区别男女，男为member_boy_ico、女为member_girl_ico：

<h6>
<a class="lorder_name" href="http://www.xiaozhu.com/fangdong/617264400/" title="ocheese" target="_blank">ocheese</a>
<span class="member_girl_ico"></span>
</h6>

代码中就是做一下判断即可

        if('member_boy_ico' in str(ownerSex['class'])):#该处用来判断是男性还是女性，男性是通过标签来区分
            data['ownerSex'] = "male"