Python实战-第二节练习项目：爬取商品信息

最终成果

[{'image': 'img/pic_0000_073a9256d9624c92a05dc680fc28865f.jpg',
  'price': '$24.99',
  'review': '65 reviews',
  'star': '★★★★★',
  'titile': 'EarPod'},
 {'image': 'img/pic_0005_828148335519990171_c234285520ff.jpg',
  'price': '$64.99',
  'review': '12 reviews',
  'star': '★★★★☆',
  'titile': 'New Pocket'},
 {'image': 'img/pic_0006_949802399717918904_339a16e02268.jpg',
  'price': '$74.99',
  'review': '31 reviews',
  'star': '★★★★☆',
  'titile': 'New sunglasses'},
 {'image': 'img/pic_0008_975641865984412951_ade7a767cfc8.jpg',
  'price': '$84.99',
  'review': '6 reviews',
  'star': '★★★☆☆',
  'titile': 'Art Cup'},
 {'image': 'img/pic_0001_160243060888837960_1c3bcd26f5fe.jpg',
  'price': '$94.99',
  'review': '18 reviews',
  'star': '★★★★☆',
  'titile': 'iphone gamepad'},
 {'image': 'img/pic_0002_556261037783915561_bf22b24b9e4e.jpg',
  'price': '$214.5',
  'review': '18 reviews',
  'star': '★★★★☆',
  'titile': 'Best Bed'},
 {'image': 'img/pic_0011_1032030741401174813_4e43d182fce7.jpg',
  'price': '$500',
  'review': '35 reviews',
  'star': '★★★★☆',
  'titile': 'iWatch'},
 {'image': 'img/pic_0010_1027323963916688311_09cc2d7648d9.jpg',
  'price': '$15.5',
  'review': '8 reviews',
  'star': '★★★★☆',
  'titile': 'Park tickets'}]

我的代码

from bs4 import BeautifulSoup

path='./1_2answer_of_homework/index.html'

with open(path,'r') as wb_data:
    soup = BeautifulSoup(wb_data,'lxml')
    reviews = soup.select(" body > div > div > div.col-md-9 > div > div > div > div.ratings > p.pull-right")
    grades = soup.select(" body > div > div > div.col-md-9 > div > div > div > div.ratings > p > span") # 删除所有冒号后的信息
    prices = soup.select(" body > div > div > div.col-md-9 > div > div > div > div.caption > h4.pull-right")
    titles = soup.select(" body > div > div > div.col-md-9 > div > div > div > div.caption > h4 > a")
    images = soup.select(" body > div > div > div.col-md-9 > div > div > div > img")
    # print(reviews,stars,prices,titles,images,sep='\n---------------\n')

grades = str(grades)
grades = grades.replace('<span class="glyphicon glyphicon-star"></span>', '★')
grades = grades.replace('<span class="glyphicon glyphicon-star-empty"></span>', '☆') # 替换grades为星符
grades = grades.replace(', ','')
grades = grades[1:(len(grades)-1)] #选取所有星符
num = range(int(len(grades)/5))

stars = []
for s in num:
    print(5*s,5*(s+1))
    stars.append(''.join(grades[5*s:5*(s+1)]))  # 每5个星符组成一个商品评价
    print(stars)

info = []
# 采用字典进行集体查询
for review,star,price,title,image in zip(reviews,stars,prices,titles,images):  
    data = {
        'review':review.get_text(),
        'star': star,
        'price': price.get_text(),
        'titile': title.get_text(),
        'image': image.get('src') # 图片形式
    }
    info.append(data)

   list(info)

总结

• 使用相对路径
• 采用BeautifulSoup( )爬取网页信息
• 检查网页，拷贝元素的CSS Selector路径而非XPath路径
• 用 ______ .select 获取网页中某类元素的所有路径时，应把冒号后的部分去掉，如：body > div:nth-child(2) > div > div.col-md-9 > div:nth-child(2) > div:nth-child(3) > div > img 应改为body > div > div > div.col-md-9 > div > div > div > img （注意： ______ 表示变量名）
• 可以采用字典进行集体查询
• 遇到多对一的情况时，如：多个标签对应同一主题，______.select 应去除Selector路径的最后一项，并且在集体查询时用列表展示 list( ____.stripped_strings)
• 图片采用 ______.get('src')，文字采用______.get_text()