【二】BeautifulSoup的常用方法
2016-12-08 本文已影响0人
HelloPy
bs4是python爬虫中一个是经典的库，在网上找了一小段html5的代码，自己码了些比较，给出了select find的常用方法，给出的是不依据CSS Selector而依据内容进行地选择
bs4
from bs4 import BeautifulSoup
html='''<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were</p>
<a href="http://www.jb51.net" class="sister" id="link1">Elsie</a>,
<a href="http://www.jb51.net" class="sister" id="link2">Lacie</a> and
<a href="http://www.jb51.net" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
<span class="glyphicon glyphicon-star"></span>'''
soup=BeautifulSoup(html, 'lxml')
print soup.title
#title>The Dormouse's story</title>
print soup.text
#The Dormouse's story
#The Dormouse's story
#Once upon a time there were three little sisters; and their names were
#Elsie,
#Lacie and
#Tillie;
#and they lived at the bottom of a well.
#...
print soup.p #只能找到p 标签的第一条
#<p class="title"><b>The Dormouse's story</b></p>
print soup.select('a') #select返回是一个list
#[<a class="sister" href="http://www.jb51.net" id="link1">Elsie</a>, <a class="sister" href="http://www.jb51.net" id="link2">Lacie</a>, <a class="sister" href="http://www.jb51.net" id="link3">Tillie</a>]
print soup.p.get_text() #效果等同于soup.p.text,只能获取list中的一个 要想得到所有p.text 一定需要遍历么？
#The Dormouse's story
print soup.find_all('p') # find_all找到所有p标签 ，并返回一个list,标头要有引号
#[<p class="title"><b>The Dormouse's story</b></p>,
#<p class="story">Once upon a time there were three little sisters; and their names were</p>, <p class="story">...</p>]
print soup.find(id="link3") #比较而言，select直接可以寻找的只有标签头，find也是只能寻找到list的第一个元素
#<a class="sister" href="http://www.jb51.net" id="link3">Tillie</a>
print soup.find_all(class_="sister") #class要注意class_
#[<a class="sister" href="http://www.jb51.net" id="link1">Elsie</a>,
#<a class="sister" href="http://www.jb51.net" id="link2">Lacie</a>,
# <a class="sister" href="http://www.jb51.net" id="link3">Tillie</a>]
print soup.find_all(href="http://www.jb51.net",class_="sister") #多元素
#[<a class="sister" href="http://www.jb51.net" id="link1">Elsie</a>,
#<a class="sister" href="http://www.jb51.net" id="link2">Lacie</a>,
# <a class="sister" href="http://www.jb51.net" id="link3">Tillie</a>]
print soup.a['href'] #只得到a标签中href值，要求所有a标签似乎需要遍历
#http://www.jb51.net
for tag in soup.find_all(True): #只返回tag
print tag.name
#html
#head
#title
#body
#p
#b
#p
#a
#a
#a
#p
print type(soup.find_all(class_="sister"))
print type(soup.find(class_="sister") )
#<class 'bs4.element.ResultSet'>
#<class 'bs4.element.Tag'>
小结
- bs4的find 与select，还是有一些值得注意的细节要分清，否则就是不停地报错
- 过滤器可以过滤的东西很多，除了上述之外还有方法、正则表达式作为内容
