Day16内建模块itertools&urllib
itertools
Python的内建模块itertools
提供了非常有用的用于操作迭代对象的函数。
首先,我们看看itertools
提供的几个“无限”迭代器:
>>> import itertools
>>> natuals = itertools.count(1)
>>> for n in natuals:
... print(n)
...
1
2
3
...
因为count()
会创建一个无限的迭代器,所以上述代码会打印出自然数序列,根本停不下来,只能按Ctrl+C
退出。
cycle()
会把传入的一个序列无限重复下去:
>>> cs = itertools.cycle('ABC')
>>> for c in cs:
print(c)
A
B
C
A
B
C
A
B
C
A
B
C
A
同样停不下来。
repeat()
负责把一个元素无限重复下去,不过如果提供第二个参数就可以限定重复次数:
>>> ns = itertools.repeat('ABC',3)
>>> for n in ns:
print(n)
ABC
ABC
ABC
无限序列只有在for
迭代时才会无限地迭代下去,如果只是创建了一个迭代对象,它不会事先把无限个元素生成出来,事实上也不可能在内存中创建无限多个元素。
无限序列虽然可以无限迭代下去,但是通常我们会通过takewhile()
等函数根据条件判断来截取出一个有限的序列:
>>> natuals = itertools.count(1)
>>> ns = itertools.takewhile(lambda x: x <= 10, natuals)
>>> list(ns)
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
tertools
提供的几个迭代器操作函数更加有用:
chain()
chain()
可以把一组迭代对象串联起来,形成一个更大的迭代器:
>>> for c in itertools.chain('ABC','XYZ'):
print(c)
A
B
C
X
Y
Z
groupby()
groupby()
把迭代器中相邻的重复元素挑出来放在一起:
>>> for key, group in itertools.groupby('AAABBBCCAAA'):
print(key,list(group))
A ['A', 'A', 'A']
B ['B', 'B', 'B']
C ['C', 'C']
A ['A', 'A', 'A']
实际上挑选规则是通过函数完成的,只要作用于函数的两个元素返回的值相等,这两个元素就被认为是在一组的,而函数返回值作为组的key。如果我们要忽略大小写分组,就可以让元素'A'
和'a'
都返回相同的key:
>>> for key, group in itertools.groupby('AaaVVvCcaaA',lambda c:c.upper()):
print(key,list(group))
A ['A', 'a', 'a']
V ['V', 'V', 'v']
C ['C', 'c']
A ['a', 'a', 'A']
计算圆周率:
import itertools, math
def pi(N):
k = itertools.count(1,2)
odd = itertools.takewhile(lambda x: x<=(2*N-1), k)
s,c = 0,0
for i in odd:
s += pow(-1,c)*4/i # 不能拿odd做运算
# TypeError: unsupported operand type(s) for /: 'int' and 'itertools.takewhile'
c += 1
print(s)
contextlib
@contextmanager
这个decorator接受一个generator,用yield
语句把with ... as var
把变量输出出去,然后,with
语句就可以正常地工作了
很多时候,我们希望在某段代码执行前后自动执行特定代码,也可以用@contextmanager
实现。例如:
@contextmanager
def tag(name):
print("<%s>" % name)
yield
print("</%s>" % name)
with tag("h1"):
print("hello")
print("world")
<h1>
hello
world
</h1>
代码的执行顺序是:
-
with
语句首先执行yield
之前的语句,因此打印出<h1>
; -
yield
调用会执行with
语句内部的所有语句,因此打印出hello
和world
; - 最后执行
yield
之后的语句,打印出</h1>
。
因此,@contextmanager
让我们通过编写generator来简化上下文管理。
@closing
如果一个对象没有实现上下文,我们就不能把它用于with
语句。这个时候,可以用closing()
来把该对象变为上下文对象。例如,用with
语句使用urlopen()
:
from contextlib import closing
from urllib.request import urlopen
with closing(urlopen('https://www.python.org')) as page:
for line in page:
print(line)
closing
也是一个经过@contextmanager装饰的generator,这个generator编写起来其实非常简单:
@contextmanager
def closing(thing):
try:
yield thing
finally:
thing.close()
urllib
Get
urllib的request
模块可以非常方便地抓取URL内容,也就是发送一个GET请求到指定的页面,然后返回HTTP的响应:
例如,对豆瓣的一个URLhttps://api.douban.com/v2/book/2129650
进行抓取,并返回响应:
from urllib import request
with request.urlopen('https://api.douban.com/v2/book/2129650') as f:
data = f.read()
print('Status:', f.status, f.reason)
for k, v in f.getheaders():
print('%s: %s' % (k, v))
print('Data:', data.decode('utf-8'))
可以看到HTTP响应的头和JSON数据:
Status: 200 OK
Date: Mon, 09 Apr 2018 13:37:07 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 2118
Connection: close
Vary: Accept-Encoding
X-Ratelimit-Remaining2: 99
X-Ratelimit-Limit2: 100
Expires: Sun, 1 Jan 2006 01:00:00 GMT
Pragma: no-cache
Cache-Control: must-revalidate, no-cache, private
Set-Cookie: bid=aChjmJyJET0; Expires=Tue, 09-Apr-19 13:37:07 GMT; Domain=.douban.com; Path=/
X-DOUBAN-NEWBID: aChjmJyJET0
X-DAE-Node: sindar14d
X-DAE-App: book
Server: dae
Data: {"rating":{"max":10,"numRaters":16,"average":"7.4","min":0},"subtitle":"","author":["廖雪峰"],"pubdate":"2007", ...}
如果我们要想模拟浏览器发送GET请求,就需要使用Request对象,通过往Request对象添加HTTP头,我们就可以把请求伪装成浏览器。例如,模拟iPhone 6去请求豆瓣首页:
from urllib import request
req = request.Request('http://www.douban.com/')
req.add_header('User-Agent', 'Mozilla/6.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/8.0 Mobile/10A5376e Safari/8536.25')
with request.urlopen(req) as f:
print('Status:', f.status, f.reason)
for k, v in f.getheaders():
print('%s: %s' % (k, v))
print('Data:', f.read().decode('utf-8'))
这样豆瓣会返回适合iPhone的移动版网页:
...
<meta name="viewport" content="width=device-width, user-scalable=no, initial-scale=1.0, minimum-scale=1.0, maximum-scale=1.0">
<meta name="format-detection" content="telephone=no">
<link rel="apple-touch-icon" sizes="57x57" href="http://img4.douban.com/pics/cardkit/launcher/57.png" />
...
Post
如果要以POST发送一个请求,只需要把参数data
以bytes形式传入。
我们模拟一个微博登录,先读取登录的邮箱和口令,然后按照weibo.cn的登录页的格式以username=xxx&password=xxx
的编码传入:
from urllib import request, parse
print('Login to weibo.cn...')
email = input('Email: ')
passwd = input('Password: ')
login_data = parse.urlencode([
('username', email),
('password', passwd),
('entry', 'mweibo'),
('client_id', ''),
('savestate', '1'),
('ec', ''),
('pagerefer', 'https://passport.weibo.cn/signin/welcome?entry=mweibo&r=http%3A%2F%2Fm.weibo.cn%2F')
])
req = request.Request('https://passport.weibo.cn/sso/login')
req.add_header('Origin', 'https://passport.weibo.cn')
req.add_header('User-Agent', 'Mozilla/6.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/8.0 Mobile/10A5376e Safari/8536.25')
req.add_header('Referer', 'https://passport.weibo.cn/signin/login?entry=mweibo&res=wel&wm=3349&r=http%3A%2F%2Fm.weibo.cn%2F')
with request.urlopen(req, data=login_data.encode('utf-8')) as f:
print('Status:', f.status, f.reason)
for k, v in f.getheaders():
print('%s: %s' % (k, v))
print('Data:', f.read().decode('utf-8'))
如果登录成功,我们获得的响应如下:
Status: 200 OK
Server: nginx/1.2.0
...
Set-Cookie: SSOLoginState=1432620126; path=/; domain=weibo.cn
...
Data: {"retcode":20000000,"msg":"","data":{...,"uid":"1658384301"}}
如果登录失败,我们获得的响应如下:
...
Data: {"retcode":50011015,"msg":"\u7528\u6237\u540d\u6216\u5bc6\u7801\u9519\u8bef","data":{"username":"example@python.org","errline":536}}