Day16内建模块itertools&urllib

2018-04-09  本文已影响16人  林清猫耳

itertools

Python的内建模块itertools提供了非常有用的用于操作迭代对象的函数。
首先,我们看看itertools提供的几个“无限”迭代器:

>>> import itertools
>>> natuals = itertools.count(1)
>>> for n in natuals:
...     print(n)
...
1
2
3
...

因为count()会创建一个无限的迭代器,所以上述代码会打印出自然数序列,根本停不下来,只能按Ctrl+C退出。
cycle()会把传入的一个序列无限重复下去:

>>> cs = itertools.cycle('ABC')
>>> for c in cs:
    print(c)

    
A
B
C
A
B
C
A
B
C
A
B
C
A

同样停不下来。
repeat()负责把一个元素无限重复下去,不过如果提供第二个参数就可以限定重复次数:

>>> ns = itertools.repeat('ABC',3)
>>> for n in ns:
    print(n)

    
ABC
ABC
ABC

无限序列只有在for迭代时才会无限地迭代下去,如果只是创建了一个迭代对象,它不会事先把无限个元素生成出来,事实上也不可能在内存中创建无限多个元素。
无限序列虽然可以无限迭代下去,但是通常我们会通过takewhile()等函数根据条件判断来截取出一个有限的序列:

>>> natuals = itertools.count(1)
>>> ns = itertools.takewhile(lambda x: x <= 10, natuals)
>>> list(ns)
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

tertools提供的几个迭代器操作函数更加有用:

chain()

chain()可以把一组迭代对象串联起来,形成一个更大的迭代器:

>>> for c in itertools.chain('ABC','XYZ'):
    print(c)

    
A
B
C
X
Y
Z

groupby()

groupby()把迭代器中相邻的重复元素挑出来放在一起:

>>> for key, group in itertools.groupby('AAABBBCCAAA'):
    print(key,list(group))

    
A ['A', 'A', 'A']
B ['B', 'B', 'B']
C ['C', 'C']
A ['A', 'A', 'A']

实际上挑选规则是通过函数完成的,只要作用于函数的两个元素返回的值相等,这两个元素就被认为是在一组的,而函数返回值作为组的key。如果我们要忽略大小写分组,就可以让元素'A''a'都返回相同的key:

>>> for key, group in itertools.groupby('AaaVVvCcaaA',lambda c:c.upper()):
    print(key,list(group))

    
A ['A', 'a', 'a']
V ['V', 'V', 'v']
C ['C', 'c']
A ['a', 'a', 'A']

计算圆周率:

import itertools, math

def pi(N):
    k = itertools.count(1,2)
    odd = itertools.takewhile(lambda x: x<=(2*N-1), k)
    s,c = 0,0
    for i in odd:
        s += pow(-1,c)*4/i   # 不能拿odd做运算
        # TypeError: unsupported operand type(s) for /: 'int' and 'itertools.takewhile'
        c += 1

    print(s)

contextlib

@contextmanager这个decorator接受一个generator,用yield语句把with ... as var把变量输出出去,然后,with语句就可以正常地工作了
很多时候,我们希望在某段代码执行前后自动执行特定代码,也可以用@contextmanager实现。例如:

@contextmanager
def tag(name):
    print("<%s>" % name)
    yield
    print("</%s>" % name)

with tag("h1"):
    print("hello")
    print("world")
<h1>
hello
world
</h1>

代码的执行顺序是:

  1. with语句首先执行yield之前的语句,因此打印出<h1>
  2. yield调用会执行with语句内部的所有语句,因此打印出helloworld
  3. 最后执行yield之后的语句,打印出</h1>
    因此,@contextmanager让我们通过编写generator来简化上下文管理。

@closing

如果一个对象没有实现上下文,我们就不能把它用于with语句。这个时候,可以用closing()来把该对象变为上下文对象。例如,用with语句使用urlopen()

from contextlib import closing
from urllib.request import urlopen

with closing(urlopen('https://www.python.org')) as page:
    for line in page:
        print(line)

closing也是一个经过@contextmanager装饰的generator,这个generator编写起来其实非常简单:

@contextmanager
def closing(thing):
    try:
        yield thing
    finally:
        thing.close()

urllib

Get

urllib的request模块可以非常方便地抓取URL内容,也就是发送一个GET请求到指定的页面,然后返回HTTP的响应:
例如,对豆瓣的一个URLhttps://api.douban.com/v2/book/2129650进行抓取,并返回响应:

from urllib import request

with request.urlopen('https://api.douban.com/v2/book/2129650') as f:
    data = f.read()
    print('Status:', f.status, f.reason)
    for k, v in f.getheaders():
        print('%s: %s' % (k, v))
    print('Data:', data.decode('utf-8'))

可以看到HTTP响应的头和JSON数据:

Status: 200 OK
Date: Mon, 09 Apr 2018 13:37:07 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 2118
Connection: close
Vary: Accept-Encoding
X-Ratelimit-Remaining2: 99
X-Ratelimit-Limit2: 100
Expires: Sun, 1 Jan 2006 01:00:00 GMT
Pragma: no-cache
Cache-Control: must-revalidate, no-cache, private
Set-Cookie: bid=aChjmJyJET0; Expires=Tue, 09-Apr-19 13:37:07 GMT; Domain=.douban.com; Path=/
X-DOUBAN-NEWBID: aChjmJyJET0
X-DAE-Node: sindar14d
X-DAE-App: book
Server: dae
Data: {"rating":{"max":10,"numRaters":16,"average":"7.4","min":0},"subtitle":"","author":["廖雪峰"],"pubdate":"2007", ...}

如果我们要想模拟浏览器发送GET请求,就需要使用Request对象,通过往Request对象添加HTTP头,我们就可以把请求伪装成浏览器。例如,模拟iPhone 6去请求豆瓣首页:

from urllib import request

req = request.Request('http://www.douban.com/')
req.add_header('User-Agent', 'Mozilla/6.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/8.0 Mobile/10A5376e Safari/8536.25')
with request.urlopen(req) as f:
    print('Status:', f.status, f.reason)
    for k, v in f.getheaders():
        print('%s: %s' % (k, v))
    print('Data:', f.read().decode('utf-8'))

这样豆瓣会返回适合iPhone的移动版网页:

...
    <meta name="viewport" content="width=device-width, user-scalable=no, initial-scale=1.0, minimum-scale=1.0, maximum-scale=1.0">
    <meta name="format-detection" content="telephone=no">
    <link rel="apple-touch-icon" sizes="57x57" href="http://img4.douban.com/pics/cardkit/launcher/57.png" />
...

Post

如果要以POST发送一个请求,只需要把参数data以bytes形式传入。
我们模拟一个微博登录,先读取登录的邮箱和口令,然后按照weibo.cn的登录页的格式以username=xxx&password=xxx的编码传入:

from urllib import request, parse

print('Login to weibo.cn...')
email = input('Email: ')
passwd = input('Password: ')
login_data = parse.urlencode([
    ('username', email),
    ('password', passwd),
    ('entry', 'mweibo'),
    ('client_id', ''),
    ('savestate', '1'),
    ('ec', ''),
    ('pagerefer', 'https://passport.weibo.cn/signin/welcome?entry=mweibo&r=http%3A%2F%2Fm.weibo.cn%2F')
])

req = request.Request('https://passport.weibo.cn/sso/login')
req.add_header('Origin', 'https://passport.weibo.cn')
req.add_header('User-Agent', 'Mozilla/6.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/8.0 Mobile/10A5376e Safari/8536.25')
req.add_header('Referer', 'https://passport.weibo.cn/signin/login?entry=mweibo&res=wel&wm=3349&r=http%3A%2F%2Fm.weibo.cn%2F')

with request.urlopen(req, data=login_data.encode('utf-8')) as f:
    print('Status:', f.status, f.reason)
    for k, v in f.getheaders():
        print('%s: %s' % (k, v))
    print('Data:', f.read().decode('utf-8'))

如果登录成功,我们获得的响应如下:

Status: 200 OK
Server: nginx/1.2.0
...
Set-Cookie: SSOLoginState=1432620126; path=/; domain=weibo.cn
...
Data: {"retcode":20000000,"msg":"","data":{...,"uid":"1658384301"}}

如果登录失败,我们获得的响应如下:

...
Data: {"retcode":50011015,"msg":"\u7528\u6237\u540d\u6216\u5bc6\u7801\u9519\u8bef","data":{"username":"example@python.org","errline":536}}
上一篇下一篇

猜你喜欢

热点阅读