Python常用的几种去重方法

一、对列表去重

case1:用集合的特性set()，去重后顺序会改变

a = ['1', '2', 'x', '5', 'x']
print(set(a))     ---{'x', '5', '1', '2'}

case1.1：可以通过列表中索引（index）的方法保证去重后的顺序不变

a = [1,4,4,2,3,4,5,6,1]
b = list(set(a))
b.sort(key=a.index)
print(b)

case2:使用循环查找的方式，不改变顺序

a = ['1', '2', 'x', '5', 'x']
list = []
for i in a:
    if i not in list:
        list.append(i)
print(list)       ----['1', '2', 'x', '5']

case3:通过删除索引

a = [1,4,4,2,3,4,5,6,1]
b = a[:]
for k in b:
    if b.count(k) > 1:
        del b[b.index(k)]
b.sort(key=a.index)  # 解决顺序问题
print(b)

case4:itertools.groupby

import itertools
l1 = [1,4,4,2,3,4,5,6,1]
l1.sort()
l = []
it = itertools.groupby(l1)
for k,g in it:
    l.append(k)
print(l)      # [1, 2, 3, 4, 5, 6]

case5:fromkeys

l1 = [1,4,4,2,3,4,5,6,1]
t = list({}.fromkeys(l1).keys())
# 解决顺序问题
t.sort(key=l1.index)
print(t)         # [1, 4, 2, 3, 5, 6]

case6:reduce方法

from functools import reduce
l1 = [1,4,4,2,3,4,5,6,1]
func = lambda x,y:x if y in x else x + [y]
print(reduce(func,[[],]+l1))   # [1, 4, 2, 3, 5, 6]