利用图片识别解决密码控件问题

在Appium操作时弹出普通窗口以及输入密码时的随机软键盘,由于焦点没有切换过来,或者整个软键盘是一个整体,导致无法直接通过id,xpath定位,这个时候只能利用图片识别来解决问题.

思路:
1. 保存要识别的图片A
2. App运行时,到了目标页面,截屏 self.driver.get_screenshot_as_file('/Users/xxx/test/jianpan.jpg')
3. 对比,寻找图片A的坐标
4. 点击

操作:

1. 安装opencv库 pip3 install opencv-python
2. 方法代码如下:

import cv2
import numpy as np
import time

def get_pay_keyboard_number_location(self,impath, target,fit_num): #fit_num是匹配度,如 0.95,0.85

        print("start find pic")
        positions = {}

        start = time.time()
        img_rgb = cv2.imread(impath)

        teNum = "done"

        template = cv2.imread(target)
        h, w = template.shape[:-1]

        res = cv2.matchTemplate(img_rgb, template, cv2.TM_CCOEFF_NORMED)
      
        threshold = fit_num  # 匹配度参数，1为完全匹配
        loc = np.where(res >= threshold)
        if len(loc) > 0:
            # positions[teNum] = zip(*loc[::-1])[0]  # python2的写法
            print("找到了匹配图")
            zipped = zip(*loc[::-1])  #list[::-1] 相当于起点为最后的一个,终点为第一个,然后一次减少一个
            positions[teNum] = (list(zipped))[0]
            
        else:
            print("Can not found pic: ")

        end = time.time()
        print(end - start)

        return positions[teNum]  

3. 调用代码:

        impath1 = "/Users/xxx/2019work/eebank/pic/jianpan1.png"
        targetPath_a = "/Users/xxx/2019work/eebank/pic/a.png"

        ls = self.get_pay_keyboard_number_location(impath1, targetPath_a,0.85) #在键盘图片找到字母a
        # ls = (62,1046) 这样的数组
     
        dd = ls[0]
        kk = ls[1]
        m = (dd/2, kk / 2)
        #点击
        self.driver.tap([m],20)