OpenCV图像全景拼接-黑边处理
2022-04-24 本文已影响0人
JS_XiaoShu
OpenCV为我们提供了图像拼接的工具类：Stitcher，用过的同学都知道，只有拼接后的Mat，没有拼接角度等数据，还需要自己处理拼接后的黑边。
参考资料（python）：https://blog.csdn.net/weixin_41611054/article/details/120340485
上面链接的效果已经很不错了，无奈我需要在Android端实现（C++）未找到相关资料接下来是自己处理的思路
拼接：
vector<cv::Mat>imgs;
imgs.push_back(cv::imread("/Users/xiaoshumac/Documents/st/f0.jpg"));
imgs.push_back(cv::imread("/Users/xiaoshumac/Documents/st/f1.jpg"));
imgs.push_back(cv::imread("/Users/xiaoshumac/Documents/st/f2.jpg"));
imgs.push_back(cv::imread("/Users/xiaoshumac/Documents/st/f3.jpg"));
cv::Mat pano;
cv::Ptr<cv::Stitcher> stitcher = cv::Stitcher::create(cv::Stitcher::PANORAMA);
cv::Stitcher::Status status = stitcher->stitch(imgs, pano);//拼接
效果：
WX20220424-153111@2x.png
可见效果是个不规则的图形，经过多次尝试无果后，我用了最笨的方法---计算像素点来判断黑边，我的思路是从四个角往里面算，找到第一个不是黑色的为止
转成黑白图片：
cv::Mat stitched;cv::copyMakeBorder(pano, stitched, 0, 0, 0, 0, cv::BORDER_CONSTANT, (0, 0, 0));
cv::Mat gray;cv::cvtColor(stitched, gray, cv::COLOR_BGR2GRAY);
cv::Mat thresh = pano.clone();
cv::threshold(gray, thresh, 0, 255, cv::THRESH_BINARY);
WX20220424-153552@2x.png
四个角往里面找思路如图：
WX20220424-153727@2x.png
找左上角：
cv::Point2f leftTopPt;
for (int i = 1; i < thresh.rows; ++i) {
char pixel = thresh.at<char>(i, i);
if (pixel != 0) { //不等于黑色
leftTopPt.x = i;
leftTopPt.y = i;
break;
}
}
cv::Scalar color(10, 10, 255);
// 用黑白图片计算的结果，在原图画个小圆点（黑白图上画不够明显）
cv::circle(pano, leftTopPt, 5, color);
WX20220424-155635@2x.png
（画红点和先是为了方便自己看到裁剪的区域）
以此类推，找出左上角、右上角、左下角、右下角的点，连接点画线：
cv::Point2f leftTopPt;
for (int i = 1; i < thresh.rows; ++i) {
char pixel = thresh.at<char>(i, i);
if (pixel != 0) { //不等于黑色
leftTopPt.x = i;
leftTopPt.y = i;
break;
}
}
cv::Point2f rightTopPt;
for (int i = 1; i < thresh.rows; ++i) {
int x = thresh.cols - i;
int y = i;
char pixel = thresh.at<char>(y, x);
if (pixel != 0) {
rightTopPt.x = x;
rightTopPt.y = y;
break;
}
}
cv::Point2f leftBottomPt;
for (int i = 1; i < thresh.rows; ++i) {
int x = i;
int y = thresh.rows - i;
char pixel = thresh.at<char>(y, x);
if (pixel != 0) {
leftBottomPt.x = x;
leftBottomPt.y = y;
break;
}
}
cv::Point2f rightBottomPt;
for (int i = 1; i < thresh.rows; ++i) {
int x = thresh.cols - i;
int y = thresh.rows - i;
char pixel = thresh.at<char>(y, x);
if (pixel != 0) {
rightBottomPt.x = x;
rightBottomPt.y = y;
break;
}
}
cv::Scalar color(10, 10, 255);
// 用黑白图片计算的结果，在原图画个小圆点（黑白图上画不够明显）
cv::circle(pano, leftTopPt, 5, color, -1);
cv::circle(pano, rightTopPt, 5, color, -1);
cv::circle(pano, leftBottomPt, 5, color, -1);
cv::circle(pano, rightBottomPt, 5, color, -1);
cv::line(pano, leftTopPt, rightTopPt, color);
cv::line(pano, leftTopPt, leftBottomPt, color);
cv::line(pano, rightTopPt, rightBottomPt, color);
cv::line(pano, leftBottomPt, rightBottomPt, color);
WX20220424-160257@2x.png
是不是有亿点点感觉了？明显按照这个线裁剪是不行的，线不是直，接下来处理四边的线，以小的为准：
int topMaxY = max(leftTopPt.y , rightTopPt.y);
int leftMaxX = max(leftTopPt.x , leftBottomPt.x);
int rightMinX = min(rightTopPt.x , rightBottomPt.x);
int bottomMinY = min(leftBottomPt.y , rightBottomPt.y);
leftTopPt.y = topMaxY;
rightTopPt.y = topMaxY;
leftTopPt.x = leftMaxX;
leftBottomPt.x = leftMaxX;
rightTopPt.x = rightMinX;
rightBottomPt.x = rightMinX;
leftBottomPt.y = bottomMinY;
rightBottomPt.y = bottomMinY;
WX20220424-160853@2x.png
然后裁剪且保存，完整代码：
vector<cv::Mat> imgs;
imgs.push_back(cv::imread("/Users/xiaoshumac/Documents/st/f0.jpg"));
imgs.push_back(cv::imread("/Users/xiaoshumac/Documents/st/f1.jpg"));
imgs.push_back(cv::imread("/Users/xiaoshumac/Documents/st/f2.jpg"));
imgs.push_back(cv::imread("/Users/xiaoshumac/Documents/st/f3.jpg"));
cv::Mat pano;
cv::Ptr<cv::Stitcher> stitcher = cv::Stitcher::create(cv::Stitcher::PANORAMA);
cv::Stitcher::Status status = stitcher->stitch(imgs, pano);//拼接
cv::Mat stitched;
cv::copyMakeBorder(pano, stitched, 0, 0, 0, 0, cv::BORDER_CONSTANT, (0, 0, 0));
cv::Mat gray;
cv::cvtColor(stitched, gray, cv::COLOR_BGR2GRAY);
cv::Mat thresh = pano.clone();
cv::threshold(gray, thresh, 0, 255, cv::THRESH_BINARY);
cv::Point2f leftTopPt;
for (int i = 1; i < thresh.rows; ++i) {
char pixel = thresh.at<char>(i, i);
if (pixel != 0) { //不等于黑色
leftTopPt.x = I;
leftTopPt.y = I;
break;
}
}
cv::Point2f rightTopPt;
for (int i = 1; i < thresh.rows; ++i) {
int x = thresh.cols - I;
int y = I;
char pixel = thresh.at<char>(y, x);
if (pixel != 0) {
rightTopPt.x = x;
rightTopPt.y = y;
break;
}
}
cv::Point2f leftBottomPt;
for (int i = 1; i < thresh.rows; ++i) {
int x = I;
int y = thresh.rows - I;
char pixel = thresh.at<char>(y, x);
if (pixel != 0) {
leftBottomPt.x = x;
leftBottomPt.y = y;
break;
}
}
cv::Point2f rightBottomPt;
for (int i = 1; i < thresh.rows; ++i) {
int x = thresh.cols - I;
int y = thresh.rows - I;
char pixel = thresh.at<char>(y, x);
if (pixel != 0) {
rightBottomPt.x = x;
rightBottomPt.y = y;
break;
}
}
int topMaxY = max(leftTopPt.y , rightTopPt.y);
int leftMaxX = max(leftTopPt.x , leftBottomPt.x);
int rightMinX = min(rightTopPt.x , rightBottomPt.x);
int bottomMinY = min(leftBottomPt.y , rightBottomPt.y);
leftTopPt.y = topMaxY;
rightTopPt.y = topMaxY;
leftTopPt.x = leftMaxX;
leftBottomPt.x = leftMaxX;
rightTopPt.x = rightMinX;
rightBottomPt.x = rightMinX;
leftBottomPt.y = bottomMinY;
rightBottomPt.y = bottomMinY;
//
// cv::Scalar color(10, 10, 255);
// // 用黑白图片计算的结果，在原图画个小圆点（黑白图上画不够明显）
// cv::circle(pano, leftTopPt, 5, color, -1);
// cv::circle(pano, rightTopPt, 5, color, -1);
// cv::circle(pano, leftBottomPt, 5, color, -1);
// cv::circle(pano, rightBottomPt, 5, color, -1);
//
// cv::line(pano, leftTopPt, rightTopPt, color);
// cv::line(pano, leftTopPt, leftBottomPt, color);
// cv::line(pano, rightTopPt, rightBottomPt, color);
// cv::line(pano, leftBottomPt, rightBottomPt, color);
// 裁剪
cv::Mat tempMat = pano(cv::Rect(leftTopPt.x, leftTopPt.y, rightBottomPt.x - leftTopPt.x, rightBottomPt.y - leftTopPt.y));
// 保存（画点画线仅为了方便测试，保存注释画的代码）
cv::imwrite("/Users/xiaoshumac/Documents/st/tempMat.jpg", tempMat);
tempMat.jpg
这就完事了？并没有，这只是针对上图这类效果有效，对于9宫格裁剪或者拼接后面有大面积黑边仅靠上面代码还无法完美处理，有需要的同学我再更新，暂时写到这了
因为是计算黑色点作为边界处理，所以上面代码对于夜景拼接处理不好，还有就是计算像素点是属于”笨“方法，有更优的思路麻烦大佬告知下
关于耗时：虽然上面用了很多for循环，好在OpenCV取像素点是非常快的，上面图像在我的电脑：500ms左右, 小米9也在2秒以内
---2022/04/26---
夜景拼接效果也处理好了，需要的同学再更新
