Shulou(Shulou.com)06/02 Report--

This article mainly explains "how to use OpenCV for image panoramic stitching", the content of the article is simple and clear, easy to learn and understand, the following please follow the editor's ideas slowly in depth, together to study and learn "how to use OpenCV for image panoramic stitching"!

Image stitching is one of the most successful applications in computer vision. Nowadays, it is difficult to find a mobile phone or image processing API that does not include this feature. In this article, we will discuss how to use Python and OpenCV for image stitching. That is, given two images that share some common areas, the goal is to "stitch" them and create a panoramic image scene. Of course, it can also be given multiple images, but it will always be converted into the problem of two images sharing some common areas, so this paper introduces it in the simplest form.

The main knowledge points of this article include the following:

Key point detection

Local invariant descriptors (SIFT,SURF, etc.)

Feature matching

Homography estimation using RANSAC

Perspective transformation

The two images we need to splice are as follows:

Feature detection and extraction

Given the above pair of images, we want to stitch them together to create a panoramic scene. It is important to note that both images need to have some common areas. Of course, the two images we have given above are ideal. Sometimes although the two images have a common area, they may also have the influence of scaling, rotation, coming from different cameras and other factors. But in either case, we need to detect the feature points in the image.

Key point detection

The initial and possibly naive approach is to use algorithms such as Harris Corners to extract key points. Then, we can try to match the corresponding key points based on some similarity measure (such as Euclidean distance). As we all know, the corner has a good characteristic: the corner does not change. This means that once a corner is detected, the corner will still exist even if the image is rotated.

But what if we rotate and then scale the image? In this case, it will be very difficult for us because the size of the corner is the same. In other words, if we enlarge the image, the previously detected corner may become a line!

All in all, we need features that remain the same as rotation and scaling. That's the more powerful approach (such as SIFT,SURF and ORB).

Key points and descriptors

Methods such as SIFT and SURF try to solve the limitations of corner detection algorithms. In general, the corner detector algorithm uses a fixed-size kernel to detect regions of interest (corners) on the image. It is not difficult to see that when we zoom the image, the kernel may become too small or too large. To address this limitation, methods such as SIFT use Gaussian difference (DoD). The idea is to apply DoD to different scaled versions of the same image. It also uses adjacent pixel information to find and refine key points and corresponding descriptors.

First, we need to load two images, a query image and a training image. Initially, we first extract key points and descriptors from both. By using the OpenCV detectAndCompute () function, we can do it in one step. Note that in order to use detectAndCompute (), we need an instance of a key detector and descriptor object. It can be ORB,SIFT or SURF and so on. In addition, we convert the image to grayscale before entering it into detectAndCompute ().

Def detectAndDescribe (image, method=None): "Compute key points and feature descriptors using an specific method"

Assert method is not None, "You need to define a feature detection method. Values are: 'sift',' surf'"

# detect and extract features from the image if method = = 'sift': descriptor = cv2.xfeatures2d.SIFT_create () elif method =' surf': descriptor = cv2.xfeatures2d.SURF_create () elif method = = 'brisk': descriptor = cv2.BRISK_create () elif method = =' orb': descriptor = cv2.ORB_create ()

# get keypoints and descriptors (kps, features) = descriptor.detectAndCompute (image, None)

Return (kps, features)

We set a set of keys and descriptors for both images. If we use SIFT as the feature extractor, it will return a 128D feature vector for each key point. If we choose SURF, we will get 64-dimensional feature vectors. The following figure shows the results obtained using SIFT,SURF,BRISK and ORB.

Using ORB and hamming distance to detect key points and descriptors

Using SIFT to detect key points and descriptors

Using SURF to detect key points and descriptors

Using BRISK and hamming distance to detect key points and descriptors

Feature matching

As we can see, both images have a large number of feature points. Now, we want to compare the two sets of features and show as many pairs of similar feature points as possible. With OpenCV, feature point matching requires a Matcher object. Here, we explore two ways: violent matcher (BruteForce) and KNN (k nearest neighbor).

The role of BruteForce (BF) Matcher is just as it should be. Two sets of features (from image An and image B) are given, and each feature of group An is compared with all the features of group B. By default, BF Matcher calculates the Euclidean distance between two points. Therefore, for each feature in set A, it returns the closest feature in set B. Euclidean distance is recommended for SIFT and SURF,OpenCV. For other feature extractors such as ORB and BRISK, hamming distance is recommended. We are going to use OpenCV to create a BruteForce Matcher. In general, we only need to specify two parameters. The first is the distance measurement. The second is whether to cross-check the Boolean parameter. The specific code is as follows:

Def createMatcher (method,crossCheck): "Create and return a Matcher Object"

If method = = 'sift' or method = =' surf': bf = cv2.BFMatcher (cv2.NORM_L2, crossCheck=crossCheck) elif method = = 'orb' or method = =' brisk': bf = cv2.BFMatcher (cv2.NORM_HAMMING, crossCheck=crossCheck) return bf

Cross-checking Boolean parameters to indicate whether the two features match each other is considered valid. In other words, for a pair of features that are considered to be valid (F1 and f2), F1 needs to match f2 and must match F1 as the closest match. This process ensures a more powerful matching feature set, as described in the original SIFT paper.

However, in cases where multiple candidate matches are to be considered, a KNN-based matching process can be used. KNN does not return a single best match for a given feature, but instead returns k best matches. It is important to note that the value of k must be predefined by the user. As we expected, KNN offers more candidate features. However, before going any further, we need to make sure that all of these matching pairs are robust.

Ratio test

In order to ensure that the features returned by KNN are comparable, the authors of the SIFT paper proposed a technique called ratio testing. In general, we iterate through KNN to get a match, and then perform a distance test. For each pair of features (F1 and f2), if the distance between F1 and f2 is within a certain proportion, it is retained, otherwise it will be discarded. Similarly, the ratio value must be selected manually.

In essence, the ratio test has the same effect as the cross-check option of BruteForce Matcher. Both ensure that a pair of detected features are indeed close enough to be considered similar. The following two figures show the matching results of BF and KNN Matcher on the SIFT feature. We chose to display only 100 matching points for clear display.

Functional matching using quantitative tests of KNN and SIFT

Feature matching using brute force matcher on SIFT features

It should be noted that even if a variety of filtering is done to ensure the correctness of the matching, it can not completely guarantee the correct matching of the feature points. Nevertheless, the Matcher algorithm will provide us with the best (more similar) feature set of the two images. Next, we use these points to calculate the transformation matrix that splices the matching points of the two images together.

This transformation is called a homography matrix. In short, homography is a 3x3 matrix that can be used in many applications, such as camera attitude estimation, perspective correction and image stitching. It maps points from one plane (image) to another.

Estimated homography

Random sampling consistency (RANSAC) is an iterative algorithm for fitting linear models. Different from other linear regression machines, RANSAC is designed to be robust to outliers.

Models such as linear regression use least square estimates to fit the best model to data. However, the ordinary least square method is very sensitive to abnormal values. If the number of outliers is large, they may fail. RANSAC solves this problem by using only one set of data to estimate parameters. The following figure shows the comparison between linear regression and RANSAC. It is important to note that the dataset contains a considerable number of outliers.

We can see that the linear regression model is easily affected by outliers. That's because it tries to reduce the average error. Therefore, it tends to support models that minimize the total distance between all data points and the model itself. Including outliers. In contrast, RANSAC fits the model only as a subset of points that are identified as points.

This feature is very important to our use cases. Here, we will use RANSAC to estimate the homography matrix. It turns out that the homography matrix is very sensitive to the quality of the data we pass to it. Therefore, it is important to have an algorithm (RANSAC) that can filter out points that clearly belong to data distribution from points that do not belong to data distribution.

After estimating the homography matrix, we need to transform one of the images to a common plane. Here, we will apply a perspective transformation to one of the images. Perspective transformations can combine one or more operations, such as rotating, zooming, panning, or cutting. We can use the OpenCV warpPerspective () function. It takes the image and the homography matrix as input.

# Apply panorama correctionwidth = trainImg.shape [1] + queryImg.shape [1] height = trainImg.shape [0] + queryImg.shape [0]

Result = cv2.warpPerspective (trainImg, H, (width, height)) result [0:queryImg.shape [0], 0:queryImg.shape [1]] = queryImg

Plt.figure (figsize= (20510)) plt.imshow (result)

Plt.axis ('off') plt.show ()

The resulting panoramic image is shown below. As we can see, the result contains the contents of the two images. In addition, we can see some problems related to lighting conditions and image boundary edge effect. Ideally, we can implement some processing techniques to standardize brightness, such as histogram matching, which will make the results look more realistic and natural.

Thank you for your reading, the above is the content of "how to use OpenCV for image panoramic stitching". After the study of this article, I believe you have a deeper understanding of how to use OpenCV for image panoramic stitching, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.