Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "how to achieve OpenCV-based gesture detection in Python machine vision". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

1 introduction

Today, the senior introduces you to a machine vision project.

Gesture Detection and recognition algorithm based on Machine Vision opencv

2 gesture detection based on traditional machine vision

The basic process of ordinary machine vision gesture detection is as follows:

Among them, the extraction of contour, the solution of polygon fitting curve, the solution of convex hull set and depression set are all adopted the function of opencv. The recognition of gesture numbers makes use of the geometric relationship between the convex point and the sunken point and the central point of the hand, and makes a simple logical discrimination (it is certain that this method is very bad). The specific method is to locate the coordinates of two central points in the hand, and the distance threshold between the coordinates of the two central points is set by the program, and one of the central points is the position of the hand tracked by OpenNI. With the coordinates of these two central points, the number of convex and concave points on the coordinates of these two central points can be calculated in the program. Of course, this is done on the premise that people gesture numbers while pointing their fingers up (because they are not trained through samples like machine learning, so the conditions are much harsher). The number 05can be identified by using the number of four kinds of points obtained above (in addition, the number of two auxiliary calculation points is set in the program, see the code part for details) and simple logic judgment. Other numbers can be designed according to specific logic (and multi-digit recognition can also be designed), but the more numbers, the more complex the design, because the interference between them should be taken into account. and this non-universal design method does not have much practical significance.

2.1 Contour detection

Use the void convexityDefects (InputArray contour, InputArray convexhull, OutputArray convexityDefects) method

The function is to detect the convex defects of the input contour contour and the convex hull set. A convex defect structure includes four elements, the starting point coordinates of the defect, the end point coordinates of the defect, the coordinates of the point farthest from the convex hull line in the defect, and the furthest distance at this time. Parameter 3 is the convex defect structure vector of its output.

The schematic diagram of the convex defect is as follows:

Although the first parameter is contour, which literally means outline, I have experimented many times and found that if the parameter is the original contour obtained by contour detection, the program will report a memory error when it runs to the onvexityDefects () function. Therefore, what is used in this program is not the original outline of the object, but the profile fitted by the polynomial curve, that is, the polynomial curve, so that the program will run smoothly. In addition, because the outline detected in a certain frame may be so small (for some reason) that there is only one point in the gesture recognition process, the following error will be reported if the program runs to the onvexityDefects () function:

Int Mat::checkVector (int _ elemChannels, int _ depth, bool _ requireContinuous) const {return (depth ()) = = _ depth | | _ depth 3. To put it simply, when using the convexityDefects () function to detect the depression of a polygonal curve, parameter 1 must be required to have at least two points in the curve itself (I don't know whether this analysis is correct or not). Therefore, I add if (Mat (approx_poly_curve). CheckVector (2, CV_32S) > 3) before the convexityDefects () function of this program to judge that only if the if condition is satisfied will the later sag detection be carried out. In this way, similar bug will not appear in the program.

The second parameter is generally obtained by the function convexHull () in opencv. In general, what is stored in this parameter is the position index of the points in the convex hull set in the polynomial curve points, and the parameter exists in the form of vector, so the type of the element in the parameter convexhull is unsigned int. In this sag detection function convexityDefects (), according to the document, the parameter is required to be Matt. Therefore, when using the parameter 2 of convexityDefects (), it is common to convert vector directly to Matt.

Parameter 3 is a collection of structures with four elements. If in the C++ version, this parameter can be directly replaced by vector. The four elements in Vec4i represent the start coordinate index of the sunken curve segment, the end coordinate index, the coordinate index of the farthest point from the convex hull set curve and the furthest distance values at this time, all of which are integers. In the c version of opencv, it is generally not the saved index, but the coordinate value.

2.2 results of the algorithm

The recognition result of the number "0":

The recognition result of the number "1"

The recognition result of the number "2"

The recognition result of the number "3":

The recognition result of the number "4":

The recognition result of the number "5":

2.3 overall code implementation

2.3.1 algorithm flow

The implementation process of the upperclassman is similar to the system flow chart above, and the process is roughly as follows:

1. Find out the mask of the hand

two。 Figure out the outline of the mask

3. The multi-deformation fitting curve of the profile is obtained.

4. Find out the convex hull set of polygon fitting curve and find out the convex point.

5. Find out the concave set of multi-deformation fitting curve and find out the concave point.

6. Use the geometric relationship between the convex and concave points above and the center point of the hand to do simple digital gesture recognition.

(here is written in C language, this code is written by seniors in the early stage, students need to learn to grow a python version.)

# include # include "opencv2/highgui/highgui.hpp" # include "opencv2/imgproc/imgproc.hpp" # include # include "copenni.cpp" # include # define DEPTH_SCALE_FACTOR 255./4096.#define ROI_HAND_WIDTH 140#define ROI_HAND_HEIGHT 140#define MEDIAN_BLUR_K 5#define XRES 640#define YRES 480#define DEPTH_SEGMENT_THRESH 5#define MAX_HANDS_COLOR 10#define MAX_HANDS_NUMBER 10#define HAND_LIKELY_AREA 2000#define DELTA_POINT_DISTENCE 25 / / threshold of distance between hand center point 1 and center point 2 # define SEGMENT_POINT1_DISTANCE 27 / / threshold of distance between bump and hand center point 1 # define SEGMENT_POINT2_DISTANCE 30 / / threshold using namespace cv of distance between bump and hand center point 2 Using namespace xn;using namespace std;int main (int argc, char * * argv) {unsigned int convex_number_above_point1 = 0; unsigned int concave_number_above_point1 = 0; unsigned int convex_number_above_point2 = 0; unsigned int concave_number_above_point2 = 0; unsigned int convex_assist_above_point1 = 0; unsigned int convex_assist_above_point2 = 0; unsigned int point_y1 = 0; unsigned int point_y2 = 0 Int number_result =-1; bool recognition_flag = false; / / start hand digit recognition logo vector color_array;// uses the default 10 colors {color_array.push_back (Scalar (255,0,0); color_array.push_back (Scalar (0,255,0)); color_array.push_back (Scalar (0,0,255)) Color_array.push_back (Scalar (255,0,255)); color_array.push_back (Scalar (255,255,0)); color_array.push_back (Scalar (0,255,255)); color_array.push_back (Scalar (128,255,0)); color_array.push_back (Scalar (0128,255)); color_array.push_back (Scalar (255,0128)) Color_array.push_back (Scalar (255,128,255);} vector hand_depth (MAX_HANDS_NUMBER, 0); vector hands_roi (MAX_HANDS_NUMBER, Rect (XRES/2, YRES/2, ROI_HAND_WIDTH, ROI_HAND_HEIGHT)); namedWindow ("color image", CV_WINDOW_AUTOSIZE); namedWindow ("depth image", CV_WINDOW_AUTOSIZE); namedWindow ("hand_segment", CV_WINDOW_AUTOSIZE) / / display the area of the segmented hand namedWindow ("handrecognition", CV_WINDOW_AUTOSIZE); / / display the digital recognition image COpenNI openni; if (! openni.Initial ()) return 1; if (! openni.Start ()) return 1; while (1) {if (! openni.UpdateData ()) {return 1 } / * acquire and display color images * / Mat color_image_src (openni.image_metadata_.YRes (), openni.image_metadata_.XRes (), CV_8UC3, (char *) openni.image_metadata_.Data ()); Mat color_image; cvtColor (color_image_src, color_image, CV_RGB2BGR) Mat hand_segment_mask (color_image.size (), CV_8UC1, Scalar::all (0)); for (auto itUser = openni.hand_points_.cbegin (); itUser! = openni.hand_points_.cend (); + + itUser) {point_y1 = itUser- > second.Y; point_y2 = itUser- > second.Y + DELTA_POINT_DISTENCE Circle (color_image, Point (itUser- > second.X, itUser- > second.Y), 5, color_array.at (itUser- > first% color_array.size ()), 3,8); / * set the depth of different hands * / hand_depth.at (itUser- > first% MAX_HANDS_COLOR) = (unsigned int) (itUser- > second.Z* DEPTH_SCALE_FACTOR) / / itUser- > first will cause the program to appear bug / * set different regions of interest in different hands * / hands_roi.at (itUser- > first% MAX_HANDS_NUMBER) = Rect (itUser- > second.X-ROI_HAND_WIDTH/2, itUser- > second.Y-ROI_HAND_HEIGHT/2, ROI_HAND_WIDTH, ROI_HAND_HEIGHT) Hands_roi.at (itUser- > first% MAX_HANDS_NUMBER). X = itUser- > second.X-ROI_HAND_WIDTH/2; hands_roi.at (itUser- > first% MAX_HANDS_NUMBER). Y = itUser- > second.Y-ROI_HAND_HEIGHT/2; hands_roi.at (itUser- > first% MAX_HANDS_NUMBER). Width = ROI_HAND_WIDTH Hands_roi.at (itUser- > first% MAX_HANDS_NUMBER). Height = ROI_HAND_HEIGHT; if (hands_roi.at (itUser- > first% MAX_HANDS_NUMBER). X first% MAX_HANDS_NUMBER). X = 0; if (hands_roi.at (itUser- > first% MAX_HANDS_NUMBER). X > XRES) hands_roi.at (itUser- > first% MAX_HANDS_NUMBER). X = XRES If (hands_roi.at (itUser- > first% MAX_HANDS_NUMBER). Y first% MAX_HANDS_NUMBER). Y = 0; if (hands_roi.at (itUser- > first% MAX_HANDS_NUMBER). Y > YRES) hands_roi.at (itUser- > first% MAX_HANDS_NUMBER). Y = YRES;} imshow ("color image", color_image) / * acquire and display depth images * / Mat depth_image_src (openni.depth_metadata_.YRes (), openni.depth_metadata_.XRes (), CV_16UC1, (char *) openni.depth_metadata_.Data ()); / / because the depth images obtained by kinect are actually unsigned 16-bit data Mat depth_image Depth_image_src.convertTo (depth_image, CV_8U, DEPTH_SCALE_FACTOR); imshow ("depth image", depth_image); / / remove the mask part of the hand / / regardless of the number of channels in the original image, the mask matrix is declared as single-channel ok for (auto itUser = openni.hand_points_.cbegin (); itUser! = openni.hand_points_.cend ()) + + itUser) {for (int I = hands_roi.at (itUser- > first% MAX_HANDS_NUMBER) .x; I

< std::min(hands_roi.at(itUser->

First% MAX_HANDS_NUMBER) .x+hands_roi.at (itUser- > first% MAX_HANDS_NUMBER) .width, XRES) for (int j = hands_roi.at (itUser- > first% MAX_HANDS_NUMBER) .y; j

< std::min(hands_roi.at(itUser->

First% MAX_HANDS_NUMBER) .y+hands_roi.at (itUser- > first% MAX_HANDS_NUMBER) .height, YRES); hand_segment_mask.at +) {hand_segment_mask.at (j, I) = (hand_depth.at (itUser- > first% MAX_HANDS_NUMBER)-DEPTH_SEGMENT_THRESH)

< depth_image.at(j, i)) & ((hand_depth.at(itUser->

First% MAX_HANDS_NUMBER) + DEPTH_SEGMENT_THRESH) > depth_image.at (jjinger I);}} medianBlur (hand_segment_mask, hand_segment_mask, MEDIAN_BLUR_K); Mat hand_segment (color_image.size (), CV_8UC3); color_image.copyTo (hand_segment, hand_segment_mask) / * extract the outline of the mask image and draw it in the gesture recognition image * / std::vector

< std::vector >

Contours; findContours (hand_segment_mask, contours, CV_RETR_LIST, CV_CHAIN_APPROX_SIMPLE); / / find the outline of the mask image Mat hand_recognition_image = Mat::zeros (color_image.rows, color_image.cols, CV_8UC3); for (int I = 0; I

< contours.size(); i++) { //只有在检测到轮廓时才会去求它的多边形，凸包集，凹陷集 recognition_flag = true; /*找出轮廓图像多边形拟合曲线*/ Mat contour_mat = Mat(contours[i]); if(contourArea(contour_mat) >

HAND_LIKELY_AREA) {/ / compare areas that are likely to look like hands std::vector approx_poly_curve; approxPolyDP (contour_mat, approx_poly_curve, 10, true); / / find out the polygon fitting curve std::vector of the outline

< std::vector >

Approx_poly_curve_debug; approx_poly_curve_debug.push_back (approx_poly_curve); drawContours (hand_recognition_image, contours, I, Scalar (255,0,0), 1,8); / / outline / / drawContours (hand_recognition_image, approx_poly_curve_debug, 0, Scalar (256,128,128), 1,8) / draw the polygon fitting curve / * find the convex hull set of the polygon fitting curve * / vector hull; convexHull (Mat (approx_poly_curve), hull, true); for (int I = 0; I)

< hull.size(); i++) { circle(hand_recognition_image, approx_poly_curve[hull[i]], 2, Scalar(0, 255, 0), 2, 8); /*统计在中心点1以上凸点的个数*/ if(approx_poly_curve[hull[i]].y SEGMENT_POINT1_DISTANCE && dis1 >

= 0) {convex_assist_above_point1++;} convex_number_above_point1++ } / * count the number of convex points above center point 2 * / if (approx_poly_ curve [hull [I]] .y SEGMENT_POINT2_DISTANCE & & dis2 > = 0) {convex_assist_above_point2++;} convex_number_above_point2++ }} / * find the sag set * / std::vector convexity_defects; if (Mat (approx_poly_curve) .checkvector (2, CV_32S) > 3) convexityDefects (approx_poly_curve, Mat (hull), convexity_defects) for the polygon fitting curve. For (int I = 0; I

< convexity_defects.size(); i++) { circle(hand_recognition_image, approx_poly_curve[convexity_defects[i][2]] , 2, Scalar(0, 0, 255), 2, 8); /*统计在中心点1以上凹陷点的个数*/ if(approx_poly_curve[convexity_defects[i][2]].y second.Y), 3, Scalar(0, 255, 255), 3, 8); circle(hand_recognition_image, Point(itUser->

Second.X, itUser- > second.Y + 25), 3, Scalar (255,0,255), 3,8);} / * recognition of gesture digits 0,5 * / / recognition of "0" if ((convex_assist_above_point1 = = 0 & & convex_number_above_point2 > = 2 & & convex_number_above_point2 = 4) number_result = 5 If (number_result! = 0 & & number_result! = 1 & & number_result! = 2 & & number_result! = 3 & & number_result! = 4 & & number_result! = 5) number_result = =-1; / * display the matching number on the gesture recognition graph * / std::stringstream number_str; number_str

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.