Shulou(Shulou.com)06/01 Report--

This article introduces the knowledge of "how to use OpenCV to achieve gesture recognition in C++". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

First, hand key point detection

As shown in the figure, it is the location of the key points of our hands. As a first step, we need to detect 21 key points on the hand. We use the deep neural network DNN module to do this. By using the DNN module, 21 key points of the hand can be detected as the result output, please see the source code for details.

1.1 function source code / / hand key point detection bool HandKeypoints_Detect (Mat src, vector&HandKeypoints) {/ / Model size int width = src.cols; int height = src.rows; float ratio = width / (float) height; int modelHeight = 368; / / int modelWidth = int (ratio*modelHeight) determined by the model input dimension / / Model file string model_file = "pose_deploy.prototxt"; / / Network model string model_weight = "pose_iter_102000.caffemodel"; / / Network training weight / / load caffe model Net net = readNetFromCaffe (model_file, model_weight) / / convert the input image into blob form Mat blob = blobFromImage (src, 1.0,255,Size (modelWidth, modelHeight), Scalar (0,0,0)); / / input the image converted blob data into the first layer "image" layer of the network, see the deploy.protxt file net.setInput (blob, "image"); / / result output Mat output = net.forward () Int H = output.size [2]; int W = output.size [3]; for (int I = 0; I

< nPoints; i++) { //结果预测 Mat probMap(H, W, CV_32F, output.ptr(0, i)); resize(probMap, probMap, Size(width, height)); Point keypoint; //最大可能性手部关键点位置 double classProb; //最大可能性概率值 minMaxLoc(probMap, NULL, &classProb, NULL, &keypoint); HandKeypoints[i] = keypoint; //结果输出，即手部关键点所在坐标 } return true;}1.2 功能效果

As shown in the figure, we have detected the location of 21 hand key points through DNN. Next, we need to use these key points for simple gesture recognition.

Second, the principle of 2.1 algorithm for gesture recognition

In this case, the realization of gesture recognition is determined by comparing the positions of key points. First take out the index of each fingertip key point (that is, 4, 8, 12, 16, 20). Next, compare the other key points of each finger with the position of its fingertip.

For example, we want to determine whether the thumb is open or closed. As shown in the following figure, because OpenCV establishes the coordinate system with the upper left corner as the starting point. When the thumb is open (palm inward), we can find that we can compare the position of key point 4 and key point 3. When the x coordinate of 4 is greater than the x coordinate of 3, the thumb is open; when the x coordinate of 4 is less than the x coordinate of 3, the thumb is closed.

Similarly, the other four fingers, take the index finger as an example. When the y coordinate of key point 8 is less than the y coordinate of key point 6, the index finger is open; when the y coordinate of key point 8 is greater than the y coordinate of key point 6, the index finger is closed.

When the fingers are open, we count 1. The purpose of gesture recognition is achieved by counting the opening number of fingers. Please look at the source code for details.

2.2 function source code / / gesture recognition bool Handpose_Recognition (vector&HandKeypoints, int& count) {vectorfingers; / / thumb if (HandKeypoints [tipIds [0]] .x > HandKeypoints [tipIds [0]-1] .x) {/ / if the x coordinate of the key point'4' is greater than the x coordinate of the key point'3', then the thumb is open. Count 1 fingers.push_back (1);} else {fingers.push_back (0);} / / the remaining 4 fingers for (int I = 1; I

< 5; i++) { if (HandKeypoints[tipIds[i]].y < HandKeypoints[tipIds[i] - 2].y) { //例：如果关键点'8'的y坐标小于关键点'6'的y坐标，则说明食指是张开的。计数1 fingers.push_back(1); } else { fingers.push_back(0); } } //结果统计 for (int i = 0; i < fingers.size(); i++) { if (fingers[i] == 1) { count++; } } return true;}三、结果显示 通过以上步骤，我们已经有了手部关键点所在坐标位置以及对应的手势结果，接下来就进行效果展示。 在这里，为了逼格高一点，我们将下面的手势模板图像作为输出结果放进我们的测试图中。具体操作请看源码。 3.1功能源码//识别效果显示bool ShowResult(Mat& src, vector&HandKeypoints, int& count){ //画出关键点所在位置 for (int i = 0; i < nPoints; i++) { circle(src, HandKeypoints[i], 3, Scalar(0, 0, 255), -1); putText(src, to_string(i), HandKeypoints[i], FONT_HERSHEY_COMPLEX, 0.8, Scalar(0, 255, 0), 2); } //为了显示骚操作，读取模板图片，作为识别结果 vectorimageList; string filename = "images/"; glob(filename, imageList); vectorTemp; for (int i = 0; i < imageList.size(); i++) { Mat temp = imread(imageList[i]); resize(temp, temp, Size(100, 100), 1, 1, INTER_AREA); Temp.push_back(temp); } //将识别结果显示在原图中 Temp[count].copyTo(src(Rect(0, src.rows- Temp[count].rows, Temp[count].cols, Temp[count].rows))); putText(src, to_string(count), Point(20, 60), FONT_HERSHEY_COMPLEX, 2, Scalar(0, 0, 128), 3); return true;}3.2效果显示 除此之外，我们还可以将所有的图片整合成一张图，具体请看源码吧。 //将所有图片整合成一张图片bool Stitching_Image(vectorimages){ Mat canvas = Mat::zeros(Size(1200, 1000), CV_8UC3); int width = 400; int height = 500; for (int i = 0; i < images.size(); i++) { resize(images[i], images[i], Size(width, height), 1, 1, INTER_LINEAR); } int col = canvas.cols / width; int row = canvas.rows / height; for (int i = 0; i < row; i++) { for (int j = 0; j < col; j++) { int index = i * col + j; images[index].copyTo(canvas(Rect(j*width, i*height, width, height))); } } namedWindow("result", WINDOW_NORMAL); imshow("result", canvas); waitKey(0); return true;} 最终结果如图所示。以上就是整个案例的流程啦。。。 四、源码#include#include#includeusing namespace std;using namespace cv;using namespace cv::dnn;//手部关键点数目const int nPoints = 21; //手指索引const int tipIds[] = { 4,8,12,16,20 };//手部关键点检测bool HandKeypoints_Detect(Mat src, vector&HandKeypoints){ //模型尺寸大小 int width = src.cols; int height = src.rows; float ratio = width / (float)height; int modelHeight = 368; //由模型输入维度决定 int modelWidth = int(ratio*modelHeight); //模型文件 string model_file = "pose_deploy.prototxt"; //网络模型 string model_weight = "pose_iter_102000.caffemodel";//网络训练权重 //加载caffe模型 Net net = readNetFromCaffe(model_file, model_weight); //将输入图像转成blob形式 Mat blob = blobFromImage(src, 1.0 / 255, Size(modelWidth, modelHeight), Scalar(0, 0, 0)); //将图像转换的blob数据输入到网络的第一层"image"层，见deploy.protxt文件 net.setInput(blob, "image"); //结果输出 Mat output = net.forward(); int H = output.size[2]; int W = output.size[3]; for (int i = 0; i < nPoints; i++) { //结果预测 Mat probMap(H, W, CV_32F, output.ptr(0, i)); resize(probMap, probMap, Size(width, height)); Point keypoint; //最大可能性手部关键点位置 double classProb; //最大可能性概率值 minMaxLoc(probMap, NULL, &classProb, NULL, &keypoint); HandKeypoints[i] = keypoint; //结果输出，即手部关键点所在坐标 } return true;}//手势识别bool Handpose_Recognition(vector&HandKeypoints, int& count){ vectorfingers; //拇指 if (HandKeypoints[tipIds[0]].x >

HandKeypoints [tipIds [0]-1] .x) {/ / if the x coordinate of key'4' is greater than the x coordinate of key'3', the thumb is open. Count 1 fingers.push_back (1);} else {fingers.push_back (0);} / / the remaining 4 fingers for (int I = 1; I < 5) ) {if (HandKeypoints [tipIds [I]]. Y < HandKeypoints [tipIds [I]-2] .y) {/ example: if the y coordinate of the key point'8' is less than the y coordinate of the key point'6', the index finger is open. Count 1 fingers.push_back (1);} else {fingers.push_back (0);}} / / result Statistics for (int I = 0; I < fingers.size () ) {if (fingers [I] = = 1) {count++;}} return true;} / / the recognition effect shows that bool ShowResult (Mat& src, vector&HandKeypoints, int& count) {/ / draw the location of the key point for (int I = 0; I < nPoints) Circle (src, HandKeypoints [I], 3, Scalar (0,0,255),-1); putText (src, to_string (I), HandKeypoints [I], FONT_HERSHEY_COMPLEX, 0.8, Scalar (0,255,0), 2);} / / to display the coquettish operation, read the template image as the recognition result vectorimageList String filename = "images/"; glob (filename, imageList); vectorTemp; for (int I = 0; I < imageList.size (); iTunes +) {Mat temp = imread (imageList [I]); resize (temp, temp, Size (100,100), 1,1, INTER_AREA) Temp.push_back (temp);} / / display the recognition results in the original map (src (Rect (0, src.rows- Tempcount) .cols, temp [count] .rows); putText (src, to_string (count), Point (20,60), FONT_HERSHEY_COMPLEX, 2, Scalar (0,0,128), 3) Return true;} / / integrate all the pictures into one picture bool Stitching_Image (vectorimages) {Mat canvas = Mat::zeros (Size (1200, 1000), CV_8UC3); int width = 400; int height = 500; for (int I = 0; I < images.size () ) {resize (images [I], images [I], Size (width, height), 1,1, INTER_LINEAR);} int col = canvas.cols / width; int row = canvas.rows / height; for (int I = 0; I < row; iTunes +) {for (int j = 0; j < col) Canvas (Rect (j*width, i*height, width, height));}} namedWindow ("result", WINDOW_NORMAL); imshow ("result", canvas); waitKey (0); return true } int main () {vectorimageList; string filename = "test/"; glob (filename, imageList); vectorimages; for (int I = 0; I < imageList.size (); iTunes +) {Mat src = imread (imageList [I]); vectorHandKeypoints (nPoints); HandKeypoints_Detect (src, HandKeypoints); int count = 0 Handpose_Recognition (HandKeypoints, count); ShowResult (src, HandKeypoints, count); images.push_back (src); imshow ("Demo", src); waitKey (0);} Stitching_Image (images); system ("pause"); return 0 } this is the end of the content of "how to use OpenCV to achieve gesture recognition in C++". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.