Shulou(Shulou.com)05/31 Report--

How to easily identify CAPTCHA through Serverless, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain for you in detail, people with this need can come to learn, I hope you can gain something.

Preface

The concept of Serverless has attracted much attention since it was put forward, especially in recent years, Serverless has shown unprecedented vitality. Engineers in various fields are trying to combine Serverless architecture with their own work in order to obtain the "technical dividend" brought by Serverless architecture.

CAPTCHA, an abbreviation for "Completely Automated Public Turing test to tell Computers and Humans Apart" (Turing test that automatically distinguishes computers from humans), is a fully automated public program that distinguishes whether a user is a computer or a human. It can prevent malicious password cracking, ticket brushing and forum flooding, and effectively prevent a hacker from constantly trying to log in to a specific registered user by means of a specific program. In fact, CAPTCHA is now a popular way for many websites, we use a relatively simple way to achieve this function. The problem of CAPTCHA is generated and judged by the computer, but this question can only be answered by human beings, and the computer cannot answer it, so the user who answers the question can be regarded as human. To put it bluntly, the CAPTCHA is the code used to verify whether it is accessed by a person or a machine.

So what sparks will there be between CAPTCHA recognition and Serverless architecture in the field of artificial intelligence? In this paper, the function of CAPTCHA recognition is realized through Serverless architecture and convolutional neural network (CNN) algorithm.

Talking about the verification code

The development of CAPTCHA can be said to be very rapid, from the initial simple numeric CAPTCHA, to the later numeric + alphabetic CAPTCHA, and then to the later numeric + alphabetic + Chinese CAPTCHA and graphic and image CAPTCHA. The material of simple CAPTCHA has become more and more. From the form of CAPTCHA, it is also different, input, click, drag and drop, SMS CAPTCHA, voice CAPTCHA.

Bilibili's login verification code includes a variety of modes, such as sliding the slider to verify:

For example, verify by clicking the text in turn:

On the other hand, the verification codes of related websites such as Baidu Tieba, Zhihu and Google are different, such as selecting the text being written, selecting the picture including the specified object, and clicking on the characters in the picture in order.

The identification of CAPTCHA may not be consistent according to the type of CAPTCHA. Of course, the simplest CAPTCHA may be the original text CAPTCHA:

Even for text CAPTCHA, there are many differences, such as simple numeric CAPTCHA, simple numeric + alphabetic CAPTCHA, text CAPTCHA, including calculation in CAPTCHA, some interference in simple CAPTCHA to become complex CAPTCHA, and so on.

CAPTCHA recognition

1. Simple CAPTCHA recognition

CAPTCHA recognition is an ancient research field, which is simply the process of converting the text on the picture into text. In recent years, with the development of big data, the majority of crawler engineers have higher and higher requirements for CAPTCHA recognition when fighting anti-crawling strategies. In the era of simple CAPTCHA, the recognition of CAPTCHA is mainly aimed at the text CAPTCHA, cutting each part of the CAPTCHA through image cutting, and then comparing the similarity of each clipping unit to get the most possible result. Finally, it is spliced, such as CAPTCHA:

Perform operations such as binarization:

Do not cut until you are finished:

After the cutting is completed, it is recognized and finally spliced, so that it is relatively easy to recognize each character.

However, with the development of time, when this simple verification code is gradually unable to meet the question of whether it is a human or a machine, the verification code has been upgraded a little, that is, some interference lines have been added to the verification code, or the verification code has been seriously distorted, resulting in strong color block interference, such as the verification code of the Dynadot website:

There are not only image distortion and overlap, but also interference lines and color blocks. At this time, if you want to identify the CAPTCHA, it is difficult to get good results by simple cutting recognition. At this time, good results can be achieved through in-depth learning.

two。 CAPTCHA recognition based on CNN

Convolution neural network (Convolutional Neural Network, referred to as CNN) is a feedforward neural network, artificial neurons can respond to the surrounding units for large-scale image processing. Convolution neural network includes convolution layer and pooling layer.

As shown in the figure, the left picture is the traditional neural network, its basic structure is: input layer, hidden layer, output layer. The figure on the right is convolution neural network, whose structure is composed of input layer, output layer, convolution layer, pooling layer and full connection layer. Convolution neural network is actually an extension of neural network, but in fact, in terms of structure, there is no difference between simple CNN and simple NN (of course, the introduction of special structure and complex CNN will be quite different from NN). Compared with the traditional neural network, CNN greatly reduces the number of network parameters in the actual effect, so that we can train a better model with fewer parameters, typically get twice the result with half the effort, and can effectively avoid over-fitting. Similarly, because of the parameter sharing of filter, even if the image has a certain translation operation, we can still recognize the feature, which is called "translation invariance". As a result, the model is more robust.

1) CAPTCHA generation

The generation of CAPTCHA is a very important step, because this part of CAPTCHA will be used as our training set and test set, and what type of CAPTCHA can be identified by our model is also related to this part.

# coding:utf-8 import random import numpy as np from PIL import Image from captcha.image import ImageCaptcha CAPTCHA_LIST = [eve for eve in "0123456789abcdefghijklmnopqrsruvwxyzABCDEFGHIJKLMOPQRSTUVWXYZ"] CAPTCHA_LEN = 4 # CAPTCHA CAPTCHA_HEIGHT = 60 # CAPTCHA height CAPTCHA_WIDTH = 160 # CAPTCHA width randomCaptchaText = lambda char=CAPTCHA_LIST, size=CAPTCHA_LEN: ".join ([random.choice (char) for _ in range (size)]) def genCaptchaTextImage (width=CAPTCHA_WIDTH, height=CAPTCHA_HEIGHT, save=None): image = ImageCaptcha (width=width Height=height) captchaText = randomCaptchaText () if save: image.write (captchaText,'. / img/%s.jpg'% captchaText) return captchaText, np.array (Image.open (image.generate (captchaText) print (genCaptchaTextImage (save=True))

Through the above code, you can generate simple Chinese and English verification codes:

2) Model training

The code for the model training is as follows (part of the code comes from the network).

Util.py files, mainly some extracted public methods:

#-*-coding:utf-8-*-import numpy as np from captcha_gen import genCaptchaTextImage from captcha_gen import CAPTCHA_LIST, CAPTCHA_LEN, CAPTCHA_HEIGHT, CAPTCHA_WIDTH # pictures are converted to black and white 3-D to 1-D convert2Gray = lambda img: np.mean (img,-1) if len (img.shape) > 2 else img # CAPTCHA vector is converted to the text vec2Text = lambda vec, captcha_list=CAPTCHA_LIST: '.join ([captcha_ list [captcha_] for v in vec]) def text2Vec (text, captchaLen=CAPTCHA_LEN) CaptchaList=CAPTCHA_LIST): "vector = np.zeros (captchaLen * len (captchaList)) for i in range (len (text)): vector [captchaList.index (text [I]) + I * len (captchaList)] = 1 return vector def getNextBatch (batchCount=60, width=CAPTCHA_WIDTH) Height=CAPTCHA_HEIGHT): "" get training picture group "batchX = np.zeros ([batchCount, width * height]) batchY = np.zeros ([batchCount, CAPTCHA_LEN * len (CAPTCHA_LIST)]) for i in range (batchCount): text Image = genCaptchaTextImage () image = convert2Gray (image) # make the array of pictures one-dimensional and correspond the text to the same line of the two two-dimensional groups batchX [I,] = image.flatten () / 255 batchY [I,:] = text2Vec (text) return batchX, batchY # print (getNextBatch (batch_count=1))

Model_train.py file, mainly for model training. In this file, the basic information of the model is defined, for example, the model is a three-layer convolution neural network, the original image size is 60 million 160, after the first convolution, it becomes 60 million 160, and after the first pool, it becomes 30 80. After the second convolution, it becomes 15: 40; the third convolution becomes 15: 40, and the third pool becomes 7: 20. After three times of convolution and pooling, the original image data is changed into flat data of 7: 20. At the same time, during the training, the project carries out data tests every 100 times to calculate the accuracy:

#-*-coding:utf-8-*-import tensorflow.compat.v1 as tf from datetime import datetime from util import getNextBatch from captcha_gen import CAPTCHA_HEIGHT, CAPTCHA_WIDTH, CAPTCHA_LEN, CAPTCHA_LIST tf.compat.v1.disable_eager_execution () variable = lambda shape, alpha=0.01: tf.Variable (alpha * tf.random_normal (shape)) conv2d = lambda x, w: tf.nn.conv2d (x, w, strides= [1,1,1,1] Padding='SAME') maxPool2x2 = lambda x: tf.nn.max_pool (x, ksize= [1,2,2,1], strides= [1,2,2,1], padding='SAME') optimizeGraph = lambda y, y_conv: tf.train.AdamOptimizer (1e-3). Minimize (tf.reduce_mean (tf.nn.sigmoid_cross_entropy_with_logits (labels=y, logits=y_conv)) hDrop = lambda image, weight, bias, keepProb: tf.nn.dropout (tf.nn.relu (conv2d (image)) Variable (weight, 0. 01) + variable (bias, 0. 1), keepProb) def cnnGraph (x, keepProb, size, captchaList=CAPTCHA_LIST, captchaLen=CAPTCHA_LEN): "imageHeight, imageWidth = size xImage = tf.reshape (x, shape= [- 1, imageHeight, imageWidth, 1]) hDrop1 = hDrop (xImage, [3,3,1,32], [32], keepProb) hDrop2 = hDrop (hDrop1) [3,3,32,64], [64], keepProb) hDrop3 = hDrop (hDrop2, [3,3,64,64], [64], keepProb) # full connection layer imageHeight = int (hDrop3.shape [1]) imageWidth = int (hDrop3.shape [2]) wFc = variable ([imageHeight * imageWidth * 64,0.01]) # there are 1024 neurons in the 64 neurons in the full connection layer ([1024]) HDrop3Re = tf.reshape (hDrop3, [- 1, imageHeight * imageWidth * 64]) hFc = tf.nn.relu (tf.matmul (hDrop3Re, wFc) + bFc) hDropFc = tf.nn.dropout (hFc, keepProb) # output layer wOut = variable ([1024, len (captchaList) * captchaLen], .01) bOut = variable ([len (captchaList) * captchaLen], 0.1) yConv = tf.matmul (hDropFc WOut) + bOut return yConv def accuracyGraph (y, yConv, width=len (CAPTCHA_LIST), height=CAPTCHA_LEN): deviation calculation chart Correct value and predicted value Calculation accuracy "" maxPredictIdx = tf.argmax (tf.reshape (yConv, [- 1, height, width]), 2) maxLabelIdx = tf.argmax (tf.reshape (y, [- 1, height, width]), 2) correct = tf.equal (maxPredictIdx, maxLabelIdx) # judge whether return tf.reduce_mean (tf.cast (correct, tf.float32) def train (height=CAPTCHA_HEIGHT, width=CAPTCHA_WIDTH) is equal YSize=len (CAPTCHA_LIST) * CAPTCHA_LEN): "" cnn training "accRate = 0.95 x = tf.placeholder (tf.float32, [None, height * width]) y = tf.placeholder (tf.float32, [None, ySize]) keepProb = tf.placeholder (tf.float32) yConv = cnnGraph (x, keepProb, (height, width)) optimizer = optimizeGraph (y, yConv) accuracy = accuracyGraph (y YConv) saver = tf.train.Saver () with tf.Session () as sess: sess.run (tf.global_variables_initializer ()) # initialization step = 0 # steps while True: batchX, batchY = getNextBatch (64) sess.run (optimizer, feed_dict= {x: batchX, y: batchY KeepProb: 0.75}) # every 100th training, test if step% 100 = 0: batchXTest, batchYTest = getNextBatch (100) acc = sess.run (accuracy, feed_dict= {x: batchXTest, y: batchYTest, keepProb: 1.0}) print (datetime.now (). Strftime ('% c'), 'step:', step,' accuracy:' Acc) # accuracy meets the requirements Save model if acc > accRate: modelPath = ". / model/captcha.model" saver.save (sess, modelPath, global_step=step) accRate + = 0.01if accRate > 0.90: break step=step + 1 train ()

After completing this part, we can train the model through the local machine. In order to speed up the training, I set the accRate part of the code to:

If accRate > 0.90: break

In other words, when the accuracy exceeds 90%, the system automatically stops and saves the model.

The next step is to train:

The training time may be long. After the training is completed, you can draw according to the results and view the curve of accuracy as the Step increases:

Operation: identify the CAPTCHA

Once the code is ready, start writing the deployment file:

Global: Service: Name: ServerlessBook Description: Serverless Book case Log: Auto Nas: Auto ServerlessBookCaptchaDemo: Component: fc Provider: alibaba Access: release Extends: deploy:-Hook: s install docker Path:. / Pre: true Properties: Region: cn-beijing Service: ${Global.Service} Function: Name: serverless_captcha Description: verification Code recognition CodeUri: Src:. / src/backend Excludes:-src/backend/.fun-src/backend/model Handler: index.handler Environment:-Key: PYTHONUSERBASE Value: / mnt/auto/.fun/python MemorySize: 3072 Runtime: python3 Timeout: 60 Triggers:-Name: ImageAI Type: HTTP Parameters: AuthType: ANONYMOUS Methods:-GET-POST-PUT Domains:-Domain: Auto ServerlessBookCaptchaWebsiteDemo: bottle Provider: alibaba Access: release Extends: deploy:-Hook: pip3 install-r requirements.txt-t. / Path:. / src/website Pre: true Properties: Region: cn-beijing CodeUri:. / src/website App: index.py Environment:-Key: url Value: ${ServerlessBookCaptchaDemo.Output.Triggers [0] .domains [0]} Detail: Service: ${Global.Service} Function: Name: serverless_captcha_website

The overall directory structure:

| |-src # project directory | |-backend # project backend, core interface | |-index.py # backend core code | |-requirements.txt # backend core code dependency | |-website # project frontend | Easy to test and use | |-html # project frontend page | |-index.html # project frontend page | |-index.py # backend service (bottle framework) | |-requirements.txt # project frontend backend service dependency

After completion, we can deploy the project under the project directory:

S deploy

After the deployment is complete, open the returned page address:

Click to obtain the verification code to generate a verification code online:

Click the identification verification code at this time to identify the verification code:

Since the target accuracy of the model is 90% during training, it can be considered that the overall accuracy is about 90% after a large number of tests of the same type of CAPTCHA code.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.