Shulou(Shulou.com)05/31 Report--

In this issue, the editor will bring you about what the Serverless-based Dance Voice Tour Checker is like. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

What is Serverless Framework

Serverless Framework is a very popular serverless application framework in the industry, and developers can deploy a fully available Serverless application architecture without paying attention to the underlying resources. Serverless Framework has the capabilities of resource orchestration, automatic scaling and event-driven, covering the whole life cycle of coding, debugging, testing and deployment, helping developers to quickly build Serverless applications through the linkage of cloud resources.

Yes, just like the I'm gonna reduce your ops in the Song of Serverless seen a few days ago, it can greatly reduce the pressure of operation and maintenance, so let's do it! Note that the development environment requires Node.js 10.0. One-click global installation: npm install-g serverless

2. Introduction to Tencent Cloud Flask Serverless Component

Tencent Cloud Flask Serverless Component supports the deployment of Restful API services

According to the usual practice, deploy demo first.

Local PyCharm creates a new Flask project

Manually create a requirements.txt with Flask content

Create a serverless.yml according to the configuration document, such as the complete content actually used in this project, which can be simplified for the first time.

Write the key to .env (of course, Wechat scan authorization can also be selected for deployment)

TENCENT_SECRET_ID=TENCENT_SECRET_KEY=

This completes the deployment of the Serverless-based Flask Demo, and then goes on to write the rest of the code in your own way.

III. Maimai_DX

Maimai is an arcade audio game.

Put a moving picture here to experience for yourself. The original material comes from "external recording maimai" QZKago Requiem Re:MASTER ALLPERFECT Player: Ruri*R.

Japanese official website

Overseas official website

In China, the results can only be checked from the official Wechat account, and Wechat's authorization to log in is required every time you enter the page, and there is a limit on the number of records stored. Only the latest 10 albums and 50 game records are stored (that is, a queue, first-in-first-out type). This is the original intention of this project. Every score you typed should be preserved.

Dance cute check separator

The results show that the front-end Fomantic-UI, back-end Flask+MySQL. Gh open source address: https://github.com/yuangezhizao/maimai_DX_CN_probe, welcome watch, star, fork & pr!

Currently, the following functions are actually installed:

Wechat_archive contains home page, game data, photo albums and game records: the original web page has been modified and a Highcharts library has been added to visualize the curve display changes

Record contains records (paging) and differences (paging): a self-written quick preview page, which is a very practical function for viewing historical records and performance changes

Info contains the list of pavements, that is, all the basic information of the pavements is output to a page to facilitate search within the page.

Development process

Next, in chronological order, describe the problems encountered in the development process and how to solve them.

1. Serverless Framework Component configuration file

Serverless Framework is now in V2, which means it cannot follow the previous version of the serverless.yml configuration file and needs to be modified against the documentation.

a. Previous versions automatically downloaded third-party libraries to the requirements folder under the .serverless folder in the project directory according to requirements.txt to participate in the final dependency packaging, compressed into zip files, and finally uploaded to the cloud function runtime environment.

b. The latest version is no longer downloaded automatically and needs to be handled on its own. Reference usage of the official example: hook

Src: # TODO: installing the python project depends on the project's current directory hook: 'pip3 install-r requirements.txt-t. / requirements' dist:. / include:-source:. / requirements prefix:.. / # prefix, can make. / requirements files/dir to. / exclude:-.env -' requirements/**'

The comments are clear. Use hook to download third-party libraries to the requirements folder under the project directory according to requirements.txt to avoid confusion in local folder management caused by third-party libraries. Then the include specifies the prefix of the requirements folder under the project directory in the cloud, that is, for the cloud function running environment, the third-party libraries in the requirements folder and the project directory are at the same level and can be imported and used normally. Of course, the local run uses a global third-party library and does not use the requirements folder under the project directory.

two。 Overview of layer Management

The former is a very reasonable design, but new problems have been found in the actual environment. A fully consistent profile

Src: hook: 'pip3 install-r. / src/requirements.txt-t. / src/requirements' dist:. / src include:-source:. / requirements prefix:.. / exclude:-.env

After the successful deployment under macOS, you can see that the requirements folder does not exist in the cloud function editor, and the third-party library and project directory are at the same level.

However, after the successful deployment under Windows, you can see the requirements folder in the cloud function editor. In other words, the third-party library and the project directory are not at the same level, so the access will cause an error in the import of no module found.

Repeated attempts to modify configuration items such as prefix failed to debug, so two solutions are proposed here:

a. Modify the configuration file as follows so that the local third-party library and the project directory exist at the same level

Src: hook: 'pip3 install-r. / src/requirements.txt-t. / src' dist:. / src exclude:-.env

However, with the expansion of projects and third-party libraries, there will be more and more folders, which is very difficult to manage.

b. Use the layer provided by the cloud function

Although the speed of sls deploy deployment is very fast, it would be better if you can only upload project code without dealing with dependencies during deployment, so that cross-end collaborative development only needs to care about project code and ok, no longer need to manage dependencies!

And there is another point. If you want to edit the function code online in the SCF console, you need to keep the deployment package below 10MB. Don't think that 10 megabytes is very big, and it's possible to run out soon.

How to operate it exactly? That is, if you want to package the third-party library folder directly and create it as a layer, you can directly refer to it through import in the function code. After all, some special libraries, such as without vc++ under Brotli,Windows, can only go to https://lfd.uci.edu/~gohlke/pythonlibs to download wheel installation.

After normal installation under macOS, it will be imported in the form of import _ brotli in _ brotli.cpython-39-darwin.so,brotli.py, but there is a new problem. The cloud will import ModuleNotFoundError: No module named'_ brotli' "."

The current SCF execution environment is based on the following: standard CentOS 7.2

In order to solve the problem, try to package in the linux environment, pick up the CentOS 8.2CVM on hand and start operation.

Pip3 install-r requirements.txt-t. / layer-- upgradezip-r layer.zip. / layer

Then you can download the packaged layer.zip locally and upload it, and for the time being, you can do it once and for all.

By the way, the configuration file can remove hook and add layers.

Src: src:. / src exclude:-.env -'_ pycache__/**' layers:-name: maimai_DX_CN_probe version: 3

The bound layer function is triggered to run. When the concurrent instance is started, the running code of the loading function will be unzipped to the / var/user/ directory, and the layer content will be unzipped and loaded to the / opt directory. If you want to use or access the file file, place it in the root directory of the compressed file when you created the layer. After unzipping and loading, the file can be accessed directly through the directory / opt/file. If you compress the dir/file through a folder when creating a layer, you need to access the specific file through / opt/dir/file when the function is running

Experience faster deployment! Because the third-party library is already packaged in the "layer".

But the strange thing is that importing any third-party library in the cloud will report an error, so debug to check the path

For path in sys.path: print (path) / var/runtime/python3/var/user/opt/var/lang/python3/lib/python36.zip/var/lang/python3/lib/python3.6/var/lang/python3/lib/python3.6/lib-dynload/var/lang/python3/lib/python3.6/site-packages/var/lang/python3/lib/python3.6/site-packages/pip-18.0-py3.6.egg

Check opt again.

Import osdirs = os.listdir ('/ opt') for file in dirs: print (file) layer

It suddenly dawned on me that when packing, you need to pack directly in the current path. After uploading, the "layer" is updated to version 2, but the ModuleNotFoundError: No module named'_ brotli' still reports an error, and confirms that the _ brotli.cpython-38-x86_64-linux-gnu.so file actually exists.

However, there is no problem with local import on both CentOS and macOS, which is difficult, and considering that it is likely to be the problem with the python version, we look for a ready-made environment such as here:

After uploading again, the "layer" is updated to version 3, and the access is successful! The problem was finally solved, and it turned out that the same version of the Python 3.6runtime environment was needed.

3. Custom entry file

The files in the components source code tencent-flask/src/_shims/ are repackaged and uploaded to the cloud function every time. Currently, there are two files.

A. Severless_wsgi.py, the function is converts an AWS API Gateway proxied request to a WSGI request. The full name of WSGI is Python Web Server Gateway Interface, or Web server gateway interface. It is a simple and general interface between Web server and Web application or framework defined for Python language.

B. sl_handler.py, which is the default entry file

Import app # Replace with your actual applicationimport severless_wsgi# If you need to send additional content types as text, add then directly# to the whitelist:## serverless_wsgi.TEXT_MIME_TYPES.append ("application/custom+json") def handler (event, context): return severless_wsgi.handle_request (app.app, event, context)

For your own project, the factory function of Flask is used. To avoid having to modify it in the cloud function editor every time, the best way is to customize the entry file:

Import severless_wsgifrom maimai_DX_CN_probe import create_app # Replace with your actual application# If you need to send additional content types as text, add then directly# to the whitelist:## serverless_wsgi.TEXT_MIME_TYPES.append ("application/custom+json") def handler (event, context): return severless_wsgi.handle_request (create_app (), event, context)

If you specify the execution method as serverless_handler.handler, you will ok.

4. Url_for outputs http instead of https's URL

The links generated by redirecting to url_for in the view function are all http, not https. In fact, the problem is described in the Flask document Standalone WSGI Containers

In the final analysis, this is not a problem with Flask, but a problem caused by the WSGI environment. The recommended method is to use middleware, and officials have also given ProxyFix.

From werkzeug.middleware.proxy_fix import ProxyFixapp.wsgi_app = ProxyFix (app.wsgi_app, x_proto=1, x_host=1)

But the value taken from X-Forwarded-Proto is http in apigw, so this ProxyFix cannot be used directly because the community of Flask is quite perfect, and many predecessors have paved the way, so go directly to Stack Overflow to search for the solution. The cause of the Flask url_for generating http URL instead of https problem is shown in the figure: Browser-HTTPS-> Reverse proxy (apigw)-HTTP-> Flask because he has set the front-end type only https in apigw, that is, it is impossible for browser to access it using http. You can know it by printing environ.

{"CONTENT_LENGTH": "0", "CONTENT_TYPE": "," PATH_INFO ":", "QUERY_STRING": "", "REMOTE_ADDR": "", "REMOTE_USER": "", "REQUEST_METHOD": "GET", "SCRIPT_NAME": "", "SERVER_NAME": "maimai.yuangezhizao.cn", "SERVER_PORT": "80" "SERVER_PROTOCOL": "HTTP/1.1", "wsgi.errors":, "wsgi.input":, "wsgi.multiprocess": False, "wsgi.multithread": False, "wsgi.run_once": False, "wsgi.url_scheme": "http", "wsgi.version": (1,0), "serverless.authorizer": None, "serverless.event": "," serverless.context ":" "API_GATEWAY_AUTHORIZER": None, "event": "", "context": "," HTTP_ACCEPT ":" text/html,application/xhtml+xml,application/xml Qimagine 0.9 gzip deflate, br, HTTP_ACCEPT_LANGUAGE: "HTTP_ACCEPT_LANGUAGE": "zh-CN,zh;q=0.9,en" Qroom0.8 "," HTTP_CONNECTION ":" keep-alive "," HTTP_COOKIE ":"," HTTP_ENDPOINT_TIMEOUT ":" 15 "," HTTP_HOST ":" maimai.yuangezhizao.cn "," HTTP_SEC_FETCH_DEST ":" document "," HTTP_SEC_FETCH_MODE ":" navigate "," HTTP_SEC_FETCH_SITE ":" none "," HTTP_SEC_FETCH_USER ":"? 1 " "HTTP_UPGRADE_INSECURE_REQUESTS": "1", "HTTP_USER_AGENT": "Mozilla/5.0 (Macintosh) Intel Mac OS X 10: 13: 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36 "," HTTP_X_ANONYMOUS_CONSUMER ":" true "," HTTP_X_API_REQUESTID ":" 5bcb29af2ca18c1e6d7b1ec5ff7b5427 "," HTTP_X_API_SCHEME ":" https "," HTTP_X_B3_TRACEID ":" 5bcb29af2ca18c1e6d7b1ec5ff7b5427 "," HTTP_X_QUALIFIER ":" $LATEST "}

The variable in HTTP_X_FORWARDED_PROTO corresponding to apigw is HTTP_X_API_SCHEME, so the solution is as follows: app.wsgi_app = ReverseProxied (app.wsgi_app)

Class ReverseProxied (object): def _ init__ (self, app): self.app = app def _ call__ (self, environ, start_response): scheme = environ.get ('HTTP_X_FORWARDED_PROTO') if scheme: environ [' wsgi.url_scheme'] = scheme return self.app (environ, start_response) app = Flask (_ name__) app.wsgi_app = ReverseProxied (app.wsgi_app) 5. Response data compression

Compression is available for both IIS, Apache and Nginx. After all, the CVM you are using has only 1m bandwidth outside the Internet, so the compression can significantly improve the effect of shortening the first screen time. For Serverless, the response data is transmitted to the client through API Gateway, so compression should also be its ability (although the speed of the external network is greatly improved, the compression still has to be compressed), but it has not been found. Seeing that some js frameworks provide compression capabilities natively, I intend to add Flask's own compression capabilities. To put it simply, you can subscribe to the @ app.after_request signal and call the compress method of the third-party library brotli (go to gh before writing to see if there is a ready-made wheel extension, sure enough. At first, Flask-Zipper was used, and then Flask-Compress solved the problem. The measured data of 3.1 MB was reduced to 76.1 kB by brotli compression algorithm.

6. The influence of three different paths in apigw environment

The default mapping is as follows:

ID environment name access path 1 publish release2 pre-release prepub3 test test

Because the configured static_url_path is "", that is, the static folder is mapped to the / path, release, prepub and test access will naturally be 404.Therefore, the custom domain name is bound, the custom path mapping is used, and the access path of the publishing environment is set to /, so there is no problem in accessing the publishing environment.

ID environment name access path 1 release / 2 pre-release prepub3 test test7. Access VPC and public network at the same time

There are several kinds of cloud databases that can be used in cloud functions

Cloud database CDB needs to be accessed from VPC. Although it can be accessed through the public network, if you can use the internal network, you will not go to the public network.

PostgreSQL for Serverless (ServerlessDB), which is the official pg database for Serverless.

MongoDB in cloud development TCB. If you remember correctly, you need to activate internal test permission access.

Since you have migrated from the old website and the data has not been migrated yet, you can directly access the original cloud database CDB and configure the network and subnet in the cloud function. However, the public network cannot be accessed at this time. One solution is to enable public network access and public network fixed IP, so that you can access both private and public network resources. With regard to the configuration file, this project is a single-instance application, that is, only one component is introduced into the project and only one component instance is generated when deployed. However, if you want to introduce the database, you have to add new components. At present, there are no database-related configuration items in Flask Components, so you need to introduce multiple components into the project and generate multiple component instances during deployment. It is also simple to create a new folder containing serverless.yml to configure postgresql

Component: postgresql # (required) component name, here is postgresqlname: maimai_DX_CN_probe # (required) component instance name. Org: yuangezhizao # (optional) is used to record organization information. The default value is your Tencent Cloud account appid, and the string app: yuangezhizao # (optional) must be used to record organization information. The default is the same as name, and must be the string stage: dev # (optional) to distinguish environment information. The default value is devinputs: region: ap-beijing # optional ap-guangzhou, ap-shanghai, ap-beijing zone: ap-beijing-3 # optional ap-guangzhou-2, ap-shanghai-2, ap-beijing-3 dBInstanceName: maimai_DX_CN_probe # projectId: 0 dBVersion: 10.4 dBCharset: UTF8 vpcConfig: vpcId: vpc-mrg5ak88 subnetId: subnet-hqwa51dh extranetAccess: false

Then on the terminal cd to this directory and then execute sls deploy to successfully deploy postgresql

Yum install python3-devel postgresql-develpip install psycopg2

Result

Import psycopg2File "/ opt/psycopg2/__init__.py", line 51, in from psycopg2._psycopg import (# noqaImportError: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory

The following issues are being resolved:

Http forced to jump to https

Test environment is pushed to production environment

This is what the editor shares with you about the Serverless-based Dance Voice Scanner. If you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.