Shulou(Shulou.com)05/31 Report--

This article shows you what is the solution for Labelhub to provide data and models for artificial intelligence enterprises based on Serverless technology. The content is concise and easy to understand, which will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

Labelhub is a company dedicated to providing comprehensive data and model solutions for artificial intelligence enterprises, which can help AI enterprises to better manage data, thus improving the iteration speed of their core AI products. Labelhub has excellent agile teams, and the development field involves machine learning, model training and software applications. At present, it has carried out in-depth cooperation with a number of large and medium-sized enterprises, and has won many awards in industry-related competitions.

During the period of rapid business expansion, the Labelhub team chose to use Serverless technology to create a lightweight internal operation and maintenance and data visualization system. By using Tencent Serverless Framework, based on cloud Serverless services (cloud functions and triggers, etc.), you can quickly develop a customized data visualization system without configuration and deployment. Serverless technology not only meets the needs of business development, but also does not require too much manpower and capital costs, so it is the best choice for Labelhub.

Preface

My team developed an artificial intelligence enterprise data tagging product Labelhub, the current formal sales work is in the exploratory stage, for the target customers, product positioning, has not been well combed. With the gradual development of business, the operation and maintenance security of the platform does not carry out systematic monitoring and management. Therefore, I consider making the business data, server data and application monitoring data of the product as a basic internal operation and maintenance and data visualization system.

Although there are many choices of open source tools, deployment and configuration can not be avoided, and secondary development is also troublesome, so consider developing a simple system that is completely customized according to internal requirements.

In the end, I decided to use Serverless to build such a lightweight internal system. Serverless is undoubtedly one of the hottest IT words nowadays. As a new Internet software product architecture, although it was proposed as early as 2012, with the rapid development of container technology, IoT, blockchain and 5G technology in recent years, the concept of Serverless has also developed rapidly.

What it can bring, whether there is a threshold for its dividend, how far away it is from us, and whether it is worth investing in now. I will discuss with you through a series of articles on this project and look at Serverless from the perspective of an ordinary developer.

Summary description

I hope this series of articles will be easy to understand, and I will introduce my understanding of Serverless in as much detail as possible during the development of this system. Through this series of articles, I hope to provide you with the following contents:

How to develop a real world serverless app

Understanding of service split, how to make more effective use of resources

The idea of migrating existing BaaS to FaaS

(if you think of more, you will continue to complete it.)

Is based on Serverless Framework tools, while the back end uses Python for development

Project construction

Build the back-end project structure

In the document center of Serverless Framework, Flask and Django are currently available in the framework support. If you follow the document example, you will find that it is very easy to deploy a simple rest api, such as the official speedy deployment. But if you look at the structure of the project, you will find that there is only one yml configuration file, one dependency text, and one app.py file. Because the hooks parameter is used in the configuration file, the dependency will be installed into the current folder, which is a very simple example. When building a backend project directory, you usually do not need to consider the following two points when developing locally:

The introduction of Tripartite Library

Calls to public functions

For Serverless, these two issues need to be considered.

When you execute sls deploy in the example, you can view the function code in the console after successful deployment, and you will find that the dependent file is also in the file list, which explains why a function is an application. But in the actual development process, it is impossible for every functional module to install a dependency, so we can use common modules to solve this problem. But how to introduce the public module. Since Serverless cli Component v2 has cancelled the include configuration, for v1 it is easy to use include configuration to include common components in the function, so that each subfunction can be easily called. As for v2, we can actually solve this problem through Layer.

For the difference and detailed introduction of v1 and v2, please refer to the evolution of the version of Serverless Framework Cli

We can split the module through multi-instance management in application management.

Project practice

Processing of project root directory

Create a project folder

Mkdir labelhub-dashboard

Apply the configuration file under the project root

Cd labelhub-dashboardtouch serverless.yml

Only define the name of the application in the application configuration file

App: labelhub-dashboard

Processing of public files and tripartite dependent directories

Create a public module folder under the root folder

Mkdir common

We put dependent and public functions into common to facilitate calls by other modules.

# labelhub-dashboard/commontouch requirements.txt# create database connection tool class touch dataUtils.py

Since you are just testing, you can also run pip install pymysql-t. / directly under common to install the dependencies to the current path.

Pymysql is used here to connect to the database for testing

Edit the dataUtils.py file:

Import pymysqlclass MysqlUtils: def _ init__ (self): self.getConn ({'host':' xxx', 'user':' xxx', 'port': 3306,' db': 'xxx',' password': 'xxx'}) def getConn (self Conf): self.connection = pymysql.connect (host=conf ['host'], user=conf [' user'], password=conf ['password'], port=conf [' port'], db=conf ['db'], charset='utf8', cursorclass=pymysql.cursors.DictCursor, autocommit=1) def doAction (self, stmt) Data): try: self.connection.ping (reconnect=True) cursor = self.connection.cursor () cursor.execute (stmt, data) result = cursor cursor.close () return result except Exception as e: try: cursor.close () except: pass return False

When you are ready, you can deploy. As mentioned earlier, because common is where we store public functions and three-party libraries, we need to deploy with Layer components. Create a profile in common

Touch serverless.yml

Edit the configuration file:

Component: layer # Note that the layer component name: common-layerorg: labelhub-dashboardapp: labelhub-dashboardstage: devinputs: name: commonfiles # remember the name region: ap-guangzhou src: src:. / exclude:-.env runtimes:-Python3.6 description: packages

Then, after the successful deployment of the sls deploy,Layer layer, the details of the function will appear, and you need to pay attention to the value of the version field. After the deployment is completed, we will start to create the function module directory.

Processing of functional module catalogue

Create a test submodule in the root directory

Mkdir user-data

Create a test file

Touch index.py

Edit test file

From mysqlUtils import MysqlUtilsimport jsondb = MysqlUtils () def get_users (): search_stmt = ("SELECT * FROM `user` limit 100" ") result = db.doAction (search_stmt,) if result = = False: return False return result def main_handler (event, context): result = get_users () data = [{'id': user [' id'], 'name': str (user [' name']) 'created_at': user [' created_at'] .strftime ('% Y-%m-%d% HV% MV% S')} for user in result.fetchall ()] return data

Here are two points to make:

It is obvious that mysqlUtils is in the common folder, but mysqlUtils is introduced directly here, which will be explained later in the configuration file.

The query function get_users () is written here. It can also be written in mysqlUtils.py, but because mysqlUtils.py is in the Layer layer, and the deployment of the Layer layer currently takes longer than the deployment of function components, I put it in the function file I need to use. That is, try not to modify the files in common.

When you are ready, you can deploy the function. The first step is still to create the configuration file:

Touch serverless.yml

Edit the configuration file:

Component: scf # Note Here we use the scf component name: userdatastage: devapp: labelhub-dashboardorg: labelhub-dashboard# component parameter inputs: name: ${name}-${stage}-${app} # function name src: src:. / # Code path exclude:-.env handler: index.main_handler # entry runtime: Python3.6 # Environment when the cloud function runs region: ap-guangzhou # cloud function Domain layers:-name: commonfiles version: 1 events: # trigger-apigw: # Gateway trigger parameters: endpoints:-path: / method: GET

Through the configuration file, we can find that the name and version in the layers configuration are the name when the common is deployed and the version number after the deployment is successful.

Finally, execute sls deploy to complete the deployment, directly access the generated url address, and you can view the correct return information.

The structure of the final folder is:

Labelhub-dashboard/serverless.ymlcommon/requirements.txtserverless.ymlmysqlUtils.pyuser-data/serverless.ymlindex.py

When deploying the user-data function, we see that mysqlUtils is introduced directly, and we can see that we use the corresponding layers configuration in the configuration file of the user-data function. As you can see here, in the configuration of the function, layers is actually equivalent to the include configuration in v1, and the files in the default Layer component are in the same directory as the function files.

In fact, after we have created all the files, execute sls deploy-all in the root directory to deploy at one time, but in the process of use, there will be an error in the deployment of the function component, and the corresponding Layer component cannot be found. This is also because the configuration of the layers will be read in the deployment process of the function component, and it is found that the deployment of the Layer component is almost much slower than the function component in the actual use process. Therefore, it is also recommended to give priority to the deployment of Layer components, and then uniformly deploy function components. Then we need to consider whether the application root of the function can be at the same level as the common component directory, so that when using sls deploy-all, we can avoid updating the Layer layer at the same time.

The above is what is the solution for Labelhub to provide data and models for artificial intelligence enterprises based on Serverless technology. Have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.