Shulou(Shulou.com)06/01 Report--

What is the construction and use of mlflow? I believe many inexperienced people are at a loss about it. Therefore, this paper summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

Background

Mlflow is Databrick's open source machine learning management platform. It decouples algorithm training and algorithm model services, making algorithm engineers focus on model training without paying too much attention to services.

And there are more than ten services in our company that have been running steadily for more than two years.

Set up

The construction of mlflow is mainly the construction of mlflow tracking server, and tracking server is mainly used for model metadata and model data storage.

This time, we use minio as the storage background of model data and mysql as the storage of model metadata, because this pattern can meet online requirements, not just for testing.

Construction of minio

Use the building of MinIO, and create a bucket named mlflow to facilitate subsequent operations.

Construction of mlflow

# create conda environment and install python 3.6conda create-n mlflow-1.11.0 python==3.6# activate conda environment conda activate mlflow-1.11.0# install mlfow tracking server python required dependency package pip install mlflow==1.11.0 pip install mysqlclientpip install boto3 expose minio url and required ID and KEY Because mlflow tracking server needs export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLEexport AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEYexport MLFLOW_S3_ENDPOINT_URL= http://localhost:9001mlflow server\-- backend-store-uri mysql://root:AO,h07ObIeH-@localhost/mlflow_test\-- host 0.0.0.0-p 5002\-- default-artifact-root s3://mlflow when uploading model files.

When you visit localhost:5002, you can see the following interface:

Startup of mlflow tracking server

Installation of conda

Refer to install conda and install different conda environments according to your own system

Mlfow tracking server installation

Use

Copy the following wine.py file

Import osimport warningsimport sysimport pandas as pdimport numpy as npfrom sklearn.metrics import mean_squared_error, mean_absolute_error, r2_scorefrom sklearn.model_selection import train_test_splitfrom sklearn.linear_model import ElasticNetimport mlflow.sklearndef eval_metrics (actual, pred): rmse = np.sqrt (mean_squared_error (actual, pred)) mae = mean_absolute_error (actual, pred) R2 = r2_score (actual, pred) return rmse, mae R2if _ _ name__ = "_ _ main__": warnings.filterwarnings ("ignore") np.random.seed (40) # Read the wine-quality csv file (make sure you're running this from the root of MLflow!) Wine_path = os.path.join (os.path.dirname (os.path.abspath (_ file__)), "wine-quality.csv") data = pd.read_csv (wine_path) # Split the data into training and test sets. (0.75,0.25) split. Train, test = train_test_split (data) # The predicted column is "quality" which is a scalar from [3,9] train_x = train.drop (["quality"], axis=1) test_x = test.drop (["quality"] Axis=1) train_y = train [["quality"]] test_y = test [["quality"]] alpha = float (sys.argv [1]) if len (sys.argv) > 1 else 0.5 l1_ratio = float (sys.argv [2]) if len (sys.argv) > 2 else 0.5 mlflow.set_tracking_uri ("http://localhost:5002") client = mlflow.tracking.MlflowClient () mlflow.set_experiment ('http_metrics_test)" ') with mlflow.start_run (): lr = ElasticNet (alpha=alpha L1_ratio=l1_ratio, random_state=42) lr.fit (train_x, train_y) predicted_qualities = lr.predict (test_x) (rmse, mae, R2) = eval_metrics (test_y, predicted_qualities) print ("Elasticnet model (alpha=%f, l1_ratio=%f):"% (alpha) L1_ratio) print ("RMSE:% s"% rmse) print ("MAE:% s"% mae) print ("R2:% s"% R2) mlflow.log_param ("alpha", alpha) mlflow.log_param ("l1_ratio", l1_ratio) mlflow.log_metric ("rmse", rmse) mlflow.log_metric ("R2") R2) mlflow.log_metric ("mae", mae) mlflow.sklearn.log_model (lr, "model")

Note:

1.mlflow.set_tracking_uri ("http://localhost:5002") is set to the address of the mlflow tracking server you just started

The name of the 2.mlflow.set_experiment ('http_metrics_test') setup experiment

3. Install the python package that the program depends on

4. If you are not in the same conda environment, you have to execute the

Export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY export MLFLOW_S3_ENDPOINT_URL= http://localhost:9001

Easy for python clients to upload model files and model metadata

If the direct execution of python wine.py is successful, you can visit the mlflow tracking server ui as follows

Click 10:34:38 on 2020-10-30, as follows:

Start the mlflow algorithm service

Execute commands in the same conda environment

Export MLFLOW_TRACKING_URI= http://localhost:5002 mlflow models serve-m runs:/e69aed0b22fb45debd115dfc09dbc75a/model-p 1234-- no-conda

Where e69aed0b22fb45debd115dfc09dbc75a is the run id in mlflow tracking server ui

If you encounter ModuleNotFoundError: No module named 'sklearn'

Execute pip install scikit-learn==0.19.1

Encountered ModuleNotFoundError: No module named 'scipy'

Execute pip install scipy

Request access to the services started by the model:

Curl-X POST-H "Content-Type:application/json Format=pandas-split "- data'{" columns ": [" alcohol "," chlorides "," citric acid "," density "," fixed acidity "," free sulfur dioxide "," pH "," residual sugar "," sulphates "," total sulfur dioxide "," volatile acidity "]," data ": [[12.8,0.48,0.48,6.2,29,3.33,1.2,0.39,75" 0.66]]} 'http://127.0.0.1:1234/invocations

The output [5.455573233630147] indicates that the model service is deployed successfully.

After reading the above, have you mastered the method of building and using mlflow? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.