Shulou(Shulou.com)06/01 Report--

This article introduces the relevant knowledge of "how to deploy models with PyTorch". In the operation process of actual cases, many people will encounter such difficulties. Next, let Xiaobian lead you to learn how to deal with these situations! I hope you can read carefully and learn something!

Docker installation

The best way to install torchserve is to use docker. You just have to pull the mirror down.

You can save the most recent image using the following command.

docker pull pytorch/torchserve:latestHandlers

The handler is responsible for using the model to predict one or more HTTP requests.

Default handlers

Torchserve supports the following default handlers

image_classifier

object_detector

text_classifier

image_segmenter

But remember, none of them support batching requests!

Custom handlers

Torchserve provides a rich interface to do almost anything you want. A Handler is a class that must have three functions.

preprocess

inference

postprocess

You can create your own class or subclass BaseHandler. The main advantage of subclassing BaseHandler is that the loaded model can be accessed on self.model. The following code snippet shows how to subclass BaseHandler.

Subclassing BaseHandler to create your own handler

Back to the image classification example. we need

Get images from each request and preprocess them

Get predictions from models

Send back a response

pretreatment

The.preprocess function accepts an array of requests. Suppose we are sending an image to the server, and the serialized image can be accessed from the data or body field of the request. Therefore, we can traverse all requests and preprocess each image individually. The complete code is shown below.

Preprocess each image in each request

self.transform is our preprocessing transform, nothing fancy. This is a classic preprocessing step for models trained on ImageNet.

Our transformation.

After we preprocess each image in each request, we concatenate them to create a pytorch tensor.

reasoning

Reasoning on models

This step is simple, we get the tensor from.preprocess. Prediction results are then extracted for each image.

post-processing

Now that we have predictions for each image, we need to return something to the customer. Torchserve always returns an array. BaseHandler also automatically opens a.json file with index -> label mapping (we'll see how to provide such a file later) and stores it in self.mapping. We can return a dictionary array for each prediction, which contains categories of label and index.

To pack everything together, our handler looks like this:

Because all the processing logic is encapsulated in a class, you can easily unit test it!

Export your model

Torchserve needs to provide a.mar file, which simply packages your model and all its dependencies. To package, you first need to export the trained model.

export models

There are three ways to derive a torchserve model. So far, the best approach I have found is to trace the model and store the results. This way we don't need to add any extra files to torchserve.

Let's look at an example where we will deploy a fully trained ResNet34 model.

In order, we:

loading model

Create a dummy input

Use torch.jit.trace to trace the input to the model

save the model

Create a.mar file

You need to install torch-model-archiver

git clone https://github.com/pytorch/serve.gitcd serve/model-archiverpip install .

Then, we are ready to create the.mar file by using the following command:

torch-model-archiver --model-name resnet34 \--version 1.0 \--serialized-file resnet34.pt \--extra-files ./ index_to_name.json,./ MyHandler.py \--handler my_handler.py \--export-path model-store -f

In order. The variable--model-name defines the final name of the model. This is important because it will be the namespace of the endpoint, responsible for making predictions. You can also specify a--version. serialized-file points to the stored.pt model we created earlier. -- A handler is a python file in which we invoke our custom handler. Generally speaking, it goes like this:

my_handler.py

It exposes a handle function from which we invoke methods in our custom handler. You can use the default handler with the default name (e.g., --handler image_classifier).

In--extra-files, you need to pass paths to all files your handlers are using. In this case, we have to add a path to the.json file. Use all human-readable tag names and define each category in MyHandler.py.

If you pass an index_to_name.json file, it will be automatically loaded into the handler and accessed via self.mapping.

--export-path is where.mar is stored, and I added-f to overwrite the original file.

If all goes well, you can see resnet34.mar stored in./ model-store path.

Using models for services

This is a simple step and we can run the torchserve docker container with all necessary parameters.

docker run --rm -it \-p 3000:8080 -p 3001:8081 \-v $(pwd)/model-store:/home/model-server/model-store pytorch/torchserve:0.1-cpu \torchserve --start --model-store model-store --models resnet34=resnet34.mar

I bind container ports 8080 and 8081 to 3000 and 3001 respectively (8080/8081 is already used in my machine). And then I went from... Model-store creates a volume. Finally, I call torchserve by padding the model-store and specifying the name of the model by way of a key-value list.

Here, torchserve has an endpoint /predictions/resnet34 that we can predict by sending images. This can be done using curl.

curl -X POST http://127.0.0.1:3000/predictions/resnet34 -T inputs/kitten.jpg

Reply:

{ "label": "tiger_cat", "index": 282}

Working normally!

"How to deploy models with PyTorch" is introduced here, thank you for reading. If you want to know more about industry-related knowledge, you can pay attention to the website. Xiaobian will output more high-quality practical articles for everyone!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.