Shulou(Shulou.com)06/03 Report--

This article mainly explains "what are the common mistakes in putting PyTorch into production". The explanation in this article is simple and clear, easy to learn and understand. Please follow the ideas of Xiaobian and go deep into it together to study and learn "what are the common mistakes in putting PyTorch into production"!

Error #1 -Saving Dynamic Graphs in Inference Mode

If you've used TensorFlow before, you probably know the key difference between TensorFlow and PyTorch-static and dynamic graphs. Debugging TensorFlow is difficult because the graph has to be rebuilt every time the model changes. It takes time, effort and hope. Of course, TensorFlow is better now.

In general, to make debugging easier, ML frameworks use dynamic graphs that relate to so-called Variables in PyTorch. Each variable you use is linked to the previous variable to build a relationship that propagates back.

Here's what it looks like in practice:

In most cases, you want to optimize all calculations after the model has been trained. If you look at torch's interface, there are many options, especially in terms of optimization. The eval pattern, detach, and no_grad methods cause a lot of confusion. Let me explain how they work. After the model is trained and deployed, here are the things you care about: speed, velocity, and CUDA memory overflow exceptions.

To speed up the PyTorch model, you need to switch it to eval mode. It informs all layers to use batchnorm and dropout layers in inference mode (simply, not dropout). Now, there is a detach method that separates a variable from its computation graph. It's useful when you're building a model from scratch, but it's less useful when you want to reuse SOTA's model. A more global solution would be to use torch.no_grad in context for forward propagation. This eliminates the need to store gradients of variables in the graph in the results, thereby reducing memory consumption. It saves memory and simplifies calculations, so you get more speed and less memory usage.

Error #2 -cdnn optimization algorithm not enabled

There are many Boolean flags you can set in nn.Module, but there is one you must know. Use ccudn.benchmark = True to optimize ccudn.benchmark. By setting ccudn.enabled = True, you can ensure that ccudn. enabled is indeed looking for the best algorithm. NVIDIA offers you a lot of amazing features in terms of optimization that you can benefit from.

Note that your data must be on the GPU and the model input size should not change. The more the shape of the data changes, the less optimization can be done. For example, to normalize the data, the image can be preprocessed. In short, there can be variations, but not too much.

Mistake #3 -Reusing JIT-compilation

PyTorch provides an easy way to optimize and reuse models from different languages (see Python-To-Cpp). If you are brave enough, you may be more creative and embed your models in other languages.

JIT-compilation allows optimization of the computation graph without changing the input shape. It means that JIT is an option if your data shape doesn't change much (see Error #2). To be honest, it's not much different than the above no_grad and cdnn, but it might be. This is only the first version and has huge potential.

Note that if you have conditions in your model, which is common in RNNs, it will not work.

Error #4 -Trying to extend CPU usage

GPUs are expensive, and cloud virtual machines are just as expensive. Even with AWS, an instance will cost you about $100/day (the lowest price is $0.7/hour). One might think,"What if I replaced one GPU with five CPUs? "。Everyone who tried knows it's a dead end. Yes, you can optimize a model for CPU, but eventually it will still be slower than GPU. Believe me, I strongly recommend forgetting that thought.

Mistake #5 -Handling vectors instead of matrices

cudnn - check

no_grad - check

GPU with correct version of CUDA - check

JIT-compilation - check

Everything is ready, what else can be done?

Now it's time to use a little math. If you remember how most NNs are trained with so-called tensors. A tensor is mathematically an n-dimensional array or multilinear geometric vector. What you can do is group the input (if you have enough time) into tensors or matrices and input it into your model. For example, use an image array as the matrix sent to PyTorch. The performance gain is equal to the number of objects delivered simultaneously.

This is an obvious solution, but few people actually use it because most of the time objects are handled one by one, and setting up such flows on the process can be a bit difficult. Don't worry, you'll make it!

Thank you for reading, the above is "PyTorch put into production what are the common mistakes" of the content, after the study of this article, I believe that we put PyTorch into production what are the common mistakes of this problem has a deeper understanding, the specific use of the situation also needs to be verified by practice. Here is, Xiaobian will push more articles related to knowledge points for everyone, welcome to pay attention!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.