Shulou(Shulou.com)05/31 Report--

This article introduces the knowledge about "Pytorch how to use Google Colab to train neural network depth". In the actual case operation process, many people will encounter such difficulties. Next, let Xiaobian lead you to learn how to deal with these situations! I hope you can read carefully and learn something!

What is Google Colab?

Google Colab is a free Jupyter laptop environment provided by Google. It can be used without any settings and environment configuration. It runs completely in the cloud. Does not affect local usage.

Google Colab provides researchers with a certain amount of free GPU to write and execute code, all of which is freely available through the browser. Students can easily run Tensorflow, Pytorch and other deep learning frameworks on it.

Although Google Colab provides some free resources, the amount of resources is limited, and all Colab runtime resets after a while. Colab Pro subscribers will still be limited in usage, but about twice as much as non-subscribers. Colab Pro+ subscribers also enjoy increased stability.

Training with Colab

This article demonstrates the use of Colab using the training of the YoloV4-Tiny-Pytorch version as an example.

1. Upload of data set and pre-training weights 1. Upload of data set

Colab and Google's own cloud disk linkage is very good, so we need to first upload the dataset to the cloud disk, this upload process is actually very simple, locally prepare the dataset first.

Since all the libraries I uploaded use VOC datasets, we need to arrange them according to VOC datasets

The VOC07+12 dataset is used as an example.

JPEGImages stores image files, Annotations stores tag files, ImageSets stores txt files that distinguish validation sets, training sets, and test sets.

Then package the VOCdevkit file as a whole. It should be noted that instead of packaging the above three folders, VOCdevkit is packaged to meet the format of data processing.

After obtaining the packaged compressed package, upload the compressed package to Google Cloud Disk. I created a VOC_datasets folder on Google Drive to store compressed packages.

The upload of the dataset is now complete.

2. Upload of pre-training weights

To create a folder on Google Drive, first create Models, then yolov4-tiny-pytorch inside Models, then create logs and model_data inside yolov4-tiny-pytorch.

Model_data is a pre-training file.

logs are weights generated during network training.

Since we are using the YoloV4-Tiny-Pytorch library this time, we upload its pre-trained weights to the model_data folder.

Second, open Colab and configure the environment 1, notebook creation

In this step, we first open Colab's official website.

Then click File to create a notebook, which creates a jupyter notebook.

After creating the file, change the name to look better.

Then click Code Execution Program, then click Change Runtime Type, select GPU in the Hardware Accelerator section, Colab will configure a machine with GPU, and the notebook will be created.

2. Simple configuration of environment

Colab already integrates the pytorch environment, so there is no need to configure pytorch specifically, but the torch version used is newer.

Since our dataset is on Google Cloud Drive, we also need to mount Cloud Drive.

from google.colab import drivedrive.mount('/content/gdrive')

We enter the above code into a notebook and execute it. Mount the cloud disk on the server. Then click Run.

At this point, click on the left column, something similar to a folder, you can open the folder to see the file deployment situation. Gdrive is our Google Drive configuration. If not, refresh on the left.

Open gdrive, which contains our dataset.

3. Download of Deep Learning Library

In this step, we need to complete the download of the deep learning repository, we use the git clone command to download. After executing the following command, the yolov4-tiny-pytorch folder is added to the file on the left. If not, refresh on the left.

Then we moved the root directory to the yolov4-tiny-pytorch folder via the cd command.

! git clone https://github.com/bubbliiiing/yolov4-tiny-pytorch.git%cd yolov4-tiny-pytorch/

4. Replication and decompression of data sets

Placing the dataset directly on Google Drive would result in a lot of cloud data transfer, which is much slower than local files, so we need to copy the dataset locally for processing.

We enter the following code to copy and extract files. The first step is to delete the empty VOCdevkit folder. Then decompress it.

Since the zip file is used here, the unzip command is used. If it is another form of compressed package, the command needs to be modified according to the format of the compressed package (please Baidu students). After executing the following instructions, you can see that the VOC dataset has been decompressed in the file on the left. If not, refresh on the left.

! rm -rf ./ VOCdevkit! cp /content/gdrive/MyDrive/VOC_datasets/VOC07+12+test.zip ./! unzip ./ VOC07+12+test.zip -d ./

5. Save path settings

The default save path of the code provided in this article is the logs folder, but Colab has unstable problems and will be disconnected after running for a while.

If the weights are stored in the logs folder under the original root directory, the network training will be in vain and a lot of time will be wasted.

Google cloud disk can be soft connection to the root directory, then even if disconnected, the weight is also retained in the cloud disk.

Earlier in this article, I created the logs folder in the cloud drive. Link the folder over.

! rm -rf logs! ln -s /content/gdrive/MyDrive/Models/yolov4-tiny-pytorch/logs logs

III. Start training 1. Processing of labeled files

Open the voc_annotation.py file, and since we are now using the VOC dataset directly, we have divided the training set validation set and test set, so we set annotation_mode to 2.

Then enter the command to complete the tag processing, generating 2007_train.txt and 2007_val.txt.

! python voc_annotation.py

2. Processing of training documents

The processing training file consists of three main parts:

1. Use of pre-training files.

2. The setting of the storage cycle, this setting is because the storage space of the cloud disk is limited, and each generation of storage will lead to full storage space.

a. Use of pre-training files

First modify the model_path to point to the weight file we uploaded to Google Drive. In the file column on the left, find models/yolov4-tiny-pytorch/model_data, copy the weight path.

Replace model_path on the right.

b. Setting of storage cycle

Some repositories have already been updated, adding the save_period parameter every generation. You can change save_period directly. In this article, we set save_period to 4, that is, save every 4 generations.

Not yet updated repository can only be saved every generation, remember to occasionally go to google cloud disk delete it.

3. Start training

At this time, enter in the notebook:

! python train.py

You can start training.

What about disconnection? 1, anti-drop measures

It is said that the frequency of dropped calls can be reduced by automatic clicking.

Press F12 in Google colab, click on the console of the web page, and paste the following code:

function ConnectButton(){ console.log("Connect pushed"); document.querySelector("#top-toolbar > colab-connect-button").shadowRoot.querySelector("#connect").click()}setInterval(ConnectButton,60000);2, Finished or dropped?

Nothing can be done about it. Cheap things must have their disadvantages.

Follow the steps to do it again, and then set the pre-training weights to the trained weight files in the logs folder.

In addition, parameters such as Init_epoch also need to be adjusted.

"Pytorch how to use Google Colab to train neural network depth" content introduced here, thank you for reading. If you want to know more about industry-related knowledge, you can pay attention to the website. Xiaobian will output more high-quality practical articles for everyone!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.