Shulou(Shulou.com)06/01 Report--

What is the role of RELU in deep learning? aiming at this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

Activation functions in neural networks and deep learning play an important role in stimulating hidden nodes to produce better output. The main purpose of the activation function is to introduce nonlinear characteristics into the model.

In an artificial neural network, given an input or a set of inputs, the activation function of a node defines the output of the node. Standard integrated circuits can be regarded as controllers for activating functions, which can be "ON" or "OFF" depending on the input.

Sigmoid and tanh are monotonous and differentiable activation functions, which were popular before the emergence of RELU. However, over time, these functions become saturated, which leads to the disappearance of gradients. Another and most popular activation function to solve this problem is the Line Correction Unit (ReLU).

In the figure above, the blue line represents the straight line unit (ReLU), while the green line is a variant of ReLU, called Softplus. Other variants of ReLU include Leaky ReLU, ELU, SiLU, and so on, to improve the performance of certain tasks.

In this article, we only consider the straight line unit (ReLU), because by default, it is still the most commonly used activation function for performing most deep learning tasks. Its variants are usually used for specific purposes, where they may have a slight advantage in ReLU.

This activation function was first introduced into a dynamic network by Hahnloser et al in 2000 and has strong biological motivation and mathematical proof. Compared with the widely used activation functions before 2011, such as logistic sigmoid (inspired by probability theory and logistic regression) and its more practical tanh (corresponding function hyperbolic tangent), it was proved for the first time in 2011 that this function can better train deeper networks.

As of 2017, the rectifier is the most popular activation function in deep neural networks. The rectifier unit is also called rectifier linear unit (ReLU).

The biggest problem with RELU is that it is not differentiable at point 0. Researchers tend to use differentiable functions, such as S-type and tanh. But the case that can be differentiable at 0 is still a special case, so ReLU is still the best activation function for deep learning so far. After all, the amount of computation it needs is very small, and the computing speed is very fast.

The ReLU activation function is differentiable at all points except point 0. For values greater than 0, we only consider the maximum value of the function. You can write like this:

F (x) = max {0, z}

In a nutshell, you can also write:

If input > 0:

Return input

Else:

Return 0

All negative numbers default to 0, and the maximum value of positive numbers is considered.

For the back propagation calculation of neural network, the discrimination of ReLU is relatively easy. The only assumption we have to make is that the derivative at point 0 is also considered to be 0. This is usually not a big problem and works well in most cases. The derivative of a function is the value of the slope. The slope of a negative value is 0.0 and the slope of a positive value is 1.0.

The main advantages of the ReLU activation function are:

Convolution layer and deep learning: they are the most commonly used activation functions in convolution layer and deep learning model training.

The calculation is simple: the rectifier function is easy to implement, requiring only a max () function.

Representative sparsity: an important advantage of the rectifier function is that it can output a true zero value.

Linear behavior: when the behavior of the neural network is linear or nearly linear, it is easier to be optimized.

However, the main problem after the RELU unit is that all negative values immediately become zero, which reduces the ability of the model to fit or train the data correctly.

This means that any negative input to the ReLU activation function immediately changes the value in the graph to zero, which in turn affects the result graph because negative values are not mapped properly. However, this problem can be easily fixed by using different variants of ReLU activation functions, such as Leaky ReLU and other functions discussed earlier.

The answer to the question about RELU and its role in deep learning is shared here. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel to learn more about it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.