ABSTRACT
Activation functions are necessary for deep learning models to be
able to represent non-linear relationships. Their derivative is required
during training, however, many non differentiable activation
functions are commonly used in neural networks, such as the Rectified
Linear Unit (ReLU) and its variants. This paper discusses the
impact of non differentiability on activation functions during the
training phase, and how these functions compare to differentiable
alternatives. To analyse this problem, we trained neural networks
in an image classification problem using various activation functions.
We showed that non-differentiable points occur rarely during
training, especially in deep models, and have little to no negative
impact in a model’s performance.