Saturated Linear Unit as an Universal Symmetric Activation Function for Deep Learning

Abstract

There is a number of symmetric activation functions used in artificial neural networks for deeplearning. In this paper, we propose a universal activation function based on the Saturated Linear Unit(SaLU) that can be used instead of any known symmetric activation function. It is not necessary forclassification tasks to have an exact calculation of the probability of detected classes. Theclassification decision is made based on the highest probability for the input values. We propose, asa proof of concept, that the two most commonly used hyperbolic tangent and algebraic sigmoidactivation functions can be effectively replaced by SaLU by choosing a single parameter. Moreover,the theoretical step function can also be replaced by SaLU for a wider transition range. Allderivations use symbolic processing. Also shown is a visualization of the range of inputs that resultin a suitable classification. This can help scientists and programmers design complex machine learning algorithms and understand how deep learning algorithms work.