Types of Activation Functions

An activation function is defined by $\phi(v)$ and defines the output of a neuron in terms of its input (aka induced local field) $v$. There are three types of activation functions

1. Threshhold function an example of which is
   $$\phi(v) = \left\{ \begin{array}{ll} 1 & \mbox{if $v \geq 0$} \\ 0 & \mbox{if $v < 0$} \end{array} \right.$$
   This function is also termed the Heaviside function.
2. Piecewise Linear
   $$\phi(v) = \left\{ \begin{array}{ll} 1 & \mbox{if $v \geq +1/2$} \... ...mbox{if $1/2 > v > -1/2$} \\ 0 & \mbox{if $v \leq -1/2$} \end{array} \right.$$

3. Sigmoid Examples include
   • Logistic function whose domain is $[0,1]$
     $$\phi(v) = \frac{1}{1 + \exp(-av)}$$

   • The hyperbolic tangent whose domain is $[-1,-1]$
     

     $$\phi(v) = \frac{1- \exp(-av)}{1 + \exp(-av)}$$

     $$= \tanh(av/2)$$

     
     

   • Algebraic sigmoid function whose domain is $[-1,-1]$
     $$\phi(v) = \frac{v}{\sqrt{1 + v^2}}$$