23 lines
1.0 KiB
Markdown
23 lines
1.0 KiB
Markdown
|
# Activation Function (AKA Transfer function)
|
||
|
|
In a neural network activation function adds non-linearity to it.
|
||
|
|
Types:
|
||
|
|
1. Sigmoid(Logistic) (used mostly for output layer(looks like probability))
|
||
|
|
2. RelU or Rectified Linear Unit (important discovery for NN - most-used for hidden layers)(not suitable for output layer if output is supposed to be probability) and leaky RelU with some slope on negative part
|
||
|
|
3. tanH (Hyperbolic) (-1 - 1) or ArcTan (Tan Inverse -> maps to -Pi/2 - Pi/2)
|
||
|
|
4. Linear(or Identity) layer (used for output layers(best for regression))
|
||
|
|
5. Softmax (classification giving probability) (probability coz outputs add upto 1)
|
||
|
|
6. SquareRoot
|
||
|
|
7. Exponential
|
||
|
|
8. Sine.
|
||
|
|
9. Ramp
|
||
|
|
10. Step (Binary)
|
||
|
|
11. Unit Sum
|
||
|
|
|
||
|
|
if the network computation is something that is multiplicative, use log as activation so that the sum becomes addition.
|
||
|
|
|
||
|
|
Constraint Optimization: optimize in such a way that the output is constrained to some value.
|
||
|
|
|
||
|
|
|
||
|
|
Steps => number of iteration of batches
|
||
|
|
Epoch => number of iterations of going throught the entire dataset
|