Deep-Learning-Course/SecondSunday/Notes.md

Activation Function (AKA Transfer function)

In a neural network activation function adds non-linearity to it. Types:

1. Sigmoid(Logistic) (used mostly for output layer(looks like probability))
2. RelU or Rectified Linear Unit (important discovery for NN - most-used for hidden layers)(not suitable for output layer if output is supposed to be probability) and leaky RelU with some slope on negative part
3. tanH (Hyperbolic) (-1 - 1) or ArcTan (Tan Inverse -> maps to -Pi/2 - Pi/2)
4. Linear(or Identity) layer (used for output layers(best for regression))
5. Softmax (classification giving probability) (probability coz outputs add upto 1)
6. SquareRoot
7. Exponential
8. Sine.
9. Ramp
10. Step (Binary)
11. Unit Sum

if the network computation is something that is multiplicative, use log as activation so that the sum becomes addition.

Constraint Optimization: optimize in such a way that the output is constrained to some value.

Steps => number of iteration of batches Epoch => number of iterations of going throught the entire dataset