Activation Functions

Activation functions add NON-LINEARITY so networks can learn complex patterns.

Common: ReLU (default for hidden layers), Sigmoid (binary output), Softmax (multi-class output), Tanh.

Example 1 (python)

import numpy as np
def relu(x): return np.maximum(0, x)
print(relu(np.array([-2, -1, 0, 1, 2])))

Output

[0 0 0 1 2]

ReLU zeros out negatives.

Example 2 (python)

def sigmoid(x): return 1/(1+np.exp(-x))
print(sigmoid(0))

Output

0.5

Sigmoid squashes to (0, 1).

Key points

💡 Note: Without activation functions, a deep network collapses to a single linear model — no matter how many layers.