Activations
Activation functions in ND4J — the Activation enum, IActivation interface, mathematical definitions, and usage in layers
Activation functions introduce non-linearity into neural networks. Without them, stacking multiple layers would be equivalent to a single linear transformation, regardless of depth. ND4J provides all activation functions through a common Activation enum and the IActivation interface.
Using Activations
In Layer Configuration
The most common way to use activations is through the Activation enum when configuring layers:
import org.nd4j.linalg.activations.Activation;
new DenseLayer.Builder()
.nIn(784).nOut(256)
.activation(Activation.RELU)
.build()As a Standalone Layer
You can use ActivationLayer when you want the activation separate from the linear transformation:
import org.deeplearning4j.nn.conf.layers.ActivationLayer;
.addLayer("relu", new ActivationLayer(Activation.RELU), "dense1")Directly on INDArrays
Apply activations to raw tensors using the Transforms class:
The second argument controls in-place behavior: true modifies x directly, false returns a copy.
Via the IActivation Interface
For programmatic access, get the IActivation instance from the enum:
Available Activations
All activations are in the org.nd4j.linalg.activations.Activation enum. Implementations are in org.nd4j.linalg.activations.impl.
ReLU Family
ReLU
RELU
f(x) = max(0, x)
Default choice for hidden layers. Fast, effective, but can suffer from "dying ReLU"
Leaky ReLU
LEAKYRELU
f(x) = max(alpha * x, x), alpha=0.01
Avoids dying ReLU by allowing small negative gradients
RReLU
RRELU
f(x) = max(alpha * x, x), alpha ~ U(l,u)
Randomized leaky ReLU. l=1/8, u=1/3 by default. Uses (l+u)/2 at test time
ReLU6
RELU6
f(x) = min(max(0, x), 6)
Capped ReLU for mobile/quantized networks
Thresholded ReLU
THRESHOLDEDRELU
f(x) = x if x > theta, 0 otherwise. theta=1.0
Sparse activations
PReLU
Via PReLULayer
f(x) = max(alpha * x, x), alpha learned
Parametric ReLU — alpha is a trainable parameter
ELU
ELU
f(x) = x if x >= 0, alpha*(exp(x)-1) if x < 0. alpha=1.0
Smooth alternative to ReLU with negative values
SELU
SELU
f(x) = lambda * (x if x >= 0, alpha*(exp(x)-1) if x < 0)
Self-normalizing. Use with WeightInit.LECUN_NORMAL and AlphaDropout
Smooth Activations
GELU
GELU
f(x) = x * Phi(x), where Phi is the Gaussian CDF
Used in Transformers (BERT, GPT). Smooth approximation of ReLU
Mish
MISH
f(x) = x * tanh(softplus(x))
Self-regularized, smooth. Good general-purpose alternative to ReLU
Swish
SWISH
f(x) = x * sigmoid(x)
Smooth, non-monotonic. Discovered via neural architecture search
Softplus
SOFTPLUS
f(x) = log(1 + exp(x))
Smooth approximation of ReLU
Sigmoid Family
Sigmoid
SIGMOID
f(x) = 1 / (1 + exp(-x))
Output range (0,1). Use for binary classification output
Hard Sigmoid
HARDSIGMOID
f(x) = min(1, max(0, 0.2x + 0.5))
Fast piecewise linear approximation of sigmoid
Tanh Family
Tanh
TANH
f(x) = (exp(x) - exp(-x)) / (exp(x) + exp(-x))
Output range (-1,1). Alternative to ReLU for hidden layers, especially RNNs
Hard Tanh
HARDTANH
f(x) = -1 if x < -1, 1 if x > 1, x otherwise
Fast piecewise linear approximation of tanh
Rectified Tanh
RECTIFIEDTANH
f(x) = max(0, tanh(x))
Combination of ReLU and tanh
Rational Tanh
RATIONALTANH
f(x) = 1.7159 * tanh(2x/3) with rational approximation
Fast approximation from LeCun 1998
Output Activations
Softmax
SOFTMAX
f_i(x) = exp(x_i - max) / sum_j(exp(x_j - max))
Multi-class classification output. Outputs sum to 1. Pair with LossMCXENT
Identity
IDENTITY
f(x) = x
Regression output (linear). Pair with LossMSE
Other Activations
Softsign
SOFTSIGN
f(x) = x / (1 + |x|)
Alternative to tanh — converges polynomially instead of exponentially
Cube
CUBE
f(x) = x^3
Rarely used in practice
Recommended Pairings
Choosing the right activation depends on the layer type and task:
Hidden Layers
Feed-forward, CNN
RELU
WeightInit.RELU
Fast convergence, avoids vanishing gradient
RNN (LSTM, GRU)
TANH
WeightInit.XAVIER
Bounded output prevents exploding activations in recurrence
Self-normalizing nets
SELU
WeightInit.LECUN_NORMAL
Maintains mean=0, variance=1 through layers
Transformer blocks
GELU
WeightInit.XAVIER
Smooth, works well with attention mechanisms
Output Layers
Multi-class classification
SOFTMAX
LossMCXENT
Outputs are valid probabilities summing to 1
Binary classification
SIGMOID
LossBinaryXENT
Output is probability in (0,1)
Multi-label classification
SIGMOID
LossBinaryXENT
Each output is independent binary decision
Regression
IDENTITY
LossMSE or LossMAE
Linear output, no bounds
Bounded regression [0,1]
SIGMOID
LossMSE
Output constrained to (0,1)
Custom Activations
Implement the IActivation interface at org.nd4j.linalg.activations.IActivation:
Use a custom activation in a layer:
Activation Function Comparison
For quick reference, key properties of each activation:
ReLU
[0, inf)
Yes
No
No
Very Low
Leaky ReLU
(-inf, inf)
Yes
No
Yes
Very Low
ELU
(-alpha, inf)
Yes
Yes
~Yes
Medium
SELU
(-lambda*alpha, inf)
Yes
Yes
~Yes
Medium
GELU
~(-0.17, inf)
No
Yes
No
High
Mish
~(-0.31, inf)
No
Yes
No
High
Swish
~(-0.28, inf)
No
Yes
No
Medium
Sigmoid
(0, 1)
Yes
Yes
No
Medium
Tanh
(-1, 1)
Yes
Yes
Yes
Medium
Softmax
(0, 1) per class
N/A
Yes
No
Medium
Identity
(-inf, inf)
Yes
Yes
Yes
None
Last updated
Was this helpful?