Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Special algorithms for gradient descent.
At a simple level, activation functions help decide whether a neuron should be activated. This helps determine whether the information that the neuron is receiving is relevant for the input. The activation function is a non-linear transformation that happens over an input signal, and the transformed output is sent to the next neuron.
The recommended method to use activations is to add an activation layer in your neural network, and configure your desired activation:
Rectified tanh
Essentially max(0, tanh(x))
Underlying implementation is in native code
f(x) = alpha (exp(x) - 1.0); x < 0 = x ; x>= 0
alpha defaults to 1, if not specified
f(x) = max(0, x)
Rational tanh approximation From https://arxiv.org/pdf/1508.01292v3
f(x) = 1.7159 tanh(2x/3) where tanh is approximated as follows, tanh(y) ~ sgn(y) { 1 - 1/(1+|y|+y^2+1.41645y^4)}
Underlying implementation is in native code
Thresholded RELU
f(x) = x for x > theta, f(x) = 0 otherwise. theta defaults to 1.0
f(x) = min(max(input, cutoff), 6)
f(x) = 1 / (1 + exp(-x))
GELU activation function - Gaussian Error Linear Units
/ Parametrized Rectified Linear Unit (PReLU)
f(x) = alpha x for x < 0, f(x) = x for x >= 0
alpha has the same shape as x and is a learned parameter.
f(x) = x
f_i(x) = x_i / (1+
x_i
)
f(x) = min(1, max(0, 0.2x + 0.5))
f_i(x) = exp(x_i - shift) / sum_j exp(x_j - shift) where shift = max_i(x_i)
f(x) = x^3
f(x) = max(0,x) + alpha min(0, x)
alpha is drawn from uniform(l,u) during training and is set to l+u/2 during test l and u default to 1/8 and 1/3 respectively
Empirical Evaluation of Rectified Activations in Convolutional Network
f(x) = (exp(x) - exp(-x)) / (exp(x) + exp(-x))
https://arxiv.org/pdf/1706.02515.pdf
Leaky RELU f(x) = max(0, x) + alpha min(0, x) alpha defaults to 0.01
f(x) = x sigmoid(x)
f(x) = log(1+e^x)
AlexNet
Dl4j’s AlexNet model interpretation based on the original paper ImageNet Classification with Deep Convolutional Neural Networks and the imagenetExample code referenced. References: http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf https://github.com/BVLC/caffe/blob/master/models/bvlc_alexnet/train_val.prototxt
Model is built in dl4j based on available functionality and notes indicate where there are gaps waiting for enhancements.
Bias initialization in the paper is 1 in certain layers but 0.1 in the imagenetExample code Weight distribution uses 0.1 std for all layers in the paper but 0.005 in the dense layers in the imagenetExample code
Darknet19 Reference: https://arxiv.org/pdf/1612.08242.pdf ImageNet weights for this model are available and have been converted from https://pjreddie.com/darknet/imagenet/ using https://github.com/allanzelener/YAD2K .
There are 2 pretrained models, one for 224x224 images and one fine-tuned for 448x448 images. Call setInputShape() with either {3, 224, 224} or {3, 448, 448} before initialization. The channels of the input images need to be in RGB order (not BGR), with values normalized within [0, 1]. The output labels are as per https://github.com/pjreddie/darknet/blob/master/data/imagenet.shortnames.list .
A variant of the original FaceNet model that relies on embeddings and triplet loss. Reference: https://arxiv.org/abs/1503.03832 Also based on the OpenFace implementation: http://reports-archive.adm.cs.cmu.edu/anon/2016/CMU-CS-16-118.pdf
A variant of the original FaceNet model that relies on embeddings and triplet loss. Reference: https://arxiv.org/abs/1503.03832 Also based on the OpenFace implementation: http://reports-archive.adm.cs.cmu.edu/anon/2016/CMU-CS-16-118.pdf
LeNet was an early promising achiever on the ImageNet dataset. References:
MNIST weights for this model are available and have been converted from https://github.com/f00-/mnist-lenet-keras.
Implementation of NASNet-A in Deeplearning4j. NASNet refers to Neural Architecture Search Network, a family of models that were designed automatically by learning the model architectures directly on the dataset of interest.
This implementation uses 1056 penultimate filters and an input shape of (3, 224, 224). You can change this.
Paper: https://arxiv.org/abs/1707.07012 ImageNet weights for this model are available and have been converted from https://keras.io/applications/.
Residual networks for deep learning.
Paper: https://arxiv.org/abs/1512.03385 ImageNet weights for this model are available and have been converted from https://keras.io/applications/</a>.
A simple convolutional network for generic image classification. Reference: https://github.com/oarriaga/face_classification/
U-Net
An implementation of SqueezeNet. Touts similar accuracy to AlexNet with a fraction of the parameters.
Paper: https://arxiv.org/abs/1602.07360 ImageNet weights for this model are available and have been converted from https://github.com/rcmalli/keras-squeezenet/.
LSTM designed for text generation. Can be trained on a corpus of text. For this model, numClasses is
Architecture follows this implementation: https://github.com/fchollet/keras/blob/master/examples/lstm_text_generation.py
Walt Whitman weights are available for generating text from his works, adapted from https://github.com/craigomac/InfiniteMonkeys.
Tiny YOLO Reference: https://arxiv.org/pdf/1612.08242.pdf
ImageNet+VOC weights for this model are available and have been converted from https://pjreddie.com/darknet/yolo using https://github.com/allanzelener/YAD2K and the following code.
String filename = “tiny-yolo-voc.h5”; ComputationGraph graph = KerasModelImport.importKerasModelAndWeights(filename, false); INDArray priors = Nd4j.create(priorBoxes);
FineTuneConfiguration fineTuneConf = new FineTuneConfiguration.Builder() .seed(seed) .iterations(iterations) .optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT) .gradientNormalization(GradientNormalization.RenormalizeL2PerLayer) .gradientNormalizationThreshold(1.0) .updater(new Adam.Builder().learningRate(1e-3).build()) .l2(0.00001) .activation(Activation.IDENTITY) .trainingWorkspaceMode(workspaceMode) .inferenceWorkspaceMode(workspaceMode) .build();
ComputationGraph model = new TransferLearning.GraphBuilder(graph) .fineTuneConfiguration(fineTuneConf) .addLayer(“outputs”, new Yolo2OutputLayer.Builder() .boundingBoxPriors(priors) .build(), “conv2d_9”) .setOutputs(“outputs”) .build();
System.out.println(model.summary(InputType.convolutional(416, 416, 3)));
ModelSerializer.writeModel(model, “tiny-yolo-voc_dl4j_inference.v1.zip”, false); }</pre>
The channels of the 416x416 input images need to be in RGB order (not BGR), with values normalized within [0, 1].
U-Net
An implementation of U-Net, a deep learning network for image segmentation in Deeplearning4j. The u-net is convolutional network architecture for fast and precise segmentation of images. Up to now it has outperformed the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks.
Paper: https://arxiv.org/abs/1505.04597 Weights are available for image segmentation trained on a synthetic dataset
VGG-16, from Very Deep Convolutional Networks for Large-Scale Image Recognition https://arxiv.org/abs/1409.1556
Deep Face Recognition http://www.robots.ox.ac.uk/~vgg/publications/2015/Parkhi15/parkhi15.pdf
ImageNet weights for this model are available and have been converted from https://github.com/fchollet/keras/tree/1.1.2/keras/applications. CIFAR-10 weights for this model are available and have been converted using “approach 2” from https://github.com/rajatvikramsingh/cifar10-vgg16. VGGFace weights for this model are available and have been converted from https://github.com/rcmalli/keras-vggface.
VGG-19, from Very Deep Convolutional Networks for Large-Scale Image Recognition https://arxiv.org/abs/1409.1556 ImageNet weights for this model are available and have been converted from https://github.com/fchollet/keras/tree/1.1.2/keras/applications.
U-Net
An implementation of Xception in Deeplearning4j. A novel deep convolutional neural network architecture inspired by Inception, where Inception modules have been replaced with depthwise separable convolutions.
Paper: https://arxiv.org/abs/1610.02357 ImageNet weights for this model are available and have been converted from https://keras.io/applications/.
YOLOv2 Reference: https://arxiv.org/pdf/1612.08242.pdf
ImageNet+COCO weights for this model are available and have been converted from https://pjreddie.com/darknet/yolo using https://github.com/allanzelener/YAD2K and the following code.
The channels of the 608x608 input images need to be in RGB order (not BGR), with values normalized within [0, 1].
pretrainedUrl
Default prior boxes for the model
Autoencoders are neural networks for unsupervised learning. Eclipse Deeplearning4j supports certain autoencoder layers such as variational autoencoders.
RBMs are no longer supported as of version 0.9.x. They are no longer best-in-class for most machine learning problems.
Autoencoder layer. Adds noise to input and learn a reconstruction function.
corruptionLevel
Level of corruption - 0.0 (none) to 1.0 (all values corrupted)
sparsity
Autoencoder sparity parameter
param sparsity Sparsity
Variational Autoencoder layer
This implementation allows multiple encoder and decoder layers, the number and sizes of which can be set independently.
A note on scores during pretraining: This implementation minimizes the negative of the variational lower bound objective as described in Kingma & Welling; the mathematics in that paper is based on maximization of the variational lower bound instead. Thus, scores reported during pretraining in DL4J are the negative of the variational lower bound equation in the paper. The backpropagation and learning procedure is otherwise as described there.
encoderLayerSizes
Size of the encoder layers, in units. Each encoder layer is functionally equivalent to a {- link org.deeplearning4j.nn.conf.layers.DenseLayer}. Typically the number and size of the decoder layers (set via {- link #decoderLayerSizes(int…)} is similar to the encoder layers.
setEncoderLayerSizes
Size of the encoder layers, in units. Each encoder layer is functionally equivalent to a {- link org.deeplearning4j.nn.conf.layers.DenseLayer}. Typically the number and size of the decoder layers (set via {- link #decoderLayerSizes(int…)} is similar to the encoder layers.
param encoderLayerSizes Size of each encoder layer in the variational autoencoder
decoderLayerSizes
Size of the decoder layers, in units. Each decoder layer is functionally equivalent to a {- link org.deeplearning4j.nn.conf.layers.DenseLayer}. Typically the number and size of the decoder layers is similar to the encoder layers (set via {- link #encoderLayerSizes(int…)}.
param decoderLayerSizes Size of each deccoder layer in the variational autoencoder
setDecoderLayerSizes
Size of the decoder layers, in units. Each decoder layer is functionally equivalent to a {- link org.deeplearning4j.nn.conf.layers.DenseLayer}. Typically the number and size of the decoder layers is similar to the encoder layers (set via {- link #encoderLayerSizes(int…)}.
param decoderLayerSizes Size of each deccoder layer in the variational autoencoder
reconstructionDistribution
The reconstruction distribution for the data given the hidden state - i.e., P(data|Z). This should be selected carefully based on the type of data being modelled. For example:
{- link GaussianReconstructionDistribution} + {identity or tanh} for real-valued (Gaussian) data
{- link BernoulliReconstructionDistribution} + sigmoid for binary-valued (0 or 1) data
param distribution Reconstruction distribution
lossFunction
Configure the VAE to use the specified loss function for the reconstruction, instead of a ReconstructionDistribution. Note that this is NOT following the standard VAE design (as per Kingma & Welling), which assumes a probabilistic output - i.e., some p(x|z). It is however a valid network configuration, allowing for optimization of more traditional objectives such as mean squared error. Note: clearly, setting the loss function here will override any previously set recontruction distribution
param outputActivationFn Activation function for the output/reconstruction
param lossFunction Loss function to use
lossFunction
Configure the VAE to use the specified loss function for the reconstruction, instead of a ReconstructionDistribution. Note that this is NOT following the standard VAE design (as per Kingma & Welling), which assumes a probabilistic output - i.e., some p(x|z). It is however a valid network configuration, allowing for optimization of more traditional objectives such as mean squared error. Note: clearly, setting the loss function here will override any previously set recontruction distribution
param outputActivationFn Activation function for the output/reconstruction
param lossFunction Loss function to use
lossFunction
Configure the VAE to use the specified loss function for the reconstruction, instead of a ReconstructionDistribution. Note that this is NOT following the standard VAE design (as per Kingma & Welling), which assumes a probabilistic output - i.e., some p(x|z). It is however a valid network configuration, allowing for optimization of more traditional objectives such as mean squared error. Note: clearly, setting the loss function here will override any previously set recontruction distribution
param outputActivationFn Activation function for the output/reconstruction
param lossFunction Loss function to use
pzxActivationFn
Activation function for the input to P(z|data). Care should be taken with this, as some activation functions (relu, etc) are not suitable due to being bounded in range [0,infinity).
param activationFunction Activation function for p(z| x)
pzxActivationFunction
Activation function for the input to P(z|data). Care should be taken with this, as some activation functions (relu, etc) are not suitable due to being bounded in range [0,infinity).
param activation Activation function for p(z | x)
nOut
Set the size of the VAE state Z. This is the output size during standard forward pass, and the size of the distribution P(Z|data) during pretraining.
param nOut Size of P(Z | data) and output size
numSamples
Set the number of samples per data point (from VAE state Z) used when doing pretraining. Default value: 1.
This is parameter L from Kingma and Welling: “In our experiments we found that the number of samples L per datapoint can be set to 1 as long as the minibatch size M was large enough, e.g. M = 100.”
param numSamples Number of samples per data point for pretraining
See: Kingma & Welling, 2013: Auto-Encoding Variational Bayes -