Instructions to build all DL4J libraries from source.
Note: This guide is outdated. We are working on updating it as soon as possible.
NOTE: MOST USERS SHOULD USE THE RELEASES ON MAVEN CENTRAL AS PER THE QUICK START GUIDE, AND NOT BUILD FROM SOURCE
Unless you have a very good reason to build from source (such as developing new features - excluding custom layers, custom activation functions, custom loss functions, etc - all of which can be added without modifying DL4J directly) then you shouldn't build from source. Building from source can be quite complex, with no benefit in a lot of cases.
For those developers and engineers who prefer to use the most up-to-date version of Deeplearning4j or fork and build their own version, these instructions will walk you through building and installing Deeplearning4j. The preferred installation destination is to your machine's local maven repository. If you are not using the master branch, you can modify these steps as needed (i.e.: switching GIT branches and modifying the build-dl4j-stack.sh script).
Building locally requires that you build the entire Deeplearning4j stack which includes:
Note that Deeplearning4j is designed to work on most platforms (Windows, OS X, and Linux) and is also includes multiple "flavors" depending on the computing architecture you choose to utilize. This includes CPU (OpenBLAS, MKL, ATLAS) and GPU (CUDA). The DL4J stack also supports x86 and PowerPC architectures.
Your local machine will require some essential software and environment variables set before you try to build and install the DL4J stack. Depending on your platform and the version of your operating system, the instructions may vary in getting them to work. This software includes:
git
cmake (3.2 or higher)
OpenMP
gcc (4.9 or higher)
maven (3.3 or higher)
Architecture-specific software includes:
CPU options:
Intel MKL
OpenBLAS
ATLAS
GPU options:
CUDA
IDE-specific requirements:
IntelliJ Lombok plugin
DL4J testing dependencies:
dl4j-test-resources
Ubuntu Assuming you are using Ubuntu as your flavor of Linux and you are running as a non-root user, follow these steps to install prerequisite software:
Homebrew is the accepted method of installing prerequisite software. Assuming you have Homebrew installed locally, follow these steps to install your necessary tools.
First, before using Homebrew we need to ensure an up-to-date version of Xcode is installed (it is used as a primary compiler):
Finally, install prerequisite tools:
Note: You can not use clang. You also can not use a new version of gcc. If you have a newer version of gcc, please switch versions with this link
libnd4j depends on some Unix utilities for compilation. So in order to compile it you will need to install Msys2.
After you have setup Msys2 by following their instructions, you will have to install some additional development packages. Start the msys2 shell and setup the dev environment with:
This will install the needed dependencies for use in the msys2 shell.
You will also need to setup your PATH environment variable to include C:\msys64\mingw64\bin (or where ever you have decided to install msys2). If you have IntelliJ (or another IDE) open, you will have to restart it before this change takes effect for applications started through them. If you don't, you probably will see a "Can't find dependent libraries" error.
Once you have installed the prerequisite tools, you can now install the required architectures for your platform.
Of all the existing architectures available for CPU, Intel MKL is currently the fastest. However, it requires some "overhead" before you actually install it.
Apply for a license at Intel's site
After a few steps through Intel, you will receive a download link
Download and install Intel MKL using the setup guide
Linux
Ubuntu Assuming you are using Ubuntu, you can install OpenBLAS via:
You will also need to ensure that /opt/OpenBLAS/lib (or any other home directory for OpenBLAS) is on your PATH. In order to get OpenBLAS to work with Apache Spark, you will also need to do the following:
CentOS Enter the following in your terminal (or ssh session) as a root user:
After that, you should see a lot of activity and installs on the terminal. To verify that you have, for example, gcc, enter this line:
For more complete instructions, go here.
OS X
You can install OpenBLAS on OS X with Homebrew:
Windows
An OpenBLAS package is available for msys2. You can install it using the pacman command.
Linux
Ubuntu An apt package is available for ATLAS on Ubuntu:
CentOS You can install ATLAS on CentOS using:
OS X
Installing ATLAS on OS X is a somewhat complicated and lengthy process. However, the following commands will work on most machines:
Linux & OS X
Detailed instructions for installing GPU architectures such as CUDA can be found here.
Windows
The CUDA Backend has some additional requirements before it can be built:
Visual Studio 2012 or 2013 (Please note: Visual Studio 2015 is NOT SUPPORTED by CUDA 7.5 and below)
In order to build the CUDA backend you will have to setup some more environment variables first, by calling vcvars64.bat. But first, set the system environment variable SET_FULL_PATH to true, so all of the variables that vcvars64.bat sets up, are passed to the mingw shell.
Inside a normal cmd.exe command prompt, run C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\amd64\vcvars64.bat
Run c:\msys64\mingw64_shell.bat inside that
Change to your libnd4j folder
./buildnativeoperations.sh -c cuda
This builds the CUDA nd4j.dll.
If you are building Deeplearning4j through an IDE such as IntelliJ, you will need to install certain plugins to ensure your IDE renders code highlighting appropriately. You will need to install a plugin for Lombok:
IntelliJ Lombok Plugin: https://plugins.jetbrains.com/plugin/6317-lombok-plugin
Eclipse Lombok Plugin: Follow instructions at https://projectlombok.org/download.html
If you want to work on ScalNet, the Scala API, or on certain modules such as the DL4J UI, you will need to ensure your IDE has Scala support installed and available to you.
Deeplearning4j uses a separate repository that contains all resources necessary for testing. This is to keep the central DL4J repository lightweight and avoid large blobs in the GIT history. To run the tests you need to install the test-resources from https://github.com/KonduitAI/dl4j-test-resources (~10gb). If you don't care about history, do a shallow clone only with
Tests will run only when testresources and a backend profile (such as test-nd4j-native) are selected
Running the tests will take a while. To run tests of just a single maven module you can add a module constraint with -pl deeplearning4j-core (for details see here)
Before running the DL4J stack build script, you must ensure certain environment variables are defined before running your build. These are outlined below depending on your architecture.
You will need to know the exact path of the directory where you are running the DL4J build script (you are encouraged to use a clean empty directory). Otherwise, your build will fail. Once you determine this path, add /libnd4j to the end of that path and export it to your local environment. This will look like:
You can link with MKL either at build time, or at runtime with binaries initially linked with another BLAS implementation such as OpenBLAS. To build against MKL, simply add the path containing libmkl_rt.so (or mkl_rt.dll on Windows), say /path/to/intel64/lib/, to the LD_LIBRARY_PATH environment variable on Linux (or PATH on Windows) and build like before. On Linux though, to make sure it uses the correct version of OpenMP, we also might need to set these environment variables:
When libnd4j cannot be rebuilt, we can use the MKL libraries after the facts and get them loaded instead of OpenBLAS at runtime, but things are a bit trickier. Please additionally follow the instructions below.
Make sure that files such as /lib64/libopenblas.so.0 and /lib64/libblas.so.3 are not available (or appear after in the PATH on Windows), or they will get loaded by libnd4j by their absolute paths, before anything else.
Inside /path/to/intel64/lib/, create a symbolic link or copy of libmkl_rt.so (or mkl_rt.dll on Windows) to the name that libnd4j expect to load, for example:
Finally, add /path/to/intel64/lib/ to the LD_LIBRARY_PATH environment variable (or early in the PATH on Windows) and run your Java application as usual.
If you prefer, you can build each piece in the DL4J stack by hand. The procedure for each piece of software is essentially:
Git clone
Build
Install
The overall procedure looks like the following commands below, with the exception that libnd4j's ./buildnativeoperations.sh accepts parameters based on the backend you are building for. You need to follow these instructions in the order they're given. If you don't, you'll run into errors. The GPU-specific instructions below have been commented out, but should be substituted for the CPU-specific commands when building for a GPU backend.
Once you've installed the DL4J stack to your local maven repository, you can now include it in your build tool's dependencies. Follow the typical Getting Started instructions for Deeplearning4j, and appropriately replace versions with the SNAPSHOT version currently on the master POM.
Note that some build tools such as Gradle and SBT don't properly pull in platform-specific binaries. You can follow instructions here for setting up your favorite build tool.
If you encounter issues while building locally, please reach out on our discourse for help.