In this tutorial we will use a LSTM neural network to predict instacart users’ purchasing behavior given a history of their past orders. The data originially comes from a Kaggle challenge (kaggle.com/c/instacart-market-basket-analysis). We first removed users that only made 1 order using the instacart app and then took 5000 users out of the remaining to be part of the data for this tutorial.
For each order, we have information on the product the user purchased. For example, there is information on the product name, what aisle it is found in, and the department it falls under. To construct features, we extracted indicators representing whether or not a user purchased a product in the given aisles for each order. In total there are 134 aisles. The targets were whether or not a user will buy a product in the breakfast department in the next order. We also used auxiliary targets to train this LSTM. The auxiliary targets were whether or not a user will buy a product in the dairy department in the next order.
We suspect that a LSTM will be effective for this task, because of the temporal dependencies in the data.
To download the data, we will create a temporary directory that will store the data files, extract the tar.gz file from the url, and place it in the specified directory.
val DATA_URL ="https://dl4jdata.blob.core.windows.net/training/tutorials/instacart.tar.gz"val DATA_PATH = FilenameUtils.concat(System.getProperty("java.io.tmpdir"), "dl4j_instacart/")
We will then extract the data from the tar.gz file, recreate directories within the tar.gz file into our temporary directories, and copy the files from the tar.gz file.
Next we will convert the raw data (csv files) into DataSetIterators, which will be fed into a neural network. Our training data will have 4000 examples which will be represented by a single DataSetIterator, and the testing data will have 1000 examples which will be represented by a separate DataSetIterator.
val path = FilenameUtils.concat(DATA_PATH, "instacart/") // set parent directoryval featureBaseDir = FilenameUtils.concat(path, "features") // set feature directoryval targetsBaseDir = FilenameUtils.concat(path, "breakfast") // set label directoryval auxilBaseDir = FilenameUtils.concat(path, "dairy") // set futures directory
We first initialize CSVSequenceRecordReaders, which will parse the raw data into record-like format. Because we will be using multitask learning, we will use two outputs. Thus we need three RecordReaders in total: one for the input, another for the first target, and the last for the second target. Next, we will need the RecordreaderMultiDataSetIterator, since we now have two outputs. We can add our SequenceRecordReaders using the addSequenceReader methods and specify the input and both outputs. The ALIGN_END alignment mode is used, since the sequences for each example vary in length.
We will create DataSetIterators for both the training data and the test data.
val testFeatures =new CSVSequenceRecordReader(1, ",");testFeatures.initialize( new NumberedFileInputSplit(featureBaseDir +"/%d.csv", 4001, 5000));val testBreakfast =new CSVSequenceRecordReader(1, ",");testBreakfast.initialize( new NumberedFileInputSplit(targetsBaseDir +"/%d.csv", 4001, 5000));val testDairy =new CSVSequenceRecordReader(1, ",");testDairy.initialize(new NumberedFileInputSplit(auxilBaseDir +"/%d.csv", 4001, 5000));val test =new RecordReaderMultiDataSetIterator.Builder(20) .addSequenceReader("rr1", testFeatures).addInput("rr1") .addSequenceReader("rr2",testBreakfast).addOutput("rr2") .addSequenceReader("rr3",testDairy).addOutput("rr3") .sequenceAlignmentMode(RecordReaderMultiDataSetIterator.AlignmentMode.ALIGN_END) .build();
Neural Network
The next task is to set up the neural network configuration. We see below that the ComputationGraph class is used to create a LSTM with two outputs. We can set the outputs using the setOutputs method of the NeuralNetConfiguraitonBuilder. One GravesLSTM layer and two RnnOutputLayers will be used. We will also set other hyperparameters of the model, such as dropout, weight initialization, updaters, and activation functions.
val conf =new NeuralNetConfiguration.Builder() .optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT) .seed(12345) .weightInit(WeightInit.XAVIER) .dropOut(0.25) .updater(new Adam()) .graphBuilder() .addInputs("input") .addLayer("L1", new LSTM.Builder() .nIn(134).nOut(150) .gradientNormalization(GradientNormalization.ClipElementWiseAbsoluteValue) .gradientNormalizationThreshold(10) .activation(Activation.TANH) .build(), "input") .addLayer("out1", new RnnOutputLayer.Builder(LossFunction.XENT) .gradientNormalization(GradientNormalization.ClipElementWiseAbsoluteValue) .gradientNormalizationThreshold(10) .activation(Activation.SIGMOID) .nIn(150).nOut(1).build(), "L1") .addLayer("out2", new RnnOutputLayer.Builder(LossFunction.XENT) .gradientNormalization(GradientNormalization.ClipElementWiseAbsoluteValue) .gradientNormalizationThreshold(10) .activation(Activation.SIGMOID) .nIn(150).nOut(1).build(), "L1") .setOutputs("out1","out2") .build();
We must then initialize the neural network.
val net =new ComputationGraph(conf);net.init();
Model Training
To train the model, we use 5 epochs with a simple call to the fit method of the ComputationGraph.
net.fit( train , 5);
Model Evaluation
We will now evaluate our trained model on the original task, which was predicting whether or not a user will purchase a product in the breakfast department. Note that we will use the area under the curve (AUC) metric of the ROC curve.
// Evaluate modelval roc =new ROC();test.reset();while(test.hasNext()){val next = test.next();val features = next.getFeatures();val output = net.output(features(0)); roc.evalTimeSeries(next.getLabels()(0), output(0));}println(roc.calculateAUC());
