Operations, such as a Function, help execute transforms and load data into DataVec. The concept of operations is low-level, meaning that most of the time you will not need to worry about them.
Loading data into Spark
If you're using Apache Spark, functions will iterate over the dataset and load it into a Spark RDD and convert the raw data format into a Writable.
The above code loads a CSV file into a 2D java RDD. Once your RDD is loaded, you can transform it, perform joins and use reducers to wrangle the data any way you want.
CalculateSortedRank: calculate the rank of each example, after sorting example. For example, we might have some numerical “score” column, and we want to know for the rank (sort order) for each example, according to that column.
The rank of each example (after sorting) will be added in a new Long column. Indexing is done from 0; examples will have values 0 to dataSetSize - 1.
Currently, CalculateSortedRank can only be applied on standard (i.e., non-sequence) data. Furthermore, the current implementation can only sort on one column
transform
public Schema transform(Schema inputSchema)
param newColumnName Name of the new column (will contain the rank for each example)
param sortOnColumn Name of the column to sort on
param comparator Comparator used to sort examples
outputColumnName
public String outputColumnName()
The output column name after the operation has been applied
return the output column name
columnName
public String columnName()
The output column names This will often be the same as the input