Schema

Why use schemas?

The unfortunate reality is that data is dirty. When trying to vecotrize a dataset for deep learning, it is quite rare to find files that have zero errors. Schema is important for maintaining the meaning of the data before using it for something like training a neural network.

Using schemas

Schemas are primarily used for programming transformations. Before you can properly execute a TransformProcess you will need to pass the schema of the data being transformed.

An example of a schema for merchant records may look like:

Schema inputDataSchema = new Schema.Builder()
    .addColumnsString("DateTimeString", "CustomerID", "MerchantID")
    .addColumnInteger("NumItemsInTransaction")
    .addColumnCategorical("MerchantCountryCode", Arrays.asList("USA","CAN","FR","MX"))
    .addColumnDouble("TransactionAmountUSD",0.0,null,false,false)   //$0.0 or more, no maximum limit, no NaN and no Infinite values
    .addColumnCategorical("FraudLabel", Arrays.asList("Fraud","Legit"))
    .build();

Joining schemas

If you have two different datasets that you want to merge together, DataVec provides a Join class with different join strategies such as Inner or RightOuter.

Schema customerInfoSchema = new Schema.Builder()
    .addColumnLong("customerID")
    .addColumnString("customerName")
    .addColumnCategorical("customerCountry", Arrays.asList("USA","France","Japan","UK"))
    .build();

Schema customerPurchasesSchema = new Schema.Builder()
    .addColumnLong("customerID")
    .addColumnTime("purchaseTimestamp", DateTimeZone.UTC)
    .addColumnLong("productID")
    .addColumnInteger("purchaseQty")
    .addColumnDouble("unitPriceUSD")
    .build();

Join join = new Join.Builder(Join.JoinType.Inner)
    .setJoinColumns("customerID")
    .setSchemas(customerInfoSchema, customerPurchasesSchema)
    .build();

Once you've defined your join and you've loaded the data into DataVec, you must use an Executor to complete the join.

Classes and utilities

DataVec comes with a few Schema classes and helper utilities for 2D and sequence types of data.

Join

[source]

Join class: used to specify a join (like an SQL join)

setSchemas

public Builder setSchemas(Schema left, Schema right) 

Type of join Inner: Return examples where the join column values occur in both LeftOuter: Return all examples from left data, whether there is a matching right value or not. (If not: right values will have NullWritable instead) RightOuter: Return all examples from the right data, whether there is a matching left value or not. (If not: left values will have NullWritable instead) FullOuter: return all examples from both left/right, whether there is a matching value from the other side or not. (If not: other values will have NullWritable instead)

setKeyColumns

public Builder setKeyColumns(String... keyColumnNames) 

• deprecated Use {- link #setJoinColumns(String…)}

setKeyColumnsLeft

public Builder setKeyColumnsLeft(String... keyColumnNames) 

• deprecated Use {- link #setJoinColumnsLeft(String…)}

setKeyColumnsRight

public Builder setKeyColumnsRight(String... keyColumnNames) 

• deprecated Use {- link #setJoinColumnsRight(String…)}

setJoinColumnsLeft

public Builder setJoinColumnsLeft(String... joinColumnNames) 

Specify the names of the columns to join on, for the left data) The idea: join examples where firstDataValues(joinColumNamesLeft[i]) == secondDataValues(joinColumnNamesRight[i]) for all i

• param joinColumnNames Names of the columns to join on (for left data)

setJoinColumnsRight

public Builder setJoinColumnsRight(String... joinColumnNames) 

Specify the names of the columns to join on, for the right data) The idea: join examples where firstDataValues(joinColumNamesLeft[i]) == secondDataValues(joinColumnNamesRight[i]) for all i

• param joinColumnNames Names of the columns to join on (for left data)

InferredSchema

[source]

If passed a CSV file that contains a header and a single row of sample data, it will return a Schema.

Only Double, Integer, Long, and String types are supported. If no number type can be inferred, the field type will become the default type. Note that if your column is actually categorical but is represented as a number, you will need to do additional transformation. Also, if your sample field is blank/null, it will also become the default type.

Schema

[source]

A Schema defines the layout of tabular data. Specifically, it contains names f or each column, as well as details of types (Integer, String, Long, Double, etc). Type information for each column may optionally include restrictions on the allowable values for each column.

sameTypes

public boolean sameTypes(Schema schema) 

Create a schema based on the given metadata

• param columnMetaData the metadata to create the schema from

newSchema

public Schema newSchema(List<ColumnMetaData> columnMetaData) 

Compute the difference in {- link ColumnMetaData} between this schema and the passed in schema. This is useful during the {- link org.datavec.api.transform.TransformProcess} to identify what a process will do to a given {- link Schema}.

• param schema the schema to compute the difference for

• return the metadata that is different (in order) between this schema and the other schema

numColumns

public int numColumns() 

Returns the number of columns or fields for this schema

• return the number of columns or fields for this schema

getName

public String getName(int column) 

Returns the name of a given column at the specified index

• param column the index of the column to get the name for

• return the name of the column at the specified index

getType

public ColumnType getType(int column) 

Returns the {- link ColumnType} for the column at the specified index

• param column the index of the column to get the type for

• return the type of the column to at the specified inde

getType

public ColumnType getType(String columnName) 

Returns the {- link ColumnType} for the column at the specified index

• param columnName the index of the column to get the type for

• return the type of the column to at the specified inde

getMetaData

public ColumnMetaData getMetaData(int column) 

Returns the {- link ColumnMetaData} at the specified column index

• param column the index to get the metadata for

• return the metadata at ths specified index

getMetaData

public ColumnMetaData getMetaData(String column) 

Retrieve the metadata for the given column name

• param column the name of the column to get metadata for

• return the metadata for the given column name

getIndexOfColumn

public int getIndexOfColumn(String columnName) 

Return a copy of the list column names

• return a copy of the list of column names for this schema

hasColumn

public boolean hasColumn(String columnName) 

Return the indices of the columns, given their namess

• param columnNames Name of the columns to get indices for

• return Column indexes

toJson

public String toJson() 

Serialize this schema to json

• return a json representation of this schema

toYaml

public String toYaml() 

Serialize this schema to yaml

• return the yaml representation of this schema

fromJson

public static Schema fromJson(String json) 

Create a schema from a given json string

• param json the json to create the schema from

• return the created schema based on the json

fromYaml

public static Schema fromYaml(String yaml) 

Create a schema from the given yaml string

• param yaml the yaml to create the schema from

• return the created schema based on the yaml

addColumnFloat

public Builder addColumnFloat(String name) 

Add a Float column with no restrictions on the allowable values, except for no NaN/infinite values allowed

• param name Name of the column

addColumnFloat

public Builder addColumnFloat(String name, Float minAllowedValue, Float maxAllowedValue) 

Add a Float column with the specified restrictions (and no NaN/Infinite values allowed)

• param name Name of the column

• param minAllowedValue Minimum allowed value (inclusive). If null: no restriction

• param maxAllowedValue Maximum allowed value (inclusive). If null: no restriction

• return

addColumnFloat

public Builder addColumnFloat(String name, Float minAllowedValue, Float maxAllowedValue, boolean allowNaN,
                                       boolean allowInfinite) 

Add a Float column with the specified restrictions

• param name Name of the column

• param minAllowedValue Minimum allowed value (inclusive). If null: no restriction

• param maxAllowedValue Maximum allowed value (inclusive). If null: no restriction

• param allowNaN If false: don’t allow NaN values. If true: allow.

• param allowInfinite If false: don’t allow infinite values. If true: allow

addColumnsFloat

public Builder addColumnsFloat(String... columnNames) 

Add multiple Float columns with no restrictions on the allowable values of the columns (other than no NaN/Infinite)

• param columnNames Names of the columns to add

addColumnsFloat

public Builder addColumnsFloat(String pattern, int minIdxInclusive, int maxIdxInclusive) 

A convenience method for adding multiple Float columns. For example, to add columns “myFloatCol_0”, “myFloatCol_1”, “myFloatCol_2”, use {- code addColumnsFloat(“myFloatCol_%d”,0,2)}

• param pattern Pattern to use (via String.format). “%d” is replaced with column numbers

• param minIdxInclusive Minimum column index to use (inclusive)

• param maxIdxInclusive Maximum column index to use (inclusive)

addColumnsFloat

public Builder addColumnsFloat(String pattern, int minIdxInclusive, int maxIdxInclusive,
                                        Float minAllowedValue, Float maxAllowedValue, boolean allowNaN, boolean allowInfinite) 

A convenience method for adding multiple Float columns, with additional restrictions that apply to all columns For example, to add columns “myFloatCol_0”, “myFloatCol_1”, “myFloatCol_2”, use {- code addColumnsFloat(“myFloatCol_%d”,0,2,null,null,false,false)}

• param pattern Pattern to use (via String.format). “%d” is replaced with column numbers

• param minIdxInclusive Minimum column index to use (inclusive)

• param maxIdxInclusive Maximum column index to use (inclusive)

• param minAllowedValue Minimum allowed value (inclusive). If null: no restriction

• param maxAllowedValue Maximum allowed value (inclusive). If null: no restriction

• param allowNaN If false: don’t allow NaN values. If true: allow.

• param allowInfinite If false: don’t allow infinite values. If true: allow

addColumnDouble

public Builder addColumnDouble(String name) 

Add a Double column with no restrictions on the allowable values, except for no NaN/infinite values allowed

• param name Name of the column

addColumnDouble

public Builder addColumnDouble(String name, Double minAllowedValue, Double maxAllowedValue) 

Add a Double column with the specified restrictions (and no NaN/Infinite values allowed)

• param name Name of the column

• param minAllowedValue Minimum allowed value (inclusive). If null: no restriction

• param maxAllowedValue Maximum allowed value (inclusive). If null: no restriction

• return

addColumnDouble

public Builder addColumnDouble(String name, Double minAllowedValue, Double maxAllowedValue, boolean allowNaN,
                        boolean allowInfinite) 

Add a Double column with the specified restrictions

• param name Name of the column

• param minAllowedValue Minimum allowed value (inclusive). If null: no restriction

• param maxAllowedValue Maximum allowed value (inclusive). If null: no restriction

• param allowNaN If false: don’t allow NaN values. If true: allow.

• param allowInfinite If false: don’t allow infinite values. If true: allow

addColumnsDouble

public Builder addColumnsDouble(String... columnNames) 

Add multiple Double columns with no restrictions on the allowable values of the columns (other than no NaN/Infinite)

• param columnNames Names of the columns to add

addColumnsDouble

public Builder addColumnsDouble(String pattern, int minIdxInclusive, int maxIdxInclusive) 

A convenience method for adding multiple Double columns. For example, to add columns “myDoubleCol_0”, “myDoubleCol_1”, “myDoubleCol_2”, use {- code addColumnsDouble(“myDoubleCol_%d”,0,2)}

• param pattern Pattern to use (via String.format). “%d” is replaced with column numbers

• param minIdxInclusive Minimum column index to use (inclusive)

• param maxIdxInclusive Maximum column index to use (inclusive)

addColumnsDouble

public Builder addColumnsDouble(String pattern, int minIdxInclusive, int maxIdxInclusive,
                        Double minAllowedValue, Double maxAllowedValue, boolean allowNaN, boolean allowInfinite) 

A convenience method for adding multiple Double columns, with additional restrictions that apply to all columns For example, to add columns “myDoubleCol_0”, “myDoubleCol_1”, “myDoubleCol_2”, use {- code addColumnsDouble(“myDoubleCol_%d”,0,2,null,null,false,false)}

• param pattern Pattern to use (via String.format). “%d” is replaced with column numbers

• param minIdxInclusive Minimum column index to use (inclusive)

• param maxIdxInclusive Maximum column index to use (inclusive)

• param minAllowedValue Minimum allowed value (inclusive). If null: no restriction

• param maxAllowedValue Maximum allowed value (inclusive). If null: no restriction

• param allowNaN If false: don’t allow NaN values. If true: allow.

• param allowInfinite If false: don’t allow infinite values. If true: allow

addColumnInteger

public Builder addColumnInteger(String name) 

Add an Integer column with no restrictions on the allowable values

• param name Name of the column

addColumnInteger

public Builder addColumnInteger(String name, Integer minAllowedValue, Integer maxAllowedValue) 

Add an Integer column with the specified min/max allowable values

• param name Name of the column

• param minAllowedValue Minimum allowed value (inclusive). If null: no restriction

• param maxAllowedValue Maximum allowed value (inclusive). If null: no restriction

addColumnsInteger

public Builder addColumnsInteger(String... names) 

Add multiple Integer columns with no restrictions on the min/max allowable values

• param names Names of the integer columns to add

addColumnsInteger

public Builder addColumnsInteger(String pattern, int minIdxInclusive, int maxIdxInclusive) 

A convenience method for adding multiple Integer columns. For example, to add columns “myIntegerCol_0”, “myIntegerCol_1”, “myIntegerCol_2”, use {- code addColumnsInteger(“myIntegerCol_%d”,0,2)}

• param pattern Pattern to use (via String.format). “%d” is replaced with column numbers

• param minIdxInclusive Minimum column index to use (inclusive)

• param maxIdxInclusive Maximum column index to use (inclusive)

addColumnsInteger

public Builder addColumnsInteger(String pattern, int minIdxInclusive, int maxIdxInclusive,
                        Integer minAllowedValue, Integer maxAllowedValue) 

A convenience method for adding multiple Integer columns. For example, to add columns “myIntegerCol_0”, “myIntegerCol_1”, “myIntegerCol_2”, use {- code addColumnsInteger(“myIntegerCol_%d”,0,2)}

• param pattern Pattern to use (via String.format). “%d” is replaced with column numbers

• param minIdxInclusive Minimum column index to use (inclusive)

• param maxIdxInclusive Maximum column index to use (inclusive)

• param minAllowedValue Minimum allowed value (inclusive). If null: no restriction

• param maxAllowedValue Maximum allowed value (inclusive). If null: no restriction

addColumnCategorical

public Builder addColumnCategorical(String name, String... stateNames) 

Add a Categorical column, with the specified state names

• param name Name of the column

• param stateNames Names of the allowable states for this categorical column

addColumnCategorical

public Builder addColumnCategorical(String name, List<String> stateNames) 

Add a Categorical column, with the specified state names

• param name Name of the column

• param stateNames Names of the allowable states for this categorical column

addColumnLong

public Builder addColumnLong(String name) 

Add a Long column, with no restrictions on the min/max values

• param name Name of the column

addColumnLong

public Builder addColumnLong(String name, Long minAllowedValue, Long maxAllowedValue) 

Add a Long column with the specified min/max allowable values

• param name Name of the column

• param minAllowedValue Minimum allowed value (inclusive). If null: no restriction

• param maxAllowedValue Maximum allowed value (inclusive). If null: no restriction

addColumnsLong

public Builder addColumnsLong(String... names) 

Add multiple Long columns, with no restrictions on the allowable values

• param names Names of the Long columns to add

addColumnsLong

public Builder addColumnsLong(String pattern, int minIdxInclusive, int maxIdxInclusive) 

A convenience method for adding multiple Long columns. For example, to add columns “myLongCol_0”, “myLongCol_1”, “myLongCol_2”, use {- code addColumnsLong(“myLongCol_%d”,0,2)}

• param pattern Pattern to use (via String.format). “%d” is replaced with column numbers

• param minIdxInclusive Minimum column index to use (inclusive)

• param maxIdxInclusive Maximum column index to use (inclusive)

addColumnsLong

public Builder addColumnsLong(String pattern, int minIdxInclusive, int maxIdxInclusive, Long minAllowedValue,
                        Long maxAllowedValue) 

A convenience method for adding multiple Long columns. For example, to add columns “myLongCol_0”, “myLongCol_1”, “myLongCol_2”, use {- code addColumnsLong(“myLongCol_%d”,0,2)}

• param pattern Pattern to use (via String.format). “%d” is replaced with column numbers

• param minIdxInclusive Minimum column index to use (inclusive)

• param maxIdxInclusive Maximum column index to use (inclusive)

• param minAllowedValue Minimum allowed value (inclusive). If null: no restriction

• param maxAllowedValue Maximum allowed value (inclusive). If null: no restriction

addColumn

public Builder addColumn(ColumnMetaData metaData) 

Add a column

• param metaData metadata for this column

addColumnString

public Builder addColumnString(String name) 

Add a String column with no restrictions on the allowable values.

• param name Name of the column

addColumnsString

public Builder addColumnsString(String... columnNames) 

Add multiple String columns with no restrictions on the allowable values

• param columnNames Names of the String columns to add

addColumnString

public Builder addColumnString(String name, String regex, Integer minAllowableLength,
                        Integer maxAllowableLength) 

Add a String column with the specified restrictions

• param name Name of the column

• param regex Regex that the String must match in order to be considered valid. If null: no regex restriction

• param minAllowableLength Minimum allowable length for the String to be considered valid

• param maxAllowableLength Maximum allowable length for the String to be considered valid

addColumnsString

public Builder addColumnsString(String pattern, int minIdxInclusive, int maxIdxInclusive) 

A convenience method for adding multiple numbered String columns. For example, to add columns “myStringCol_0”, “myStringCol_1”, “myStringCol_2”, use {- code addColumnsString(“myStringCol_%d”,0,2)}

• param pattern Pattern to use (via String.format). “%d” is replaced with column numbers

• param minIdxInclusive Minimum column index to use (inclusive)

• param maxIdxInclusive Maximum column index to use (inclusive)

addColumnsString

public Builder addColumnsString(String pattern, int minIdxInclusive, int maxIdxInclusive, String regex,
                        Integer minAllowedLength, Integer maxAllowedLength) 

A convenience method for adding multiple numbered String columns. For example, to add columns “myStringCol_0”, “myStringCol_1”, “myStringCol_2”, use {- code addColumnsString(“myStringCol_%d”,0,2)}

• param pattern Pattern to use (via String.format). “%d” is replaced with column numbers

• param minIdxInclusive Minimum column index to use (inclusive)

• param maxIdxInclusive Maximum column index to use (inclusive)

• param regex Regex that the String must match in order to be considered valid. If null: no regex restriction

• param minAllowedLength Minimum allowed length of strings (inclusive). If null: no restriction

• param maxAllowedLength Maximum allowed length of strings (inclusive). If null: no restriction

addColumnTime

public Builder addColumnTime(String columnName, TimeZone timeZone) 

Add a Time column with no restrictions on the min/max allowable times NOTE: Time columns are represented by LONG (epoch millisecond) values. For time values in human-readable formats, use String columns + StringToTimeTransform

• param columnName Name of the column

• param timeZone Time zone of the time column

addColumnTime

public Builder addColumnTime(String columnName, DateTimeZone timeZone) 

Add a Time column with no restrictions on the min/max allowable times NOTE: Time columns are represented by LONG (epoch millisecond) values. For time values in human-readable formats, use String columns + StringToTimeTransform

• param columnName Name of the column

• param timeZone Time zone of the time column

addColumnTime

public Builder addColumnTime(String columnName, DateTimeZone timeZone, Long minValidValue, Long maxValidValue) 

Add a Time column with the specified restrictions NOTE: Time columns are represented by LONG (epoch millisecond) values. For time values in human-readable formats, use String columns + StringToTimeTransform

• param columnName Name of the column

• param timeZone Time zone of the time column

• param minValidValue Minumum allowable time (in milliseconds). May be null.

• param maxValidValue Maximum allowable time (in milliseconds). May be null.

addColumnNDArray

public Builder addColumnNDArray(String columnName, long[] shape) 

Add a NDArray column

• param columnName Name of the column

• param shape shape of the NDArray column. Use -1 in entries to specify as “variable length” in that dimension

build

public Schema build() 

Create the Schema

inferMultiple

public static Schema inferMultiple(List<List<Writable>> record) 

Infers a schema based on the record. The column names are based on indexing.

• param record the record to infer from

• return the infered schema

infer

public static Schema infer(List<Writable> record) 

Infers a schema based on the record. The column names are based on indexing.

• param record the record to infer from

• return the infered schema

SequenceSchema

[source]

inferSequenceMulti

public static SequenceSchema inferSequenceMulti(List<List<List<Writable>>> record) 

Infers a sequence schema based on the record

• param record the record to infer the schema based on

• return the inferred sequence schema

inferSequence

public static SequenceSchema inferSequence(List<List<Writable>> record) 

Infers a sequence schema based on the record

• param record the record to infer the schema based on

• return the inferred sequence schema