Schemas

Why use schemas?

The unfortunate reality is that data is dirty. When trying to vecotrize a dataset for deep learning, it is quite rare to find files that have zero errors. Schema is important for maintaining the meaning of the data before using it for something like training a neural network.

Using schemas

Schemas are primarily used for programming transformations. Before you can properly execute a TransformProcess you will need to pass the schema of the data being transformed.

An example of a schema for merchant records may look like:

Schema inputDataSchema = new Schema.Builder()
    .addColumnsString("DateTimeString", "CustomerID", "MerchantID")
    .addColumnInteger("NumItemsInTransaction")
    .addColumnCategorical("MerchantCountryCode", Arrays.asList("USA","CAN","FR","MX"))
    .addColumnDouble("TransactionAmountUSD",0.0,null,false,false)   //$0.0 or more, no maximum limit, no NaN and no Infinite values
    .addColumnCategorical("FraudLabel", Arrays.asList("Fraud","Legit"))
    .build();

Joining schemas

If you have two different datasets that you want to merge together, DataVec provides a Join class with different join strategies such as Inner or RightOuter.

Schema customerInfoSchema = new Schema.Builder()
    .addColumnLong("customerID")
    .addColumnString("customerName")
    .addColumnCategorical("customerCountry", Arrays.asList("USA","France","Japan","UK"))
    .build();

Schema customerPurchasesSchema = new Schema.Builder()
    .addColumnLong("customerID")
    .addColumnTime("purchaseTimestamp", DateTimeZone.UTC)
    .addColumnLong("productID")
    .addColumnInteger("purchaseQty")
    .addColumnDouble("unitPriceUSD")
    .build();

Join join = new Join.Builder(Join.JoinType.Inner)
    .setJoinColumns("customerID")
    .setSchemas(customerInfoSchema, customerPurchasesSchema)
    .build();

Once you've defined your join and you've loaded the data into DataVec, you must use an Executor to complete the join.

Classes and utilities

DataVec comes with a few Schema classes and helper utilities for 2D and sequence types of data.

Join

[source]

Join class: used to specify a join (like an SQL join)

setSchemas

public Builder setSchemas(Schema left, Schema right)

Type of join Inner: Return examples where the join column values occur in both LeftOuter: Return all examples from left data, whether there is a matching right value or not. (If not: right values will have NullWritable instead) RightOuter: Return all examples from the right data, whether there is a matching left value or not. (If not: left values will have NullWritable instead) FullOuter: return all examples from both left/right, whether there is a matching value from the other side or not. (If not: other values will have NullWritable instead)

setKeyColumns

public Builder setKeyColumns(String... keyColumnNames)

• deprecated Use {- link #setJoinColumns(String…)}

setKeyColumnsLeft

public Builder setKeyColumnsLeft(String... keyColumnNames)

• deprecated Use {- link #setJoinColumnsLeft(String…)}

setKeyColumnsRight

public Builder setKeyColumnsRight(String... keyColumnNames)

• deprecated Use {- link #setJoinColumnsRight(String…)}

setJoinColumnsLeft

public Builder setJoinColumnsLeft(String... joinColumnNames)

Specify the names of the columns to join on, for the left data) The idea: join examples where firstDataValues(joinColumNamesLeft[i]) == secondDataValues(joinColumnNamesRight[i]) for all i

• param joinColumnNames Names of the columns to join on (for left data)

setJoinColumnsRight

public Builder setJoinColumnsRight(String... joinColumnNames)

Specify the names of the columns to join on, for the right data) The idea: join examples where firstDataValues(joinColumNamesLeft[i]) == secondDataValues(joinColumnNamesRight[i]) for all i

• param joinColumnNames Names of the columns to join on (for left data)

InferredSchema

[source]

If passed a CSV file that contains a header and a single row of sample data, it will return a Schema.

Only Double, Integer, Long, and String types are supported. If no number type can be inferred, the field type will become the default type. Note that if your column is actually categorical but is represented as a number, you will need to do additional transformation. Also, if your sample field is blank/null, it will also become the default type.

Schema

[source]

A Schema defines the layout of tabular data. Specifically, it contains names f or each column, as well as details of types (Integer, String, Long, Double, etc). Type information for each column may optionally include restrictions on the allowable values for each column.

sameTypes

public boolean sameTypes(Schema schema)

Create a schema based on the given metadata

• param columnMetaData the metadata to create the schema from

newSchema

public Schema newSchema(List<ColumnMetaData> columnMetaData)

Compute the difference in {- link ColumnMetaData} between this schema and the passed in schema. This is useful during the {- link org.datavec.api.transform.TransformProcess} to identify what a process will do to a given {- link Schema}.

• param schema the schema to compute the difference for

• return the metadata that is different (in order) between this schema and the other schema

numColumns

public int numColumns()

Returns the number of columns or fields for this schema

• return the number of columns or fields for this schema

getName

public String getName(int column)

Returns the name of a given column at the specified index

• param column the index of the column to get the name for

• return the name of the column at the specified index

getType

public ColumnType getType(int column)

Returns the {- link ColumnType} for the column at the specified index

• param column the index of the column to get the type for

• return the type of the column to at the specified inde

getType

public ColumnType getType(String columnName)

Returns the {- link ColumnType} for the column at the specified index

• param columnName the index of the column to get the type for

• return the type of the column to at the specified inde

getMetaData

public ColumnMetaData getMetaData(int column)

Returns the {- link ColumnMetaData} at the specified column index

• param column the index to get the metadata for

• return the metadata at ths specified index

getMetaData

public ColumnMetaData getMetaData(String column)

Retrieve the metadata for the given column name

• param column the name of the column to get metadata for

• return the metadata for the given column name

getIndexOfColumn

public int getIndexOfColumn(String columnName)

Return a copy of the list column names

• return a copy of the list of column names for this schema

hasColumn

public boolean hasColumn(String columnName)

Return the indices of the columns, given their namess

• param columnNames Name of the columns to get indices for

• return Column indexes

toJson

public String toJson()

Serialize this schema to json

• return a json representation of this schema

toYaml

public String toYaml()

Serialize this schema to yaml

• return the yaml representation of this schema

fromJson

public static Schema fromJson(String json)

Create a schema from a given json string

• param json the json to create the schema from

• return the created schema based on the json

fromYaml

public static Schema fromYaml(String yaml)

Create a schema from the given yaml string

• param yaml the yaml to create the schema from

• return the created schema based on the yaml

addColumnFloat

public Builder addColumnFloat(String name)

Add a Float column with no restrictions on the allowable values, except for no NaN/infinite values allowed

• param name Name of the column

addColumnFloat

public Builder addColumnFloat(String name, Float minAllowedValue, Float maxAllowedValue)

Add a Float column with the specified restrictions (and no NaN/Infinite values allowed)

• param name Name of the column