Filters
Selection of data using conditions.

Using filters

Filters are a part of transforms and gives a DSL for you to keep parts of your dataset. Filters can be one-liners for single conditions or include complex boolean logic.
TransformProcess tp = new TransformProcess.Builder(inputDataSchema)
.filter(new ConditionFilter(new CategoricalColumnCondition("MerchantCountryCode", ConditionOp.NotInSet, new HashSet<>(Arrays.asList("USA","CAN")))))
.build();
You can also write your own filters by implementing the Filter interface, though it is much more often that you may want to create a custom condition instead.

Available filters

ConditionFilter

[source]
If condition is satisfied (returns true): remove the example or sequence If condition is not satisfied (returns false): keep the example or sequence
removeExample
public boolean removeExample(Object writables)
  • param writables Example
  • return true if example should be removed, false to keep
removeSequence
public boolean removeSequence(Object sequence)
  • param sequence sequence example
  • return true if example should be removed, false to keep
transform
public Schema transform(Schema inputSchema)
Get the output schema for this transformation, given an input schema
  • param inputSchema
outputColumnName
public String outputColumnName()
The output column name after the operation has been applied
  • return the output column name
columnName
public String columnName()
The output column names This will often be the same as the input
  • return the output column names

Filter

[source]
Filter: a method of removing examples (or sequences) according to some condition

FilterInvalidValues

[source]
FilterInvalidValues: a filter operation that removes any examples (or sequences) if the examples/sequences contains invalid values in any of a specified set of columns. Invalid values are determined with respect to the schema
transform
public Schema transform(Schema inputSchema)
  • param columnsToFilterIfInvalid Columns to check for invalid values
removeExample
public boolean removeExample(Object writables)
  • param writables Example
  • return true if example should be removed, false to keep
removeSequence
public boolean removeSequence(Object sequence)
  • param sequence sequence example
  • return true if example should be removed, false to keep
outputColumnName
public String outputColumnName()
The output column name after the operation has been applied
  • return the output column name
columnName
public String columnName()
The output column names This will often be the same as the input
  • return the output column names

InvalidNumColumns

[source]
Remove invalid records of a certain size.
removeExample
public boolean removeExample(Object writables)
  • param writables Example
  • return true if example should be removed, false to keep
removeSequence
public boolean removeSequence(Object sequence)
  • param sequence sequence example
  • return true if example should be removed, false to keep
removeExample
public boolean removeExample(List<Writable> writables)
  • param writables Example
  • return true if example should be removed, false to keep
removeSequence
public boolean removeSequence(List<List<Writable>> sequence)
  • param sequence sequence example
  • return true if example should be removed, false to keep
transform
public Schema transform(Schema inputSchema)
Get the output schema for this transformation, given an input schema
  • param inputSchema
outputColumnName
public String outputColumnName()
The output column name after the operation has been applied
  • return the output column name
columnName
public String columnName()
The output column names This will often be the same as the input
  • return the output column names
Export as PDF
Copy link
On this page
Using filters
Available filters
ConditionFilter
Filter
FilterInvalidValues
InvalidNumColumns