Arm Treasure Data's remove_columns filter plugin for data connectors removes columns from the schema as data is imported.

Option	type	description	required?
remove	array	An array of names of columns that it removes from schema.	[] by default
keep	array	An array of names of columns that it keeps in schema.	[] by default
accept_unmatched_columns	boolean	If true, skip columns that aren’t included in schemas.	false by default

You cannot specify both remove: and keep: options in the same .yml file.

Configuration

Add the filter section with type remove_columns to your load.yml, for example:

in:
...

# This configuration removes "_c0" and "_c1" named columns from schema.
filters:
...
- type: remove_columns
  remove: ["_c0", "_c1"]

out:
...

remove_columns filter Plugin for Data Connectors

Configuration

Comments

Configuration

Related articles