The CloudQuery transformer plugin provides basic transformation capabilities, such as removing columns, adding literal string columns, obfuscating string columns and renaming tables
Loading plugin documentation
We use tracking cookies to understand how you use the product and help us improve it. Please accept cookies to help us improve. You can always opt out later via the link in the footer.
{{.OldName}} to refer to the original name, see example below)kind: destination
spec:
name: "postgresql"
path: "cloudquery/postgresql"
registry: "cloudquery"
version: "v8.0.7"
write_mode: "overwrite-delete-stale"
migrate_mode: forced # optional
transformers:
- "basic"
spec:
connection_string: "postgresql://your.user:your.password@localhost:5432/db_name"
migrate_mode: forced setting might make sense if you plan on modifying the schema from a previous sync.kind: transformer
spec:
name: "basic"
path: "cloudquery/basic"
version: v2.6.5
spec:
transformations:
- kind: obfuscate_columns
tables: ["xkcd_comics"]
columns: ["safe_title", "title"]
- kind: obfuscate_sensitive_columns
- kind: remove_columns
tables: ["xkcd_comics"]
columns: ["transcript", "news"]
- kind: add_column
tables: ["xkcd_comics"]
name: "source"
value: "xkcd"
- kind: add_primary_keys
tables: ["xkcd_comics"]
columns: ["_cq_source_name"]
- kind: add_current_timestamp_column
tables: ["xkcd_comics"]
name: "_record_processed_at"
- kind: change_table_names
tables: ["*"]
new_table_name_template: "cq_sync_{{.OldName}}"
- kind: rename_column
tables: ["xkcd_comics"]
name: img
value: img_url
- kind: uppercase
tables: ["xkcd_comics"]
columns: ["title"]
- kind: lowercase
tables: ["xkcd_comics"]
columns: ["title"]
- kind: drop_rows
tables: ["xkcd_comics"]
columns: ["year"]
value: "2023"
tags:{"foo":{"bar":["a","b","c"]},"hello":"world","kubectl.kubernetes.io/last-applied-configuration":"secrets"}
"a" and remove "b", "world", and "secrets" with:kind: transformer
spec:
name: "basic"
path: "cloudquery/basic"
registry: "cloudquery"
spec:
transformations:
- kind: obfuscate_columns
tables: ["example"]
columns: ["tags.foo.bar.0"]
- kind: remove_columns
tables: ["example"]
columns: ["tags.hello", "tags.foo.bar.1", "tags.kubectl\\.kubernetes\\.io\\/last-applied-configuration"]
example_column with value: {"top_foo":[{"foo": "baz0"},{"foo": "baz1"},{"foo": "baz2"}]} you can use the following syntax:kind: transformer
spec:
name: "basic"
path: "cloudquery/basic"
registry: "cloudquery"
spec:
transformations:
- kind: obfuscate_columns
tables: ["example"]
columns: ["example_column.top_foo.#.foo"]
#.foo syntax will cause all foo values to be replaced with the same obfuscated value {"top_foo":[{"foo": "Redacted by CloudQuery | XXX"},{"foo": "Redacted by CloudQuery | XXX"},{"foo": "Redacted by CloudQuery | XXX"}]}.obfuscate_sensitive_columns transformation to automatically obfuscate all columns marked by the source plugin as sensitive and possibly containing secret information.drop_rows transformation:nil values, configure value: null, value: ~ or drop the value configuration altogethertags has a value of {"foo": "bar"}, you should specify the value as {"foo":"bar"} without any whitespace.