This destination plugin lets you sync data from a CloudQuery source to local files in various formats. It currently supports CSV, line-delimited JSON and Parquet
Loading plugin documentation
We use tracking cookies to understand how you use the product and help us improve it. Please accept cookies to help us improve. You can always opt out later via the link in the footer.
./cq_csv_output. You can also choose json or parquet as the output format.kind: destination
spec:
name: "file"
path: "cloudquery/file"
registry: "cloudquery"
version: "v5.4.25"
write_mode: "append"
# Learn more about the configuration options at https://cql.ink/file_destination
spec:
path: "path/to/files/{{TABLE}}/{{UUID}}.{{FORMAT}}"
format: "parquet" # options: parquet, json, csv
# Optional parameters
# format_spec:
# CSV specific parameters:
# delimiter: ","
# skip_header: false
# Parquet specific parameters:
# version: "v2Latest"
# root_repetition: "repeated"
# max_row_group_length: 134217728 # 128 * 1024 * 1024
# compression: "" # options: gzip
# no_rotate: false
# batch_size: 10000
# batch_size_bytes: 52428800 # 50 MiB
# batch_timeout: 30s
append write_mode. The (top level) spec section is described in the Destination Spec Reference.path (string) (required)path/to/files/{{TABLE}}/{{UUID}}.parquet.{{TABLE}} will be replaced with the table name{{FORMAT}} will be replaced with the file format, such as csv, json or parquet. If compression is enabled, the format will be csv.gz, json.gz etc.{{UUID}} will be replaced with a random UUID to uniquely identify each file{{YEAR}} will be replaced with the current year in YYYY format{{MONTH}} will be replaced with the current month in MM format{{DAY}} will be replaced with the current day in DD format{{HOUR}} will be replaced with the current hour in HH format{{MINUTE}} will be replaced with the current minute in mm formatUTC and will be the current time at the time the file is written, not when the sync started.format (string) (required)csv, json and parquet.format_spec (format_spec) (optional)no_rotate (boolean) (optional) (default: false)true, the plugin will write to one file per table.
Otherwise, for every batch a new file will be created with a different .<UUID> suffix.compression (string) (optional) (default: "")"" and gzip. Not supported for parquet format.batch_size (integer) (optional) (default: 10000)batch_size_bytes (integer) (optional) (default: 52428800 (50 MiB))batch_timeout (duration) (optional) (default: 30s (30 seconds))delimiter (string) (optional) (default: ,)skip_header (boolean) (optional) (default: false)true, the CSV file will not contain a header row as the first row.version (string) (optional) (default: v2Latest)v1.0, v2.4, v2.6 and v2Latest.
v2Latest is an alias for the latest version available in the Parquet library which is currently v2.6.root_repetition (string) (optional) (default: repeated)undefined, required, optional and repeated.undefined.max_row_group_length (integer) (optional) (default: 134217728 (= 128 _ 1024 _ 1024))