Back to destination list
Official
ClickHouse destination integration documentation
This destination plugin lets you sync data from a CloudQuery source to ClickHouse database
Loading plugin documentation
This destination plugin lets you sync data from a CloudQuery source to ClickHouse database
Loading plugin documentation
CloudQuery's use of cookies
We use tracking cookies to understand how you use the product and help us improve it. Your consent is required before we can enable these cookies. You can opt out via the link in the footer.
append write mode only.
Write mode selection is required through write_mode.24.8.1kind: destination
spec:
name: "clickhouse"
path: "cloudquery/clickhouse"
registry: "cloudquery"
version: "v8.1.1"
write_mode: "append"
send_sync_summary: true
# Learn more about the configuration options at https://cql.ink/clickhouse_destination
spec:
connection_string: "clickhouse://${CH_USER}:${CH_PASSWORD}@localhost:9000/${CH_DATABASE}"
# Optional parameters
# cluster: ""
# ca_cert: ""
# engine:
# name: MergeTree
# parameters: []
#
# batch_size: 10000
# batch_size_bytes: 5242880 # 5 MiB
# batch_timeout: 20s
localhost:9000.
It expects CH_USER, CH_PASSWORD and CH_DATABASE environment variables to be set.
The (top level) spec section is described in the Destination Spec Reference.connection_string (string) (required)"clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60"cluster (string) (optional) (default: not used)ca_cert (string) (optional) (default: not used)*MergeTree family is supported at the moment.batch_size (integer) (optional) (default: 10000)batch_size_bytes (integer) (optional) (default: 5242880 (= 5 MiB))batch_timeout (duration) (optional) (default: 20s)partition (optional, partitioning) (default: no partitioning)PARTITION BY clause in CREATE TABLE statements).order (optional, ordering) (default: use existing primary key)ORDER BY clause in CREATE TABLE statements).name (string) (required)*MergeTree family is supported at the moment.parameters (array of parameters) (optional) (default: empty)kind: destination
spec:
name: "clickhouse"
path: "cloudquery/clickhouse"
registry: "cloudquery"
version: "v8.1.1"
write_mode: "append"
send_sync_summary: true
spec:
connection_string: "clickhouse://${CH_USER}:${CH_PASSWORD}@localhost:9000/${CH_DATABASE}"
engine:
name: ReplicatedMergeTree
parameters:
- "/clickhouse/tables/{shard}/{database}/{table}"
- "{replica}"
tables (array of strings) (optional) (default: ["*"])tables option.tables and skip_tables, the table will be skipped.skip_tables (array of strings) (optional) (default: empty)skip_tables option.tables and skip_tables, the table will be skipped.skip_incremental_tables (boolean) (optional) (default: false)true, incremental tables will not be partitioned by this strategy.partition_by (string) (required)toYYYYMM(_cq_sync_time), the string is passed as is after "PARTITION BY" clause with no validation or quoting.partition_by is not valid.partition:
- tables: ["*"]
skip_tables: ["special_partition_table", "non_partitioned_table"]
partition_by: "toYYYYMM(_cq_sync_time)"
- tables: ["special_partition_table"]
partition_by: "toYYYYMMDD(_cq_sync_time)"
ORDER BY clauses for tables or groups of tables. It is an array of objects.tables (array of strings) (optional) (default: ["*"])tables option.tables and skip_tables, the table will be skipped.skip_tables (array of strings) (optional) (default: empty)skip_tables option.tables and skip_tables, the table will be skipped.order_by (array of strings) (required)order:
- tables: ["aws_ec2_instances"]
order_by:
- "`account_id`"
- "`region`"
- "toYYYYMM(`_cq_sync_time`) DESC"
- "`_cq_id`"
tables (array of strings) (optional) (default: ["*"])tables option.tables and skip_tables, the table will be skipped.skip_tables (array of strings) (optional) (default: empty)skip_tables option.tables and skip_tables, the table will be skipped.ttl (string) (required)INTERVAL 60 DAY, the string is passed as-is after a "TTL _cq_sync_time + " clause with no validation or quoting.ttl:
- tables: ["*"]
skip_tables: ["no_ttl_table"]
ttl: "INTERVAL 60 DAY"
secure=true parameter, username is default, and the port is 9440. Use a connection string similar to: connection_string: "clickhouse://default:${CH_PASSWORD}@<your-server-id>.<region>.<provider>.clickhouse.cloud:9440/${CH_DATABASE}?secure=true"
debug=true option to connection_string.
See SDK documentation for more details.kind: destination
spec:
name: "clickhouse"
path: "cloudquery/clickhouse"
registry: "cloudquery"
version: "v8.1.1"
write_mode: "append"
send_sync_summary: true
spec:
connection_string: "clickhouse://${CH_USER}:${CH_PASSWORD}@localhost:9000/${CH_DATABASE}?debug=true"