Back to destination list
Official
ClickHouse
This destination plugin lets you sync data from a CloudQuery source to ClickHouse database
Loading plugin documentation
This destination plugin lets you sync data from a CloudQuery source to ClickHouse database
Loading plugin documentation
We use tracking cookies to understand how you use the product and help us improve it. Please accept cookies to help us improve. You can always opt out later via the link in the footer.
append
write mode only.
Write mode selection is required through write_mode
.24.8.1
kind: destination
spec:
name: "clickhouse"
path: "cloudquery/clickhouse"
registry: "cloudquery"
version: "v7.3.0"
write_mode: "append"
# Learn more about the configuration options at https://cql.ink/clickhouse_destination
spec:
connection_string: "clickhouse://${CH_USER}:${CH_PASSWORD}@localhost:9000/${CH_DATABASE}"
# Optional parameters
# cluster: ""
# ca_cert: ""
# engine:
# name: MergeTree
# parameters: []
#
# batch_size: 10000
# batch_size_bytes: 5242880 # 5 MiB
# batch_timeout: 20s
localhost:9000
.
It expects CH_USER
, CH_PASSWORD
and CH_DATABASE
environment variables to be set.
The (top level) spec section is described in the Destination Spec Reference.connection_string
(string
) (required)"clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60"
cluster
(string
) (optional) (default: not used)ca_cert
(string
) (optional) (default: not used)*MergeTree
family is supported at the moment.batch_size
(integer
) (optional) (default: 10000
)batch_size_bytes
(integer
) (optional) (default: 5242880
(= 5 MiB))batch_timeout
(duration
) (optional) (default: 20s
)partition
(optional, partitioning) (default: no partitioning)PARTITION BY
clause in CREATE TABLE
statements).order
(optional, ordering) (default: use existing primary key)ORDER BY
clause in CREATE TABLE
statements).name
(string
) (required)*MergeTree
family is supported at the moment.parameters
(array of parameters) (optional) (default: empty)kind: destination
spec:
name: "clickhouse"
path: "cloudquery/clickhouse"
registry: "cloudquery"
version: "v7.3.0"
write_mode: "append"
spec:
connection_string: "clickhouse://${CH_USER}:${CH_PASSWORD}@localhost:9000/${CH_DATABASE}"
engine:
name: ReplicatedMergeTree
parameters:
- "/clickhouse/tables/{shard}/{database}/{table}"
- "{replica}"
tables
(array of strings) (optional) (default: ["*"]
)tables
option.tables
and skip_tables
, the table will be skipped.skip_tables
(array of strings) (optional) (default: empty)skip_tables
option.tables
and skip_tables
, the table will be skipped.skip_incremental_tables
(boolean) (optional) (default: false
)true
, incremental tables will not be partitioned by this strategy.partition_by
(string) (required)toYYYYMM(_cq_sync_time)
, the string is passed as is after "PARTITION BY" clause with no validation or quoting.partition_by
is not valid.partition:
- tables: ["*"]
skip_tables: ["special_partition_table", "non_partitioned_table"]
partition_by: "toYYYYMM(_cq_sync_time)"
- tables: ["special_partition_table"]
partition_by: "toYYYYMMDD(_cq_sync_time)"
ORDER BY
clauses for tables or groups of tables. It is an array of objects.tables
(array of strings) (optional) (default: ["*"]
)tables
option.tables
and skip_tables
, the table will be skipped.skip_tables
(array of strings) (optional) (default: empty)skip_tables
option.tables
and skip_tables
, the table will be skipped.order_by
(array of strings) (required)order:
- tables: ["aws_ec2_instances"]
order_by:
- "`account_id`"
- "`region`"
- "toYYYYMM(`_cq_sync_time`) DESC"
- "`_cq_id`"
tables
(array of strings) (optional) (default: ["*"]
)tables
option.tables
and skip_tables
, the table will be skipped.skip_tables
(array of strings) (optional) (default: empty)skip_tables
option.tables
and skip_tables
, the table will be skipped.ttl
(string) (required)INTERVAL 60 DAY
, the string is passed as-is after a "TTL _cq_sync_time + " clause with no validation or quoting.ttl:
- tables: ["*"]
skip_tables: ["no_ttl_table"]
ttl: "INTERVAL 60 DAY"
secure=true
parameter, username is default
, and the port is 9440
. Use a connection string similar to: connection_string: "clickhouse://default:${CH_PASSWORD}@<your-server-id>.<region>.<provider>.clickhouse.cloud:9440/${CH_DATABASE}?secure=true"
debug=true
option to connection_string
.
See SDK documentation for more details.kind: destination
spec:
name: "clickhouse"
path: "cloudquery/clickhouse"
registry: "cloudquery"
version: "v7.3.0"
write_mode: "append"
spec:
connection_string: "clickhouse://${CH_USER}:${CH_PASSWORD}@localhost:9000/${CH_DATABASE}?debug=true"