Back to plugin list
clickhouse
Official

ClickHouse

This destination plugin lets you sync data from a CloudQuery source to ClickHouse database

Publisher

cloudquery

Repositorygithub.com
Latest version

v4.0.9

Type

Destination

Platforms
Date Published

Price

Free

Overview

ClickHouse destination plugin

This destination plugin lets you sync data from a CloudQuery source to ClickHouse database.
It supports append write mode only. Write mode selection is required through write_mode.
Supported database versions: >= 22.1.2

Configuration

Example

kind: destination
spec:
  name: "clickhouse"
  path: "cloudquery/clickhouse"
  registry: "cloudquery"
  version: "v4.0.9"
  write_mode: "append"
  spec:
    connection_string: "clickhouse://${CH_USER}:${CH_PASSWORD}@localhost:9000/${CH_DATABASE}"
    # Optional parameters
    # cluster: ""
    # ca_cert: ""
    # engine:
    #   name: MergeTree
    #   parameters: []
    #
    # batch_size: 10000
    # batch_size_bytes: 5242880 # 5 MiB
    # batch_timeout: 20s
This example configures a ClickHouse instance, located at localhost:9000. It expects CH_USER, CH_PASSWORD and CH_DATABASE environment variables to be set. The (top level) spec section is described in the Destination Spec Reference.

ClickHouse spec

This is the (nested) spec used by the ClickHouse destination plugin.
  • connection_string (string) (required)
    Connection string to connect to the database. See SDK documentation for more details.
  • cluster (string) (optional) (default: not used)
    Cluster name to be used for distributed DDL. If the value is empty, DDL operations will affect only the server the plugin is connected to.
  • ca_cert (string) (optional) (default: not used)
    PEM-encoded certificate authorities. When set, a certificate pool will be created by appending the certificates to the system pool.
    See file variable substitution for how to read this value from a file.
  • engine (optional, table engine settings. Default: MergeTree engine)
    Engine to be used for tables. Only *MergeTree family is supported at the moment.
  • batch_size (integer) (optional) (default: 10000)
    Maximum number of items that may be grouped together to be written in a single write.
  • batch_size_bytes (integer) (optional) (default: 5242880 (= 5 MiB))
    Maximum size of items that may be grouped together to be written in a single write.
  • batch_timeout (duration) (optional) (default: 20s)
    Maximum interval between batch writes.
ClickHouse table engine
This option allows to specify a custom table engine to be used.
  • name (string) (required)
    Name of the table engine. Only *MergeTree family is supported at the moment.
  • parameters (array of parameters) (optional) (default: empty)
    Engine parameters. Currently, no restrictions are imposed on the parameter types.
kind: destination
spec:
  name:       "clickhouse"
  path:       "cloudquery/clickhouse"
  registry:   "cloudquery"
  version:    "v4.0.9"
  write_mode: "append"

  spec:
    connection_string: "clickhouse://${CH_USER}:${CH_PASSWORD}@localhost:9000/${CH_DATABASE}"
    engine:
      name: ReplicatedMergeTree
      parameters:
      - "/clickhouse/tables/{shard}/{database}/{table}"
      - "{replica}"

Connecting to ClickHouse Cloud

To connect to ClickHouse Cloud, you need to set the secure=true parameter, username is default, and the port is 9440. Use a connection string similar to:
connection_string: "clickhouse://default:${CH_PASSWORD}@<your-server-id>.<region>.<provider>.clickhouse.cloud:9440/${CH_DATABASE}?secure=true"
Verbose logging for debug
The ClickHouse destination can be run in debug mode. To achieve this pass the debug=true option to connection_string. See SDK documentation for more details.
Note: This will use SDK built-in logging and might output data and sensitive information to logs. Make sure not to use it in production environment.
kind: destination
spec:
  name:       "clickhouse"
  path:       "cloudquery/clickhouse"
  registry:   "cloudquery"
  version:    "v4.0.9"
  write_mode: "append"

  spec:
    connection_string: "clickhouse://${CH_USER}:${CH_PASSWORD}@localhost:9000/${CH_DATABASE}?debug=true"


Types

Apache Arrow type conversion

The ClickHouse destination plugin supports most of Apache Arrow types. It uses the same approach as documented in ClickHouse reference. The following table shows the supported types and how they are mapped to ClickHouse data types.
Arrow Column TypeClickHouse Type
BinaryString
Binary ViewString
BooleanBool
Date32Date32
Date64DateTime
Decimal128 (Decimal)Decimal
Decimal256Decimal
Fixed Size BinaryFixedString
Fixed Size ListArray
Float16Float32
Float32Float32
Float64Float64
Int8Int8
Int16Int16
Int32Int32
Int64Int64
Large BinaryString
Large ListArray
Large StringString
ListArray
MapMap
StringString
String ViewString
StructTuple
Time32DateTime64
Time64DateTime64
TimestampDateTime64
UUID (CloudQuery extension)UUID
Uint8UInt8
Uint16UInt16
Uint32UInt32
Uint64UInt64


Subscribe to product updates

Be the first to know about new features.