Official

Premium

ClickHouse

Sync from ClickHouse to any destination

Publisher

cloudquery

Latest version

v1.2.25

Type

Source

Platforms

Date Published

Download CloudQuery CLI

Documentation Tables Changelog Destinations

Overview Types Licenses

Overview #

ClickHouse source plugin

The CloudQuery ClickHouse plugin syncs your ClickHouse database to any of the supported CloudQuery destinations (e.g. PostgreSQL, BigQuery, Snowflake, and more).

Supported database versions: >= 24.2

Configuration #

Example #

kind: source
spec:
  name: "clickhouse"
  path: "cloudquery/clickhouse"
  registry: "cloudquery"
  version: "v1.2.25"
  destinations: ["postgresql"]
  tables: ["*"]
  # Learn more about the configuration options at https://cql.ink/clickhouse_source
  spec:
    connection_string: "clickhouse://${CH_USER}:${CH_PASSWORD}@localhost:9000/${CH_DATABASE}"
    # Optional parameters
    # ca_cert: ""
    # rows_per_record: 500
    # concurrency: 100
    # table_options:
    #   test_table:
    #     incremental_column: "created_at"

This example configures sync from a ClickHouse instance, located at localhost:9000. It expects CH_USER, CH_PASSWORD and CH_DATABASE environment variables to be set. The (top level) spec section is described in the Source Spec Reference.

Make sure you use environment variable expansion in production instead of committing the credentials to the configuration file directly.

ClickHouse spec #

This is the (nested) spec used by the ClickHouse source plugin.

connection_string (string) (required)
Connection string to connect to the database. See SDK documentation for more details.
ca_cert (string) (optional) (default: not used)
PEM-encoded certificate authorities. When set, a certificate pool will be created by appending the certificates to the system pool.
See file variable substitution for how to read this value from a file.
rows_per_record (integer) (optional) (default: 500)
Amount of rows to be packed into a single Apache Arrow record to be sent over the wire during sync.
concurrency (integer) (optional) (default: 100)
Number of tables to sync concurrently. Lower or increase this number based on your database size and available resources.
table_options (object) (optional) (default: not used)
Table options to set an incremental column.

Connecting to ClickHouse Cloud #

To connect to ClickHouse Cloud, you need to set the secure=true parameter, username to default, and the port to 9440. Use a connection string similar to:

    connection_string: "clickhouse://default:${CH_PASSWORD}@<your-server-id>.<region>.<provider>.clickhouse.cloud:9440/${CH_DATABASE}?secure=true"

See Quick Start: Using the ClickHouse Client for more details.

Verbose logging for debug

The ClickHouse source can be run in debug mode. To achieve this pass the debug=true option to connection_string. See SDK documentation for more details.

Note: This will use SDK built-in logging and might output data and sensitive information to logs. Make sure not to use it in production environment.

kind: source
spec:
  name: "clickhouse"
  path: "cloudquery/clickhouse"
  registry: "cloudquery"
  version: "v1.2.25"
  tables: ["*"]
  spec:
    connection_string: "clickhouse://${CH_USER}:${CH_PASSWORD}@localhost:9000/${CH_DATABASE}?debug=true"

Types #

Apache Arrow type conversion #

The ClickHouse source plugin supports most of ClickHouse data types. It uses the same approach as documented in ClickHouse reference. The following table shows the supported types and how they are mapped to Apache Arrow types.

Unsupported types are always mapped to String.

ClickHouse Type	Arrow Column Type
Bool	Boolean
UInt8	Uint8
UInt16	Uint16
UInt32	Uint32
UInt64	Uint64
Int8	Int8
Int16	Int16
Int32	Int32
Int64	Int64
Float32	Float32
Float64	Float64
String	String
FixedString	Fixed Size Binary
Date32	Date32
DateTime	Timestamp
DateTime64	Timestamp
Decimal	Decimal128 (Decimal) or Decimal256 based on precision
Array	List
Map	Map
Tuple	Struct
UUID	UUID (CloudQuery extension)

Nested ClickHouse types have their values converted according to the aforementioned rules.

Licenses #

The following tools / packages are used in this plugin:

Name	License
github.com/ClickHouse/ch-go	Apache-2.0
github.com/ClickHouse/clickhouse-go/v2	Apache-2.0
github.com/adrg/xdg	MIT
github.com/andybalholm/brotli	MIT
github.com/apache/arrow/go/v13	Apache-2.0
github.com/apache/arrow-go/v18	Apache-2.0
github.com/apapsch/go-jsonmerge/v2	MIT
github.com/aws/aws-sdk-go-v2	Apache-2.0
github.com/aws/aws-sdk-go-v2/config	Apache-2.0
github.com/aws/aws-sdk-go-v2/credentials	Apache-2.0
github.com/aws/aws-sdk-go-v2/feature/ec2/imds	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/configsources	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/ini	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/sync/singleflight	BSD-3-Clause
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/licensemanager	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/marketplacemetering	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/sso	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/ssooidc	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/sts	Apache-2.0
github.com/aws/smithy-go	Apache-2.0
github.com/aws/smithy-go/internal/sync/singleflight	BSD-3-Clause
github.com/bahlo/generic-list-go	BSD-3-Clause
github.com/buger/jsonparser	MIT
github.com/cenkalti/backoff/v4	MIT
github.com/cloudquery/cloudquery-api-go	MPL-2.0
github.com/cloudquery/cloudquery/plugins/destination/clickhouse	MPL-2.0
github.com/cloudquery/plugin-pb-go	MPL-2.0
github.com/cloudquery/plugin-sdk/v2/internal/glob	MIT
github.com/cloudquery/plugin-sdk/v2/schema	MIT
github.com/cloudquery/plugin-sdk/v2/types	MPL-2.0
github.com/cloudquery/plugin-sdk/v4	MPL-2.0
github.com/cloudquery/plugin-sdk/v4/glob	MIT
github.com/cloudquery/plugin-sdk/v4/scalar	MIT
github.com/davecgh/go-spew/spew	ISC
github.com/ghodss/yaml	MIT
github.com/go-faster/city	MIT
github.com/go-faster/errors	BSD-3-Clause
github.com/go-logr/logr	Apache-2.0
github.com/go-logr/stdr	Apache-2.0
github.com/goccy/go-json	MIT
github.com/google/flatbuffers/go	Apache-2.0
github.com/google/uuid	BSD-3-Clause
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors	Apache-2.0
github.com/grpc-ecosystem/grpc-gateway/v2	BSD-3-Clause
github.com/hashicorp/go-cleanhttp	MPL-2.0
github.com/hashicorp/go-retryablehttp	MPL-2.0
github.com/invopop/jsonschema	MIT
github.com/klauspost/compress	Apache-2.0
github.com/klauspost/compress/internal/snapref	BSD-3-Clause
github.com/klauspost/compress/zstd/internal/xxhash	MIT
github.com/mailru/easyjson	MIT
github.com/mattn/go-colorable	MIT
github.com/mattn/go-isatty	MIT
github.com/oapi-codegen/runtime	Apache-2.0
github.com/paulmach/orb	MIT
github.com/pierrec/lz4/v4	BSD-3-Clause
github.com/pkg/errors	BSD-2-Clause
github.com/pmezard/go-difflib/difflib	BSD-3-Clause
github.com/rs/zerolog	MIT
github.com/santhosh-tekuri/jsonschema/v6	Apache-2.0
github.com/segmentio/asm	MIT
github.com/shopspring/decimal	MIT
github.com/spf13/cobra	Apache-2.0
github.com/spf13/pflag	BSD-3-Clause
github.com/stretchr/testify	MIT
github.com/thoas/go-funk	MIT
github.com/wk8/go-ordered-map/v2	Apache-2.0
github.com/zeebo/xxh3	BSD-2-Clause
go.opentelemetry.io/otel	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp	Apache-2.0
go.opentelemetry.io/otel/log	Apache-2.0
go.opentelemetry.io/otel/metric	Apache-2.0
go.opentelemetry.io/otel/sdk	Apache-2.0
go.opentelemetry.io/otel/sdk/log	Apache-2.0
go.opentelemetry.io/otel/sdk/metric	Apache-2.0
go.opentelemetry.io/otel/trace	Apache-2.0
go.opentelemetry.io/proto/otlp	Apache-2.0
golang.org/x/exp	BSD-3-Clause
golang.org/x/net	BSD-3-Clause
golang.org/x/sync/errgroup	BSD-3-Clause
golang.org/x/sys	BSD-3-Clause
golang.org/x/text	BSD-3-Clause
golang.org/x/xerrors	BSD-3-Clause
google.golang.org/genproto/googleapis/api/httpbody	Apache-2.0
google.golang.org/genproto/googleapis/rpc/status	Apache-2.0
google.golang.org/grpc	Apache-2.0
google.golang.org/protobuf	BSD-3-Clause
gopkg.in/yaml.v2	Apache-2.0
gopkg.in/yaml.v3	MIT

Loading plugin documentation

CloudQuery

Test CloudQuery's capabilities with a demo

Learning Center

ClickHouse

Overview #

ClickHouse source plugin

Configuration #

Example #

ClickHouse spec #

Connecting to ClickHouse Cloud #

Verbose logging for debug

Types #

Apache Arrow type conversion #

Licenses #

Overview #

ClickHouse source plugin

Configuration #

Example #

ClickHouse spec #

Connecting to ClickHouse Cloud #

Verbose logging for debug

Types #

Apache Arrow type conversion #

Licenses #