ClickHouse source plugin

Configuration #

Example #

kind: source
spec:
  name: "clickhouse"
  path: "cloudquery/clickhouse"
  registry: "cloudquery"
  version: "v1.2.4"
  destinations: ["postgresql"]
  tables: ["*"]
  # Learn more about the configuration options at https://cql.ink/clickhouse_source
  spec:
    connection_string: "clickhouse://${CH_USER}:${CH_PASSWORD}@localhost:9000/${CH_DATABASE}"
    # Optional parameters
    # ca_cert: ""
    # rows_per_record: 500
    # concurrency: 100
    # table_options:
    #   test_table:
    #     incremental_column: "created_at"

ClickHouse spec #

• connection_string (string) (required)

  Connection string to connect to the database. See SDK documentation for more details.
• ca_cert (string) (optional) (default: not used)

  PEM-encoded certificate authorities. When set, a certificate pool will be created by appending the certificates to the system pool.

  See file variable substitution for how to read this value from a file.
• rows_per_record (integer) (optional) (default: 500)

  Amount of rows to be packed into a single Apache Arrow record to be sent over the wire during sync.
• concurrency (integer) (optional) (default: 100)

  Number of tables to sync concurrently. Lower or increase this number based on your database size and available resources.
• table_options (object) (optional) (default: not used)

  Table options to set an incremental column.

Connecting to ClickHouse Cloud #

connection_string: "clickhouse://default:${CH_PASSWORD}@<your-server-id>.<region>.<provider>.clickhouse.cloud:9440/${CH_DATABASE}?secure=true"

Verbose logging for debug

kind: source
spec:
  name: "clickhouse"
  path: "cloudquery/clickhouse"
  registry: "cloudquery"
  version: "v1.2.4"
  tables: ["*"]
  spec:
    connection_string: "clickhouse://${CH_USER}:${CH_PASSWORD}@localhost:9000/${CH_DATABASE}?debug=true"