Databricks destination plugin

Configuration #

Example #

kind: destination
spec:
  name: "databricks"
  path: "cloudquery/databricks"
  registry: "cloudquery"
  version: "v1.3.8"
  write_mode: "append"
  spec:
    hostname: ${DATABRICKS_HOSTNAME} # optionally it can include protocol like https://abc.cloud.databricks.com
    http_path: ${DATABRICKS_HTTP_PATH} # HTTP path for SQL compute
    staging_path: ${DATABRICKS_STAGING_PATH} # Databricks FileStore or Unity volume path to store temporary files for staging
    auth:
      access_token: ${DATABRICKS_ACCESS_TOKEN}
    # Optional parameters
    # protocol: https
    # port: 443
    # catalog: ""
    # schema: "default"
    # migration_concurrency: 10
    # timeout: 1m
    # batch:
    #   size: 10000
    #   bytes: 5242880 # 5 MiB
    #   timeout: 20s

Databricks spec #

• hostname (string) (required)

  SQL compute hostname. May optionally include protocol value as well (like https://server.databricks.com).
• http_path (string) (required)

  SQL compute HTTP path.
• staging_path (string) (required)

  Unity volume path where temporary (staging) files should be uploaded to.
• auth (Auth spec) (required)

  Authentication options.
• catalog (string) (required)

  Catalog to be used.
• protocol (string) (optional) (default: https)

  Protocol for connecting to Databricks. Can be also specified in the hostname.
• port (integer) (optional) (default: 443)

  Port for connecting to Databricks.
• schema (string) (optional) (default: cloudquery)

  Schema to be used. If it doesn't exist, it will be created.
• batch (Batching spec) (optional)

  Batching options.
• migration_concurrency (integer) (optional) (default: 10)

  How many table operations will be performed in parallel during migration.
• timeout (duration) (optional) (default: 1m (= 1 minute))

  Timeout for the queries.

Databricks authentication spec

• access_token (string) (required)

  Personal access token.

Batching spec

• size (integer) (optional) (default: 10000)

  Maximum number of items that may be grouped together to be written in a single write.
• bytes (integer) (optional) (default: 5242880 (= 5 MiB))

  Maximum size of items that may be grouped together to be written in a single write.
• timeout (duration) (optional) (default: 1m (= 1 minute))

  Maximum interval between batch writes.