BigQuery Destination Plugin

Before you begin #

1. Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.
2. Create a BigQuery dataset that will contain the tables synced by CloudQuery. CloudQuery will automatically create the tables as part of a migration run on the first sync.
3. Ensure that you have write access to the dataset. See Required Permissions for details.

Example config #

kind: destination
spec:
  name: bigquery
  path: cloudquery/bigquery
  registry: cloudquery
  version: "v4.1.18"
  write_mode: "append"
  # Learn more about the configuration options at https://cql.ink/bigquery_destination
  spec:
    project_id: ${PROJECT_ID}
    dataset_id: ${DATASET_ID}
    # Optional parameters
    # dataset_location: ""
    # time_partitioning: none # options: "none", "hour", "day"
    # service_account_key_json: ""
    # endpoint: ""
    # batch_size: 10000
    # batch_size_bytes: 5242880 # 5 MiB
    # batch_timeout: 10s

Authentication #

BigQuery Spec #

• project_id (string) (required)

  The id of the project where the destination BigQuery database resides.

• dataset_id (string) (required)

• dataset_location (string) (optional)

• time_partitioning (string) (options: none, hour, day) (default: none)

• service_account_key_json (string) (optional) (default: empty).

  GCP service account key content. This allows for using different service accounts for the GCP source and BigQuery destination. If using service account keys, it is best to use environment or file variable substitution.
• endpoint (string) (optional)

  The BigQuery API endpoint to use. This is useful for testing against a local emulator.
• batch_size (integer) (optional) (default: 10000)

  Number of records to write before starting a new object.
• batch_size_bytes (integer) (optional) (default: 5242880 (5 MiB))

  Number of bytes (as Arrow buffer size) to write before starting a new object.
• batch_timeout (duration) (optional) (default: 10s (10 seconds))

  Maximum interval between batch writes.

Underlying library #