Back to destination list
Official
BigQuery destination integration documentation
The BigQuery plugin syncs data from any CloudQuery source plugin(s) to a BigQuery database running on Google Cloud Platform
Loading plugin documentation
The BigQuery plugin syncs data from any CloudQuery source plugin(s) to a BigQuery database running on Google Cloud Platform
Loading plugin documentation
We use tracking cookies to understand how you use the product and help us improve it. Please accept cookies to help us improve. You can always opt out later via the link in the footer.
sync.kind: destination
spec:
name: bigquery
path: cloudquery/bigquery
registry: cloudquery
version: "v4.6.3"
write_mode: "append"
send_sync_summary: true
# Learn more about the configuration options at https://cql.ink/bigquery_destination
spec:
project_id: ${PROJECT_ID}
dataset_id: ${DATASET_ID}
# Optional parameters
# service_account_key_json: ${file:./path-to-your-file.json} # GCP service account key, can be placed in a file and referenced with this syntax.
# dataset_location: ""
# time_partitioning: none # options: "none", "hour", "day", "month", "year"
# time_partitioning_expiration: 0 # duration, e.g. "24h" or "720h" (30 days)
# service_account_key_json: ""
# endpoint: ""
# batch_size: 10000
# batch_size_bytes: 5242880 # 5 MiB
# batch_timeout: 10s
# client_project_id: "*detect-project-id*"
PROJECT_ID - The Google Cloud Project IDDATASET_ID - The Google Cloud BigQuery Dataset IDclient_project_id variable can be used to run BigQuery queries in a project different from where the destination table is located.
If you set client_project_id to *detect-project-id*, it will automatically detect the project ID from the environment variable or application default credentials.batch_size and batch_size_bytes.append write mode.gcloud auth application-default login (recommended when running locally)GOOGLE_APPLICATION_CREDENTIALS. (Not recommended as long-lived keys are a security risk)project_id (string) (required)dataset_id (string) (required)my_dataset.
This dataset needs to be created before running a sync or migration.dataset_location (string) (optional)time_partitioning (string) (options: none, hour, day) (default: none)_cq_sync_time so that all rows for a sync run will be partitioned on the hour/day the sync started.time_partitioning_expiration (duration) (optional)3600s, 60m, 24h, 720h.
This option is only valid if time_partitioning is set a value other than none.service_account_key_json (string) (optional) (default: empty).endpoint (string) (optional)batch_size (integer) (optional) (default: 10000)batch_size_bytes (integer) (optional) (default: 5242880 (5 MiB))batch_timeout (duration) (optional) (default: 10s (10 seconds))text_embeddings (object) (optional)remote_model_name field.remote_model_name (string) (required)tables (array) (required)source_table_name (string) (required)target_table_name (string) (required)embed_columns (array) (required)metadata_columns (array) (optional)_cq_id is always included.text_splitter (object) (optional)recursive_text is supported.recursive_text (object) (required)chunk_size (integer) (required)1000.chunk_overlap (integer) (required)100.v3.0.0 and later) supports most Apache Arrow
types. The following table shows the supported types and how they are mapped
to BigQuery data types.REPEATED columns to represent lists, lists of
lists are not supported right now.