Back to source list
Official
Premium
BigQuery
Sync from BigQuery to any destination
Loading plugin documentation
Sync from BigQuery to any destination
Loading plugin documentation
We use tracking cookies to understand how you use the product and help us improve it. Please accept cookies to help us improve. You can always opt out later via the link in the footer.
kind: source
spec:
name: bigquery
path: cloudquery/bigquery
registry: cloudquery
version: "v1.11.5"
tables: ["*"]
destinations: ["postgresql"]
# Learn more about the configuration options at https://cql.ink/bigquery_source
spec:
project_id: ${PROJECT_ID}
dataset_id: ${DATASET_ID}
# Optional parameters
# dataset_location: ""
# service_account_key_json: ""
# endpoint: ""
project_id (string) (required)dataset_id (string) (required)my_dataset.dataset_location (string) (optional)service_account_key_json (string) (optional) (default: empty).endpoint (string) (optional)table_options (map of Table Options Spec) (optional)queries is empty.queries (list of queries Spec) (optional)tables in top-level spec should be left as * or can be a subset of these tables.destination_table_name (string) (optional) (default: {{TABLE}}){{TABLE}} will be replaced with the table name{{DATASET}} will be replaced with the dataset name{{UUID}} will be replaced with a random UUID in raw format to uniquely identify each table{{YEAR}} will be replaced with the current year in YYYY format{{MONTH}} will be replaced with the current month in MM format{{DAY}} will be replaced with the current day in DD format{{HOUR}} will be replaced with the current hour in HH format{{MINUTE}} will be replaced with the current minute in mm format{{TABLE}} or {{UUID}} is required.UTC and will be the time sync started.concurrency (integer) (optional) (default: 100)discovery_concurrency (integer) (optional) (default: 100)scheduler (string) (optional) (default: dfs)dfs (depth-first search), round-robin, shuffle and shuffle-queue.incremental_column (string) (optional)name (string) (required)query (string) (required){incremental_column} > @cursor or similar in the WHERE clause if incremental_column is set.incremental_column (string) (optional)@cursor in its WHERE clause, and a column with this name in the result set. If empty, no incremental column will be used.incremental_column_type (enum) (required if incremental_column is set)TIMESTAMP, INTEGER or STRING. This is required if incremental_column is set.gcloud auth application-default login (recommended when running locally)GOOGLE_APPLICATION_CREDENTIALS. (Not recommended as long-lived keys are a security risk)incremental_column specified in the table_options section of the configuration. The tables without an incremental_column will be synced fully.incremental_column in the table_options section of the configuration. Here's an example source spec to sync a list of tables incrementally:kind: source
spec:
name: bigquery
path: cloudquery/bigquery
registry: cloudquery
version: "v1.11.5"
tables:
- "my_table"
- "another_table"
- "yet_another_table"
destinations: ["postgresql"]
backend_options:
table_name: "cq_state_bq"
connection: "@@plugins.postgresql.connection"
spec:
project_id: my-project
dataset_id: my_dataset
table_options:
my_table:
incremental_column: updated_at
another_table:
incremental_column: id
my_table table will be synced incrementally based on the updated_at column. another_table will be synced incrementally based on the id column. yet_another_table will be synced fully.cq_state_bq table in the PostgreSQL destination.incremental_column as well as incremental_column_type in the queries Spec. Here's an example source spec sync a custom query incrementally:kind: source
spec:
name: bigquery
path: cloudquery/bigquery
registry: cloudquery
version: "v1.11.5"
tables:
- "my_query_result"
destinations: ["postgresql"]
backend_options:
table_name: "cq_state_bq"
connection: "@@plugins.postgresql.connection"
spec:
project_id: my-project
dataset_id: my_dataset
queries:
- name: my_query_result
query: "SELECT * FROM my_table WHERE updated_at > @cursor"
incremental_column: updated_at
incremental_column_type: timestamp
my_query_result table will be generated from the custom query SELECT * FROM my_table WHERE updated_at > @cursor and will be synced incrementally based on the updated_at column.
@cursor is a placeholder that will be replaced with the last sync time. If no sync time is available, it will be replaced with the positive minimum value of that type.