The GCP Source plugin for CloudQuery extracts configuration from a variety of GCP APIs and loads it into any supported CloudQuery destination
Publisher
cloudquery
Latest version
v19.1.1
Type
Source
Platforms
Date Published
Loading plugin documentation
We use tracking cookies to understand how you use the product and help us improve it. Please accept cookies to help us improve. You can always opt out later via the link in the footer.
gcloud auth application-default login (recommended when running locally)GOOGLE_APPLICATION_CREDENTIALS. Highly not recommended as long-lived keys are a security riskresourcemanager.organizations.getresourcemanager.projects.getresourcemanager.projects.listresourcemanager.folders.listresourcemanager.folders.getserviceusage.services.list (if enabled_services_only: true)compute.regions.listroles/viewer role. This provides all necessary permissions and is the simplest option.roles/browser + roles/serviceusage.serviceUsageConsumer instead.select project_id, name from gcp_storage_buckets where uniform_bucket_level_access->>'Enabled' = 'true';
kind: source
spec:
# Source spec section
name: "gcp"
path: "cloudquery/gcp"
registry: "cloudquery"
version: "v19.1.1"
tables: ["gcp_storage_buckets"]
destinations: ["postgresql"]
# GCP Spec
# Learn more about the configuration options at https://cql.ink/gcp_source
spec:
project_ids: ["my-project"]
project_ids ([]string) (default: empty. will use all projects available to the current authenticated account)folder_ids or project_filter is specified, these projects will be synced in addition
to the projects from the folder/filter.service_account_key_json (string) (default: empty)folder_ids ([]string) (default: empty)folder_ids must be of the format folders/<folder_id> or organizations/<organization_id>.
This feature requires the resourcemanager.folders.list permission.sync from sub-folders recursively (up to depth 100).
To reduce this, set folder_recursion_depth to a lower value (or to 0 to disable recursion completely).project_filter.* then all folders in all organizations will be synced.folder_recursion_depth (integer) (default: 100)0 means no recursion (only the top-level projects in folders will be used for sync).project_filter (string) (default: empty)folder_ids.how-, set project_filter to name:how-*."name:how-* OR name:test-*" matches projects starting with how- or test-"NOT name:test-*" matches all projects not starting with test-organization_ids ([]string) (default: empty. will use all organizations available to the current authenticated account)organization_filter is specified, these organizations will be used in addition to the organizations from the filter.organization_filter (string) (default: empty)cloudquery.io domain, set organization_filter to domain:cloudquery.io.backoff_retries (integer) (default: 5)backoff_delay (integer) (default: 30)enabled_services_only (boolean) (default: false)500 projects)
you should also set the backoff_retries to a value greater than 0, otherwise you may hit the API rate limits.>=v9.0.0 if an error is returned then CloudQuery will assume that all services are enabled
and will continue to attempt to sync all specified tables rather than just ending the sync.concurrency (integer) (default: 50000)discovery_concurrency (integer) (default: 100)enabled_services_only is set to true.scheduler (string) (default: round-robin)dfs (depth-first search), round-robin, shuffle and shuffle-queue.service_account_impersonation (Service Account Impersonation spec, optional. Default: empty)table_options (map) (default: not used)target_principal (string) (required)scopes ([]string) (default: ["https://www.googleapis.com/auth/cloud-platform"])delegates ([]string) (default: empty)roles/iam.serviceAccountTokenCreator on the next service account in the chain.subject (string) (default: empty)sub).
This field should only be set if you wish to impersonate a user.
This feature is useful when using domain wide delegation.kind: source
spec:
name: gcp
path: "cloudquery/gcp"
registry: cloudquery
version: "v19.1.1"
tables: ["gcp_container_clusters"]
destinations: ["<destination>"]
---
kind: source
spec:
name: k8s
path: "cloudquery/k8s"
registry: cloudquery
version: "v7.9.10"
tables: ["*"]
destinations: ["<destination>"]
WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.26+; use gcloud instead.
gke-gcloud-auth-plugin from gcloud components on Mac or Windows:gcloud components install gke-gcloud-auth-plugin
sudo apt-get install google-cloud-sdk-gke-gcloud-auth-plugin
gke-gcloud-auth-plugin --version
gke-gcloud-auth-plugin.exe --version
export USE_GKE_GCLOUD_AUTH_PLUGIN=True
gcloud components update
gcloud container clusters get-credentials {$CLUSTER_NAME}
kubectl as normal, and you
should no longer see the warning in the CloudQuery output.kind: source
spec:
name: gcp
path: cloudquery/gcp
registry: cloudquery
version: "v19.1.1-fips"
...
table_options object is as follows:table_options:
<table_name>:
<api_method_name>:
- <input_object>
<input_object> objects should be provided.
The plugin will iterate through these to make multiple API calls.
This is useful for APIs like the Compute AggregatedListInstances method that only supports a single filter per call. For example: table_options:
gcp_compute_instances:
aggregated_list_instances:
- include_all_scopes: true
filter: '(cpuPlatform = "Intel Skylake") AND (scheduling.automaticRestart = true)'
- include_all_scopes: false
filter: '(cpuPlatform = "Intel Broadwell") AND (scheduling.automaticRestart = true)'
table_options:
gcp_compute_instances:
aggregated_list_instances:
- <Compute.AggregatedListInstancesRequest> # PageToken, MaxResults and Project are prohibited
Table Options section of each table in the GCP plugin tables documentation.