The GCP Source plugin for CloudQuery extracts configuration from a variety of GCP APIs and loads it into any supported CloudQuery destination
Publisher
cloudquery
Latest version
v21.1.0
Type
Source
Platforms
Date Published
Loading plugin documentation
CloudQuery's use of cookies
We use tracking cookies to understand how you use the product and help us improve it. Your consent is required before we can enable these cookies. You can opt out via the link in the footer.
select project_id, name from gcp_storage_buckets where uniform_bucket_level_access->>'Enabled' = 'true';
gcloud auth application-default login
roles/viewer, roles/serviceusage.serviceUsageViewer and roles/compute.viewer roles to sync from GCP. This provides all necessary permissions and is the simplest option.resourcemanager.organizations.getresourcemanager.projects.getresourcemanager.projects.listresourcemanager.folders.listresourcemanager.folders.getserviceusage.services.list (if enabled_services_only: true)compute.regions.listgcloud auth application-default login to log in.GOOGLE_APPLICATION_CREDENTIALS.project-1 and you want to sync data from project project-2.
When syncing gcp_compute_instances from project-2 the compute.googleapis.com service should be enabled in both project-1 and project-2.
When syncing gcp_compute_instances from project-1 the compute.googleapis.com service should be enabled only in project-1.kind: source
spec:
# Source spec section
name: "gcp"
path: "cloudquery/gcp"
registry: "cloudquery"
version: "v21.1.0"
tables: ["gcp_storage_buckets"]
destinations: ["postgresql"]
# GCP Spec
# Learn more about the configuration options at https://cql.ink/gcp_source
spec:
project_ids: ["my-project"]
enabled_services_only: true # Optional, recommended. If enabled CloudQuery will skip any resources that belong to a service that has been disabled or not been enabled.
# service_account_key_json: ${file:./path-to-your-file.json} # Optional, not recommended (service account impersonation recommended) GCP service account key, can be placed in a file and referenced with this syntax.
# folder_ids: ["folders/<folder_id>", "organizations/<organization_id>"] # Optional: Sync from all projects in the specified folders.
# folder_recursion_depth: 100 # Optional: Recursion depth when syncing from folders.
# project_filter: "" # Optional: Filter projects using a CEL expression. Example: "labels.env == 'prod' && name.startsWith('cloudquery-')"
# organization_ids: ["<organization_id>"] # Optional: Sync from all projects in the specified organizations (default is all visible organizations).
# If organization_filter is specified, this list of user IDs will be used in addition to those matched by the filter.
# organization_filter: "" # Optional: Filter organizations using a CEL expression. Example: "displayName.startsWith('cloudquery-')"
# backoff_retries : 5 # Optional: Number of retries for rate-limited requests (default is 5).
# backoff_delay : 30 # Optional: Minimum delay between retries (default is 30s).
# concurrency: 50000 # Optional: Number of concurrent API requests (default is 50000).
# discovery_concurrency: 100 # Optional: Number of concurrent API discovery requests (default is 100).
# scheduler: "round-robin" # Optional: Scheduler for API requests. Options are "round-robin" (default) "dfs", "shuffle" and "shuffle-queue".
# service_account_impersonation: # Optional: Configuration for impersonating a service account.
# target_principal: "@example.iam.gserviceaccount.com" # Optional: Service account to impersonate.
# scopes: ["https://www.googleapis.com/auth/cloud-platform"] # Optional: OAuth2 scopes to use when impersonating the service account.
# delegates: [] # Optional: Service accounts in the delegation chain.
# subject: "" # Optional: The subject field of a JWT (sub).
# table_options: # Optional: Table-specific options (map).
# gcp_compute_instances:
# aggregated_list_instances:
# - include_all_scopes: true
# filter: '(cpuPlatform = "Intel Skylake") AND (scheduling.automaticRestart = true)'
# - include_all_scopes: false
# filter: '(cpuPlatform = "Intel Broadwell") AND (scheduling.automaticRestart = true)'
project_ids ([]string) (default: empty. will use all projects available to the current authenticated account)folder_ids or project_filter is specified, these projects will be synced in addition
to the projects from the folder/filter.service_account_key_json (string) (default: empty)folder_ids ([]string) (default: empty)folder_ids must be of the format folders/<folder_id> or organizations/<organization_id>.
This feature requires the resourcemanager.folders.list permission.sync from sub-folders recursively (up to depth 100).
To reduce this, set folder_recursion_depth to a lower value (or to 0 to disable recursion completely).project_filter.* then all folders in all organizations will be synced.folder_recursion_depth (integer) (default: 100)0 means no recursion (only the top-level projects in folders will be used for sync).project_filter (string) (default: empty)folder_ids.how-, set project_filter to name:how-*."name:how-* OR name:test-*" matches projects starting with how- or test-"NOT name:test-*" matches all projects not starting with test-organization_ids ([]string) (default: empty. will use all organizations available to the current authenticated account)organization_filter is specified, these organizations will be used in addition to the organizations from the filter.organization_filter (string) (default: empty)cloudquery.io domain, set organization_filter to domain:cloudquery.io.backoff_retries (integer) (default: 5)backoff_delay (integer) (default: 30)request_timeout (integer) (default: 60)enabled_services_only (boolean) (default: false)500 projects)
you should also set the backoff_retries to a value greater than 0, otherwise you may hit the API rate limits.>=v9.0.0 if an error is returned then CloudQuery will assume that all services are enabled
and will continue to attempt to sync all specified tables rather than just ending the sync.strict_enabled_services_validation (boolean) (default: false)enabled_services_only.
When this flag is enabled, enabled_services_only must also be enabled.
If service enablement status cannot be determined, the table will be skipped instead of being processed.concurrency (integer) (default: 50000)discovery_concurrency (integer) (default: 100)enabled_services_only is set to true.scheduler (string) (default: round-robin)dfs (depth-first search), round-robin, shuffle and shuffle-queue.service_account_impersonation (Service Account Impersonation spec, optional. Default: empty)table_options (map) (default: not used)target_principal (string) (required)scopes ([]string) (default: ["https://www.googleapis.com/auth/cloud-platform"])delegates ([]string) (default: empty)roles/iam.serviceAccountTokenCreator on the next service account in the chain.subject (string) (default: empty)sub).
This field should only be set if you wish to impersonate a user.
This feature is useful when using domain wide delegation.kind: source
spec:
name: gcp
path: "cloudquery/gcp"
registry: cloudquery
version: "v21.1.0"
tables: ["gcp_container_clusters"]
destinations: ["<destination>"]
---
kind: source
spec:
name: k8s
path: "cloudquery/k8s"
registry: cloudquery
version: "v7.10.4"
tables: ["*"]
destinations: ["<destination>"]
WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.26+; use gcloud instead.
gke-gcloud-auth-plugin from gcloud components on Mac or Windows:gcloud components install gke-gcloud-auth-plugin
sudo apt-get install google-cloud-sdk-gke-gcloud-auth-plugin
gke-gcloud-auth-plugin --version
gke-gcloud-auth-plugin.exe --version
export USE_GKE_GCLOUD_AUTH_PLUGIN=True
gcloud components update
gcloud container clusters get-credentials {$CLUSTER_NAME}
kubectl as normal, and you
should no longer see the warning in the CloudQuery output.kind: source
spec:
name: gcp
path: cloudquery/gcp
registry: cloudquery
version: "v21.1.0-fips"
...
table_options object is as follows:table_options:
<table_name>:
<api_method_name>:
- <input_object>
<input_object> objects should be provided.
The plugin will iterate through these to make multiple API calls.
This is useful for APIs like the Compute AggregatedListInstances method that only supports a single filter per call. For example: table_options:
gcp_compute_instances:
aggregated_list_instances:
- include_all_scopes: true
filter: '(cpuPlatform = "Intel Skylake") AND (scheduling.automaticRestart = true)'
- include_all_scopes: false
filter: '(cpuPlatform = "Intel Broadwell") AND (scheduling.automaticRestart = true)'
table_options:
gcp_compute_instances:
aggregated_list_instances:
- <Compute.AggregatedListInstancesRequest> # PageToken, MaxResults and Project are prohibited
Table Options section of each table in the GCP plugin tables documentation.