We use tracking cookies to understand how you use the product and help us improve it. Please accept cookies to help us improve. You can always opt out later via the link in the footer.
gcloud auth application-default login
(recommended when running locally)GOOGLE_APPLICATION_CREDENTIALS
. Highly not recommended as long-lived keys are a security riskresourcemanager.organizations.get
resourcemanager.projects.get
resourcemanager.projects.list
resourcemanager.folders.list
resourcemanager.folders.get
serviceusage.services.list
(if enabled_services_only: true
)compute.regions.list
roles/viewer
role. This provides all necessary permissions and is the simplest option.roles/browser
+ roles/serviceusage.serviceUsageConsumer
instead.select project_id, name from gcp_storage_buckets where uniform_bucket_level_access->>'Enabled' = 'true';
kind: source
spec:
# Source spec section
name: "gcp"
path: "cloudquery/gcp"
registry: "cloudquery"
version: "v18.20.0"
tables: ["gcp_storage_buckets"]
destinations: ["postgresql"]
# GCP Spec
# Learn more about the configuration options at https://cql.ink/gcp_source
spec:
project_ids: ["my-project"]
project_ids
([]string
) (default: empty. will use all projects available to the current authenticated account)folder_ids
or project_filter
is specified, these projects will be synced in addition
to the projects from the folder/filter.service_account_key_json
(string
) (default: empty)folder_ids
([]string
) (default: empty)folder_ids
must be of the format folders/<folder_id>
or organizations/<organization_id>
.
This feature requires the resourcemanager.folders.list
permission.sync
from sub-folders recursively (up to depth 100
).
To reduce this, set folder_recursion_depth
to a lower value (or to 0
to disable recursion completely).project_filter
.*
then all folders in all organizations will be synced.folder_recursion_depth
(integer
) (default: 100
)0
means no recursion (only the top-level projects in folders will be used for sync).project_filter
(string
) (default: empty)folder_ids
.how-
, set project_filter
to name:how-*
."name:how-* OR name:test-*"
matches projects starting with how-
or test-
"NOT name:test-*"
matches all projects not starting with test-
organization_ids
([]string
) (default: empty. will use all organizations available to the current authenticated account)organization_filter
is specified, these organizations will be used in addition to the organizations from the filter.organization_filter
(string
) (default: empty)cloudquery.io
domain, set organization_filter
to domain:cloudquery.io
.backoff_retries
(integer
) (default: 5
)backoff_delay
(integer
) (default: 30
)enabled_services_only
(boolean
) (default: false
)500
projects)
you should also set the backoff_retries
to a value greater than 0
, otherwise you may hit the API rate limits.>=v9.0.0
if an error is returned then CloudQuery will assume that all services are enabled
and will continue to attempt to sync all specified tables rather than just ending the sync.concurrency
(integer
) (default: 50000
)discovery_concurrency
(integer
) (default: 100
)enabled_services_only
is set to true
.scheduler
(string
) (default: round-robin
)dfs
(depth-first search), round-robin
, shuffle
and shuffle-queue
.service_account_impersonation
(Service Account Impersonation spec, optional. Default: empty)table_options
(map
) (default: not used)target_principal
(string
) (required)scopes
([]string
) (default: ["https://www.googleapis.com/auth/cloud-platform"]
)delegates
([]string
) (default: empty)roles/iam.serviceAccountTokenCreator
on the next service account in the chain.subject
(string
) (default: empty)sub
).
This field should only be set if you wish to impersonate a user.
This feature is useful when using domain wide delegation.kind: source
spec:
name: gcp
path: "cloudquery/gcp"
registry: cloudquery
version: "v18.20.0"
tables: ["gcp_container_clusters"]
destinations: ["<destination>"]
---
kind: source
spec:
name: k8s
path: "cloudquery/k8s"
registry: cloudquery
version: "v7.9.6"
tables: ["*"]
destinations: ["<destination>"]
WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.26+; use gcloud instead.
gke-gcloud-auth-plugin
from gcloud components
on Mac or Windows:gcloud components install gke-gcloud-auth-plugin
sudo apt-get install google-cloud-sdk-gke-gcloud-auth-plugin
gke-gcloud-auth-plugin --version
gke-gcloud-auth-plugin.exe --version
export USE_GKE_GCLOUD_AUTH_PLUGIN=True
gcloud components update
gcloud container clusters get-credentials {$CLUSTER_NAME}
kubectl
as normal, and you
should no longer see the warning in the CloudQuery output.kind: source
spec:
name: gcp
path: cloudquery/gcp
registry: cloudquery
version: "v18.20.0-fips"
...
table_options
object is as follows:table_options:
<table_name>:
<api_method_name>:
- <input_object>
<input_object>
objects should be provided.
The plugin will iterate through these to make multiple API calls.
This is useful for APIs like the Compute AggregatedListInstances
method that only supports a single filter per call. For example: table_options:
gcp_compute_instances:
aggregated_list_instances:
- include_all_scopes: true
filter: '(cpuPlatform = "Intel Skylake") AND (scheduling.automaticRestart = true)'
- include_all_scopes: false
filter: '(cpuPlatform = "Intel Broadwell") AND (scheduling.automaticRestart = true)'
table_options:
gcp_compute_instances:
aggregated_list_instances:
- <Compute.AggregatedListInstancesRequest> # PageToken, MaxResults and Project are prohibited
Table Options
section of each table in the GCP plugin tables documentation.