Set up process #

GCP Source Plugin Configuration Reference

Example #

kind: source
spec:
  # Source spec section
  name: "gcp"
  path: "cloudquery/gcp"
  registry: "cloudquery"
  version: "v17.3.0"
  tables: ["gcp_storage_buckets"]
  destinations: ["postgresql"]
  # GCP Spec
  # Learn more about the configuration options at https://cql.ink/gcp_source
  spec:
    project_ids: ["my-project"]

GCP Spec #

• project_ids ([]string) (default: empty. will use all projects available to the current authenticated account)

  Specify projects to connect to. If either folder_ids or project_filter is specified, these projects will be synced in addition to the projects from the folder/filter.
• service_account_key_json (string) (default: empty)

  GCP service account key content.

  Using service accounts is not recommended, but if it is used it is better to use environment or file variable substitution.
• folder_ids ([]string) (default: empty)

  CloudQuery will sync from all the projects in the specified folders, recursively. folder_ids must be of the format folders/<folder_id> or organizations/<organization_id>. This feature requires the resourcemanager.folders.list permission.

  By default, CloudQuery will also sync from sub-folders recursively (up to depth 100). To reduce this, set folder_recursion_depth to a lower value (or to 0 to disable recursion completely).

  Mutually exclusive with project_filter.

  If you specify * then all folders in all organizations will be synced.
• folder_recursion_depth (integer) (default: 100)

  The maximum depth to recurse into sub-folders. 0 means no recursion (only the top-level projects in folders will be used for sync).
• project_filter (string) (default: empty)

  A filter to determine the projects that are synced, mutually exclusive with folder_ids.

  For instance, to only sync projects where the name starts with how-, set project_filter to name:how-*.

  More examples:

  • "name:how-* OR name:test-*" matches projects starting with how- or test-
  • "NOT name:test-*" matches all projects not starting with test-

  For syntax and example queries refer to API References here and here.
• organization_ids ([]string) (default: empty. will use all organizations available to the current authenticated account)

  Specify organizations to use when syncing organization level resources (e.g. folders or security findings).

  If organization_filter is specified, these organizations will be used in addition to the organizations from the filter.
• organization_filter (string) (default: empty)

  A filter to determine the organizations to use when syncing organization level resources (e.g. folders or security findings).

  For instance, to use only organizations from the cloudquery.io domain, set organization_filter to domain:cloudquery.io.

  For syntax and example queries refer to API Reference here.
• backoff_retries (integer) (default: 5)

  Maximum number of retries to make when rate limited.
• backoff_delay (integer) (default: 30)

  Maximum delay in seconds between retries when rate limited.
• enabled_services_only (boolean) (default: false)

  If enabled CloudQuery will skip any resources that belong to a service that has been disabled or not been enabled.

  If you use this option on a large organization (with more than 500 projects) you should also set the backoff_retries to a value greater than 0, otherwise you may hit the API rate limits.

  In >=v9.0.0 if an error is returned then CloudQuery will assume that all services are enabled and will continue to attempt to sync all specified tables rather than just ending the sync.
• concurrency (integer) (default: 50000)

  The best effort maximum number of Go routines to use. Lower this number to reduce memory usage.
• discovery_concurrency (integer) (default: 100)

  The number of concurrent requests that CloudQuery will make to resolve enabled services. This is only used when enabled_services_only is set to true.
• scheduler (string) (default: round-robin)

  The scheduler to use when determining the priority of resources to sync. Supported values are dfs (depth-first search), round-robin, shuffle and shuffle-queue.

  For more information about this, see performance tuning.
• service_account_impersonation (Service Account Impersonation spec, optional. Default: empty)

  Service Account impersonation configuration.
• table_options (map) (default: not used)

  Table options is a premium feature. Even if some tables are free, syncing data for them (& their relations) using table options counts towards paid usage.

  Please refer to the Table Options documentation for more information.

Service Account Impersonation Spec #

• target_principal (string) (required)

  The email address of the service account to impersonate.
• scopes ([]string) (default: ["https://www.googleapis.com/auth/cloud-platform"])

  Scopes that the impersonated credential should have.

  See available scopes in the documentation.
• delegates ([]string) (default: empty)

  Delegates are the service account email addresses in a delegation chain. Each service account must be granted roles/iam.serviceAccountTokenCreator on the next service account in the chain.
• subject (string) (default: empty)

  The subject field of a JWT (sub). This field should only be set if you wish to impersonate a user. This feature is useful when using domain wide delegation.

GCP + Kubernetes (GKE) #

kind: source
spec:
  name: gcp
  path: "cloudquery/gcp"
  registry: cloudquery
  version: "v17.3.0"
  tables: ["gcp_container_clusters"]
  destinations: ["<destination>"]
---
kind: source
spec:
  name: k8s
  path: "cloudquery/k8s"
  registry: cloudquery
  version: "v7.3.0"
  tables: ["*"]
  destinations: ["<destination>"]

WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.26+; use gcloud instead.

What does this mean for CloudQuery users? #

Install #

gcloud components install gke-gcloud-auth-plugin

sudo apt-get install google-cloud-sdk-gke-gcloud-auth-plugin

Verify #

gke-gcloud-auth-plugin --version

gke-gcloud-auth-plugin.exe --version

Switch authentication methods #

export USE_GKE_GCLOUD_AUTH_PLUGIN=True

gcloud components update

gcloud container clusters get-credentials {$CLUSTER_NAME}