Yandex Cloud

The Yandex Cloud plugin extracts information from many of the supported services by Yandex Cloud and loads it into any supported CloudQuery destination.

Publisher

yandex-cloud

Repositorygithub.com

Latest version

v1.2.0

Type

Source

Platforms

Date Published

August 13, 2024

Price

Free

Documentation Tables Changelog Destinations

Overview Configuration

Set up process #

brew install cloudquery/tap/cloudquery

1. Download CLI and login

See installation options

2. Create source and destination configs

Plugin configuration

cloudquery sync yc.yml postgresql.yml

3. Run the sync

CloudQuery sync

Overview #

Yandex Cloud Source Plugin

The CloudQuery Yandex Cloud plugin pulls configuration out of Yandex Cloud resources and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more).

Authentication #

Authentication is done via IAM tokens/keys. You can read more about it here

It is recommended to grant read-only permissions/roles:

auditor role – primitive role which gives access to all configurations, but not the data.
*.viewer roles – per-sevice roles to read resource. Sometimes it is necessary to grant this role, if the service doesn't support auditor role.

You can read more about roles here

The plugin will get the following environment variables:

YC_SERVICE_ACCOUNT_KEY – Service Account key (in json format)
YC_TOKEN – IAM or OAuth token

If none of the variables are set, plugin will try to use Compute Metadata API to get IAM token.

Query Examples #

Find all compute instances having public IPv4 address #

with instances as (select yci.*,
                          iface #>> '{primary_v4_address, one_to_one_nat, address}' address
                   from (select id,
                                folder_id,
                                name,
                                labels,
                                fqdn,
                                jsonb_array_elements(network_interfaces) as iface
                         from yc_compute_instances) yci)
select *
from instances
where address is not null

Find all public object storage buckets (access flags only) #

with buckets as (select *,
                        (anonymous_access_flags #> '{list, value}')::bool        as flags_list,
                        (anonymous_access_flags #> '{read, value}')::bool        as flags_read,
                        (anonymous_access_flags #> '{config_read, value}')::bool as flags_config_read
                 from yc_storage_buckets)
select *
from buckets
where flags_list
   or flags_read
   or flags_config_read

Configuration #

Yandex Cloud Source Plugin Configuration Reference

Example #

kind: source
spec:
  name: "yc"
  registry: cloudquery
  path: yandex-cloud/yc
  version: "v1.0.0"
  destinations: ["postgresql"]
  tables: 
    ["*"]
  spec:
    organization_ids: # sync these organizations only
      - bpf...
    cloud_ids: # sync these clouds only
      - b1g... 
    folder_ids: # sync these folders only
      - b1g... 
---
kind: destination
spec:
  name: "postgresql"
  path: "cloudquery/postgresql"
  registry: "cloudquery"
  version: "v7.1.2"
  spec:
    connection_string: "${PG_CONNECTION_STRING}"

YC Spec #

organization_ids ([]string, optional, default: empty): List of Organization IDs to target. If empty, all available Organization will be targeted.
cloud_ids ([]string, optional, default: empty): List of Cloud IDs to target. If empty, all available Clouds will be targeted.
folder_ids ([]string, optional, default: empty): List of Folder IDs to target. If empty, all available Folders will be targeted.
debug_grpc (bool, default false): If true, will log all GRPC calls
max_retries (int, default 3): Maxiumum number of retries for YC Client
endpoint (string, default api.cloud.yandex.net:443): Yandex Cloud endpoint
concurrency (int, optional, default: 10000): A best effort maximum number of Go routines to use. Lower this number to reduce memory usage.
scheduler (string, default: shuffle): The scheduler to use when determining the priority of resources to sync. Currently, the only supported values are dfs (depth-first search), round-robin and shuffle.