Set up process #

brew install cloudquery/tap/cloudquery

1. Download CLI and login

See installation options

2. Create source and destination configs

Plugin configuration

cloudquery sync apigee.yml postgresql.yml

3. Run the sync

CloudQuery sync

Overview #

The Apigee Source plugin for CloudQuery extracts information from the Apigee API and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more).

Configuration #

This plugin relies on Google Application Default Credentials for authentication. When running locally you can use gcloud to authenticate. In other environments you must define credentials_file_path or credentials_json in the spec.

The following configuration syncs from Apigee to a Postgres destination. The (top level) source spec section is described in the Source Spec Reference. The config for the postgresql destination is not shown here. See our Quickstart if you need help setting up the destination.

kind: source
spec:
  name: "apigee"
  path: "cloudquery/apigee"
  registry: "cloudquery"
  version: "v1.5.1"
  tables: ["*"]
  destinations:
    - "postgresql"
  # Learn more about the configuration options at https://cql.ink/apigee_source
  spec:

Apigee Spec #

This is the (nested) spec used by the Apigee source plugin:

endpoint_url (string) (optional) (default: https://apigee.googleapis.com/)
URL of the Apigee REST API.
credentials_file_path (string) (optional)
Path to a JSON credential file.
credentials_json (string) (optional)
Same as credentials_file_path except the JSON is directly embedded into the spec.
concurrency (integer) (optional) (default: 50000)
A best effort maximum number of Go routines to use. Lower this number to reduce memory usage.
scheduler (string) (optional) (default: dfs) The scheduler to use when determining the priority of resources to sync. Supported values are dfs (depth-first search), round-robin, shuffle and shuffle-queue.
For more information about this, see performance tuning.

Full config example #

Below is an example of the full config file with all the optional fields and their default values.

kind: source
spec:
  name: "apigee"
  path: "cloudquery/apigee"
  registry: "cloudquery"
  version: "v1.5.1"
  tables: ["*"]
  destinations:
    - "postgresql"
  spec:
    # optional
    endpoint_url: https://apigee.googleapis.com/
    # optional
    credentials_file_path:
    # optional
    credentials_json:
    # optional
    concurrency: 50000