Scaleway

A Scaleway source plugin for CloudQuery that loads data from Scaleway to any database, data warehouse or data lake supported by CloudQuery, such as PostgreSQL, BigQuery, Athena, and many more.

Publisher

scaleway

Repositorygithub.com

Latest version

v2.0.0

Type

Source

Platforms

Date Published

November 10, 2023

Price

Free

Documentation Tables Changelog Destinations

Overview

Set up process #

brew install cloudquery/tap/cloudquery

1. Download CLI and login

See installation options

2. Create source and destination configs

Plugin configuration

cloudquery sync scaleway.yml postgresql.yml

3. Run the sync

CloudQuery sync

Overview #

A Scaleway source plugin for CloudQuery that loads data from Scaleway to any database, data warehouse or data lake supported by CloudQuery, such as PostgreSQL, BigQuery, Athena, and many more.

Authentication #

Credentials are used from the default config file and the environment. Get your credentials from the IAM dashboard

Set credentials in environment variables or use the default config file at ~/.config/scw/config.yaml. Config format and locations for other platforms are documented in the SDK Docs

The environment variables to set are:

SCW_ACCESS_KEY
SCW_SECRET_KEY
SCW_DEFAULT_ORGANIZATION_ID

Env vars override config values if both are set. By default all regions and zones are queried.

Incremental Syncing #

The Scaleway plugin supports incremental syncing. This means that only new data will be fetched from Scaleway and loaded into your destination for supported tables (support depending on API endpoint). This is done by keeping track of the last item fetched and only fetching data that has been created since then. To enable this, backend_options must be set in the spec (as shown below). This is documented in the Managing Incremental Tables section.

Configuration #

The following source configuration file will sync all data from Scaleway to a PostgreSQL database. See the CloudQuery Quickstart for more information on how to configure the source and destination.

kind: source
spec:
  name: "scaleway"
  registry: "cloudquery"
  path: "scaleway/scaleway"
  version: "v2.0.0"
  # use this to enable incremental syncing
  # backend_options:
  #   table_name: "cq_state_scaleway"
  #   connection: "@@plugins.postgresql.connection"
  tables: 
    - "*"
  skip_tables:
    - "scaleway_ipfs_volumes"
    - "scaleway_marketplace_image_versions"
  destinations: 
    - "postgresql"
  spec:
    # plugin spec section

Plugin Spec #

regions ([]string) (optional) (default: all regions)
List of regions to query.
zones ([]string) (optional) (default: to all zones)
List of zones to query.
timeout_secs (integer in seconds) (optional) (default: 10)
Timeout for requests against the Scaleway API endpoint.
concurrency (integer) (optional) (default: 1000)
Best effort maximum number of Go routines to use. Lower this number to reduce memory usage.

Development #

Run tests #

make test

Run linter #

make lint

Generate docs #

make gen-docs

Release a new version #

Run git tag v1.0.0 to create a new tag for the release (replace v1.0.0 with the new version number)
Run git push origin v1.0.0 to push the tag to GitHub

Once the tag is pushed, a new GitHub Actions workflow will be triggered to build the release binaries and create the new release on GitHub. To customize the release notes, see the Go releaser changelog configuration docs.

Publish a new version to the Cloudquery Hub #

Tagging a new release should invoke the GitHub Actions workflow to publish to the Cloudquery Hub.

CQ_CI_CLOUDQUERY_API_KEY should be set up in GitHub secrets (Settings -> Security -> Secrets and variables -> Actions) beforehand, with a valid API key.

After publishing the new version, it will show up in the hub.

For more information please refer to the official Publishing a Plugin to the Hub guide.