Official

Premium

Shopify

The CloudQuery Shopify plugin pulls data from Shopify and loads it into any supported CloudQuery destination

Publisher

cloudquery

Latest version

v8.3.0

Type

Source

Platforms

Date Published

October 22, 2024

Price per 1M rows

Starting from $15

monthly free quota

1M rows

Documentation Tables Changelog Destinations

Overview

Set up process #

brew install cloudquery/tap/cloudquery

1. Download CLI and login

See installation options

2. Create source and destination configs

Plugin configuration

cloudquery sync shopify.yml postgresql.yml

3. Run the sync

CloudQuery sync

Overview #

The CloudQuery Shopify plugin pulls data from Shopify and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more).

Authentication #

In order to fetch information from Shopify, cloudquery needs to be authenticated. Either an API key and password (in the case of basic custom/private apps) or an access token (for OAuth apps) is required for authentication.

Refer to the Shopify Help Center article on Custom apps and create a custom app. Follow Get the API credentials for a custom app section to get the credentials for Admin API and put them in your plugin configuration as api_key and api_secret.

If you have a large or busy store, API key/secret type credentials might not be enough due to the heavy rate limiting. In this case, you can use OAuth in your custom app to get an access token which allow many more requests a second. To use that token in your plugin configuration instead, just set it in access_token and remove api_key and api_secret sections. For more information, refer to Shopify.dev on the subject.

Incremental Syncing #

The Shopify plugin supports incremental syncing. This means that only new data will be fetched from Shopify and loaded into your destination for supported tables (support depending on API endpoint). This is done by keeping track of the last item fetched and only fetching data that has been created since then. To enable this, backend_options must be set in the spec (as shown below). This is documented in the Managing Incremental Tables section.

Example Configuration #

This example syncs from Shopify to a Postgres destination. The (top level) source spec section is described in the Source Spec Reference. Incremental syncing is enabled and will be saved to a cq_state_shopify table by default.

kind: source
# Common source-plugin configuration
spec:
  name: shopify
  path: cloudquery/shopify
  registry: cloudquery
  version: "v8.3.0"
  tables: ["*"]
  destinations: ["postgresql"]
  backend_options:
    table_name: "cq_state_shopify"
    connection: "@@plugins.postgresql.connection"
  # Shopify specific configuration
  # Learn more about the configuration options at https://cql.ink/shopify_source
  spec:
    # required, or alternatively use access_token
    api_key: "${SHOPIFY_API_KEY}"
    # required, or alternatively use access_token
    api_secret: "${SHOPIFY_API_SECRET}"
    # required, e.g. https://mystore.myshopify.com
    shop_url: "${SHOPIFY_SHOP_URL}"

Note that if backend_options is omitted, by default no backend will be used. This will result in all items being fetched on every sync.

For more information about managing state for incremental tables, see Managing Incremental Tables.

Configuration Reference #

This is the (nested) spec used by the Shopify source plugin:

api_key (string) (required if access_token isn't used)
The API Key for your custom app in your store.
api_secret (string) (required if access_token isn't used)
The API Secret for your custom app in your store.
access_token (string) (required if api_key & api_secret aren't used)
An access token for your Shopify custom app. This is an alternative way of authenticating, use either this or the ones above.
shop_url (string) (required)
The URL of your Shopify store, e.g. https://mystore.myshopify.com.
api_version (string) (optional) (default: 2023-10)
The Shopify Admin API version to use. See here for more information.
timeout_secs (integer) (optional) (default: 10)
Timeout (in seconds) for requests against the Shopify Admin API.
max_retries (integer) (optional) (default: 30)
Number of retries if a request was rate limited.
page_size (integer) (optional) (default: 50)
Maximum number of items queried each request. Find an optimum value to balance amount of data fetched and requests timing out. Maximum value 250.
max_pages (integer) (optional)
If set, stop after fetching this many pages for each resource. Useful for debugging.
concurrency (integer) (optional) (default: 1000)
A best effort maximum number of Go routines to use. Lower this number to reduce memory usage.
scheduler (string) (optional) (default: dfs) The scheduler to use when determining the priority of resources to sync. Supported values are dfs (depth-first search), round-robin, shuffle and shuffle-queue.
For more information about this, see performance tuning.

Query Examples

Get all your active products with a specific tag #

SELECT * FROM shopify_products WHERE status='active' AND 'your-tag' = ANY(tags);