New
Join our webinar! Building a customizable and extensible cloud asset inventory at scale
Back to source list
shopify
Official
Premium

Shopify

The CloudQuery Shopify plugin pulls data from Shopify and loads it into any supported CloudQuery destination

Publisher

cloudquery

Latest version

v8.4.4

Type

Source

Platforms
Date Published

Price per 1M rows

Starting from $17

monthly free quota

1M rows

Overview #

The CloudQuery Shopify plugin pulls data from Shopify and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more).

Authentication #

In order to fetch information from Shopify, cloudquery needs to be authenticated. Either an API key and password (in the case of basic custom/private apps) or an access token (for OAuth apps) is required for authentication.
Refer to the Shopify Help Center article on Custom apps and create a custom app. Follow Get the API credentials for a custom app section to get the credentials for Admin API and put them in your plugin configuration as api_key and api_secret.
If you have a large or busy store, API key/secret type credentials might not be enough due to the heavy rate limiting. In this case, you can use OAuth in your custom app to get an access token which allow many more requests a second. To use that token in your plugin configuration instead, just set it in access_token and remove api_key and api_secret sections. For more information, refer to Shopify.dev on the subject.

Incremental Syncing #

The Shopify plugin supports incremental syncing. This means that only new data will be fetched from Shopify and loaded into your destination for supported tables (support depending on API endpoint). This is done by keeping track of the last item fetched and only fetching data that has been created since then. To enable this, backend_options must be set in the spec (as shown below). This is documented in the Managing Incremental Tables section.

Example Configuration #

This example syncs from Shopify to a Postgres destination. The (top level) source spec section is described in the Source Spec Reference. Incremental syncing is enabled and will be saved to a cq_state_shopify table by default.
kind: source
# Common source-plugin configuration
spec:
  name: shopify
  path: cloudquery/shopify
  registry: cloudquery
  version: "v8.4.4"
  tables: ["*"]
  destinations: ["postgresql"]
  backend_options:
    table_name: "cq_state_shopify"
    connection: "@@plugins.postgresql.connection"
  # Shopify specific configuration
  # Learn more about the configuration options at https://cql.ink/shopify_source
  spec:
    # required, or alternatively use access_token
    api_key: "${SHOPIFY_API_KEY}"
    # required, or alternatively use access_token
    api_secret: "${SHOPIFY_API_SECRET}"
    # required, e.g. https://mystore.myshopify.com
    shop_url: "${SHOPIFY_SHOP_URL}"

Configuration Reference #

This is the (nested) spec used by the Shopify source plugin:
  • api_key (string) (required if access_token isn't used)
    The API Key for your custom app in your store.
  • api_secret (string) (required if access_token isn't used)
    The API Secret for your custom app in your store.
  • access_token (string) (required if api_key & api_secret aren't used)
    An access token for your Shopify custom app. This is an alternative way of authenticating, use either this or the ones above.
  • shop_url (string) (required)
    The URL of your Shopify store, e.g. https://mystore.myshopify.com.
  • api_version (string) (optional) (default: 2023-10)
    The Shopify Admin API version to use. See here for more information.
  • timeout_secs (integer) (optional) (default: 10)
    Timeout (in seconds) for requests against the Shopify Admin API.
  • max_retries (integer) (optional) (default: 30)
    Number of retries if a request was rate limited.
  • page_size (integer) (optional) (default: 50)
    Maximum number of items queried each request. Find an optimum value to balance amount of data fetched and requests timing out. Maximum value 250.
  • max_pages (integer) (optional)
    If set, stop after fetching this many pages for each resource. Useful for debugging.
  • concurrency (integer) (optional) (default: 1000)
    A best effort maximum number of Go routines to use. Lower this number to reduce memory usage.
  • scheduler (string) (optional) (default: dfs) The scheduler to use when determining the priority of resources to sync. Supported values are dfs (depth-first search), round-robin, shuffle and shuffle-queue.
    For more information about this, see performance tuning.

Query Examples

Get all your active products with a specific tag #

SELECT * FROM shopify_products WHERE status='active' AND 'your-tag' = ANY(tags);


Licenses #

The following tools / packages are used in this plugin:
NameLicense
github.com/adrg/xdgMIT
github.com/apache/arrow/go/v13Apache-2.0
github.com/apache/arrow-go/v18Apache-2.0
github.com/apapsch/go-jsonmerge/v2MIT
github.com/aws/aws-sdk-go-v2Apache-2.0
github.com/aws/aws-sdk-go-v2/configApache-2.0
github.com/aws/aws-sdk-go-v2/credentialsApache-2.0
github.com/aws/aws-sdk-go-v2/feature/ec2/imdsApache-2.0
github.com/aws/aws-sdk-go-v2/internal/configsourcesApache-2.0
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/iniApache-2.0
github.com/aws/aws-sdk-go-v2/internal/sync/singleflightBSD-3-Clause
github.com/aws/aws-sdk-go-v2/service/internal/accept-encodingApache-2.0
github.com/aws/aws-sdk-go-v2/service/internal/presigned-urlApache-2.0
github.com/aws/aws-sdk-go-v2/service/licensemanagerApache-2.0
github.com/aws/aws-sdk-go-v2/service/marketplacemeteringApache-2.0
github.com/aws/aws-sdk-go-v2/service/ssoApache-2.0
github.com/aws/aws-sdk-go-v2/service/ssooidcApache-2.0
github.com/aws/aws-sdk-go-v2/service/stsApache-2.0
github.com/aws/smithy-goApache-2.0
github.com/aws/smithy-go/internal/sync/singleflightBSD-3-Clause
github.com/bahlo/generic-list-goBSD-3-Clause
github.com/bold-commerce/go-shopify/v4MIT
github.com/buger/jsonparserMIT
github.com/cenkalti/backoff/v4MIT
github.com/cloudquery/cloudquery-api-goMPL-2.0
github.com/cloudquery/plugin-pb-goMPL-2.0
github.com/cloudquery/plugin-sdk/v2/internal/globMIT
github.com/cloudquery/plugin-sdk/v2/schemaMIT
github.com/cloudquery/plugin-sdk/v2/typesMPL-2.0
github.com/cloudquery/plugin-sdk/v4MPL-2.0
github.com/cloudquery/plugin-sdk/v4/globMIT
github.com/cloudquery/plugin-sdk/v4/scalarMIT
github.com/davecgh/go-spew/spewISC
github.com/ghodss/yamlMIT
github.com/go-logr/logrApache-2.0
github.com/go-logr/stdrApache-2.0
github.com/goccy/go-jsonMIT
github.com/google/flatbuffers/goApache-2.0
github.com/google/go-querystring/queryBSD-3-Clause
github.com/google/uuidBSD-3-Clause
github.com/gorilla/muxBSD-3-Clause
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptorsApache-2.0
github.com/grpc-ecosystem/grpc-gateway/v2BSD-3-Clause
github.com/hashicorp/go-cleanhttpMPL-2.0
github.com/hashicorp/go-retryablehttpMPL-2.0
github.com/invopop/jsonschemaMIT
github.com/jarcoal/httpmockMIT
github.com/klauspost/compressApache-2.0
github.com/klauspost/compress/internal/snaprefBSD-3-Clause
github.com/klauspost/compress/zstd/internal/xxhashMIT
github.com/mailru/easyjsonMIT
github.com/mattn/go-colorableMIT
github.com/mattn/go-isattyMIT
github.com/oapi-codegen/runtimeApache-2.0
github.com/pierrec/lz4/v4BSD-3-Clause
github.com/pmezard/go-difflib/difflibBSD-3-Clause
github.com/rs/zerologMIT
github.com/samber/loMIT
github.com/santhosh-tekuri/jsonschema/v6Apache-2.0
github.com/shopspring/decimalMIT
github.com/spf13/cobraApache-2.0
github.com/spf13/pflagBSD-3-Clause
github.com/stretchr/testifyMIT
github.com/thoas/go-funkMIT
github.com/wk8/go-ordered-map/v2Apache-2.0
github.com/zeebo/xxh3BSD-2-Clause
go.opentelemetry.io/otelApache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttpApache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttpApache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptraceApache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttpApache-2.0
go.opentelemetry.io/otel/logApache-2.0
go.opentelemetry.io/otel/metricApache-2.0
go.opentelemetry.io/otel/sdkApache-2.0
go.opentelemetry.io/otel/sdk/logApache-2.0
go.opentelemetry.io/otel/sdk/metricApache-2.0
go.opentelemetry.io/otel/traceApache-2.0
go.opentelemetry.io/proto/otlpApache-2.0
golang.org/x/expBSD-3-Clause
golang.org/x/netBSD-3-Clause
golang.org/x/oauth2BSD-3-Clause
golang.org/x/syncBSD-3-Clause
golang.org/x/sysBSD-3-Clause
golang.org/x/textBSD-3-Clause
golang.org/x/time/rateBSD-3-Clause
golang.org/x/xerrorsBSD-3-Clause
google.golang.org/genproto/googleapis/api/httpbodyApache-2.0
google.golang.org/genproto/googleapis/rpc/statusApache-2.0
google.golang.org/grpcApache-2.0
google.golang.org/protobufBSD-3-Clause
gopkg.in/yaml.v2Apache-2.0
gopkg.in/yaml.v3MIT


Join our mailing list

Subscribe to our newsletter to make sure you don't miss any updates.

Legal

© 2024 CloudQuery, Inc. All rights reserved.

We use tracking cookies to understand how you use the product and help us improve it. Please accept cookies to help us improve. You can always opt out later via the link in the footer.