Back to source list
Official
Premium
Shopify
The CloudQuery Shopify plugin pulls data from Shopify and loads it into any supported CloudQuery destination
Publisher
cloudquery
Latest version
v8.4.4
Type
Source
Platforms
Date Published
Price per 1M rows
Starting from $17
monthly free quota
1M rows
Overview #
The CloudQuery Shopify plugin pulls data from Shopify and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more).
Authentication #
In order to fetch information from Shopify,
cloudquery
needs to be authenticated. Either an API key and password (in the case of basic custom/private apps) or an access token (for OAuth apps) is required for authentication.Refer to the Shopify Help Center article on Custom apps and create a custom app. Follow Get the API credentials for a custom app section to get the credentials for Admin API and put them in your plugin configuration as
api_key
and api_secret
.If you have a large or busy store, API key/secret type credentials might not be enough due to the heavy rate limiting. In this case, you can use OAuth in your custom app to get an access token which allow many more requests a second. To use that token in your plugin configuration instead, just set it in
access_token
and remove api_key
and api_secret
sections. For more information, refer to Shopify.dev on the subject.Incremental Syncing #
The Shopify plugin supports incremental syncing. This means that only new data will be fetched from Shopify and loaded into your destination for supported tables (support depending on API endpoint). This is done by keeping track of the last item fetched and only fetching data that has been created since then.
To enable this,
backend_options
must be set in the spec (as shown below). This is documented in the Managing Incremental Tables section.Example Configuration #
This example syncs from Shopify to a Postgres destination. The (top level) source spec section is described in the Source Spec Reference. Incremental syncing is enabled and will be saved to a
cq_state_shopify
table by default.kind: source
# Common source-plugin configuration
spec:
name: shopify
path: cloudquery/shopify
registry: cloudquery
version: "v8.4.4"
tables: ["*"]
destinations: ["postgresql"]
backend_options:
table_name: "cq_state_shopify"
connection: "@@plugins.postgresql.connection"
# Shopify specific configuration
# Learn more about the configuration options at https://cql.ink/shopify_source
spec:
# required, or alternatively use access_token
api_key: "${SHOPIFY_API_KEY}"
# required, or alternatively use access_token
api_secret: "${SHOPIFY_API_SECRET}"
# required, e.g. https://mystore.myshopify.com
shop_url: "${SHOPIFY_SHOP_URL}"
Configuration Reference #
This is the (nested) spec used by the Shopify source plugin:
api_key
(string
) (required ifaccess_token
isn't used)The API Key for your custom app in your store.api_secret
(string
) (required ifaccess_token
isn't used)The API Secret for your custom app in your store.access_token
(string
) (required ifapi_key
&api_secret
aren't used)An access token for your Shopify custom app. This is an alternative way of authenticating, use either this or the ones above.shop_url
(string
) (required)The URL of your Shopify store, e.g.https://mystore.myshopify.com
.api_version
(string) (optional) (default:2023-10
)The Shopify Admin API version to use. See here for more information.timeout_secs
(integer
) (optional) (default:10
)Timeout (in seconds) for requests against the Shopify Admin API.max_retries
(integer
) (optional) (default:30
)Number of retries if a request was rate limited.page_size
(integer
) (optional) (default:50
)Maximum number of items queried each request. Find an optimum value to balance amount of data fetched and requests timing out. Maximum value 250.max_pages
(integer
) (optional)If set, stop after fetching this many pages for each resource. Useful for debugging.concurrency
(integer
) (optional) (default:1000
)A best effort maximum number of Go routines to use. Lower this number to reduce memory usage.scheduler
(string
) (optional) (default:dfs
) The scheduler to use when determining the priority of resources to sync. Supported values aredfs
(depth-first search),round-robin
,shuffle
andshuffle-queue
.For more information about this, see performance tuning.
Query Examples
Get all your active products with a specific tag #
SELECT * FROM shopify_products WHERE status='active' AND 'your-tag' = ANY(tags);
Licenses #
The following tools / packages are used in this plugin:
Name | License |
---|---|
github.com/adrg/xdg | MIT |
github.com/apache/arrow/go/v13 | Apache-2.0 |
github.com/apache/arrow-go/v18 | Apache-2.0 |
github.com/apapsch/go-jsonmerge/v2 | MIT |
github.com/aws/aws-sdk-go-v2 | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/config | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/credentials | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/feature/ec2/imds | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/internal/configsources | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/internal/ini | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/internal/sync/singleflight | BSD-3-Clause |
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/licensemanager | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/marketplacemetering | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/sso | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/ssooidc | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/sts | Apache-2.0 |
github.com/aws/smithy-go | Apache-2.0 |
github.com/aws/smithy-go/internal/sync/singleflight | BSD-3-Clause |
github.com/bahlo/generic-list-go | BSD-3-Clause |
github.com/bold-commerce/go-shopify/v4 | MIT |
github.com/buger/jsonparser | MIT |
github.com/cenkalti/backoff/v4 | MIT |
github.com/cloudquery/cloudquery-api-go | MPL-2.0 |
github.com/cloudquery/plugin-pb-go | MPL-2.0 |
github.com/cloudquery/plugin-sdk/v2/internal/glob | MIT |
github.com/cloudquery/plugin-sdk/v2/schema | MIT |
github.com/cloudquery/plugin-sdk/v2/types | MPL-2.0 |
github.com/cloudquery/plugin-sdk/v4 | MPL-2.0 |
github.com/cloudquery/plugin-sdk/v4/glob | MIT |
github.com/cloudquery/plugin-sdk/v4/scalar | MIT |
github.com/davecgh/go-spew/spew | ISC |
github.com/ghodss/yaml | MIT |
github.com/go-logr/logr | Apache-2.0 |
github.com/go-logr/stdr | Apache-2.0 |
github.com/goccy/go-json | MIT |
github.com/google/flatbuffers/go | Apache-2.0 |
github.com/google/go-querystring/query | BSD-3-Clause |
github.com/google/uuid | BSD-3-Clause |
github.com/gorilla/mux | BSD-3-Clause |
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors | Apache-2.0 |
github.com/grpc-ecosystem/grpc-gateway/v2 | BSD-3-Clause |
github.com/hashicorp/go-cleanhttp | MPL-2.0 |
github.com/hashicorp/go-retryablehttp | MPL-2.0 |
github.com/invopop/jsonschema | MIT |
github.com/jarcoal/httpmock | MIT |
github.com/klauspost/compress | Apache-2.0 |
github.com/klauspost/compress/internal/snapref | BSD-3-Clause |
github.com/klauspost/compress/zstd/internal/xxhash | MIT |
github.com/mailru/easyjson | MIT |
github.com/mattn/go-colorable | MIT |
github.com/mattn/go-isatty | MIT |
github.com/oapi-codegen/runtime | Apache-2.0 |
github.com/pierrec/lz4/v4 | BSD-3-Clause |
github.com/pmezard/go-difflib/difflib | BSD-3-Clause |
github.com/rs/zerolog | MIT |
github.com/samber/lo | MIT |
github.com/santhosh-tekuri/jsonschema/v6 | Apache-2.0 |
github.com/shopspring/decimal | MIT |
github.com/spf13/cobra | Apache-2.0 |
github.com/spf13/pflag | BSD-3-Clause |
github.com/stretchr/testify | MIT |
github.com/thoas/go-funk | MIT |
github.com/wk8/go-ordered-map/v2 | Apache-2.0 |
github.com/zeebo/xxh3 | BSD-2-Clause |
go.opentelemetry.io/otel | Apache-2.0 |
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp | Apache-2.0 |
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp | Apache-2.0 |
go.opentelemetry.io/otel/exporters/otlp/otlptrace | Apache-2.0 |
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp | Apache-2.0 |
go.opentelemetry.io/otel/log | Apache-2.0 |
go.opentelemetry.io/otel/metric | Apache-2.0 |
go.opentelemetry.io/otel/sdk | Apache-2.0 |
go.opentelemetry.io/otel/sdk/log | Apache-2.0 |
go.opentelemetry.io/otel/sdk/metric | Apache-2.0 |
go.opentelemetry.io/otel/trace | Apache-2.0 |
go.opentelemetry.io/proto/otlp | Apache-2.0 |
golang.org/x/exp | BSD-3-Clause |
golang.org/x/net | BSD-3-Clause |
golang.org/x/oauth2 | BSD-3-Clause |
golang.org/x/sync | BSD-3-Clause |
golang.org/x/sys | BSD-3-Clause |
golang.org/x/text | BSD-3-Clause |
golang.org/x/time/rate | BSD-3-Clause |
golang.org/x/xerrors | BSD-3-Clause |
google.golang.org/genproto/googleapis/api/httpbody | Apache-2.0 |
google.golang.org/genproto/googleapis/rpc/status | Apache-2.0 |
google.golang.org/grpc | Apache-2.0 |
google.golang.org/protobuf | BSD-3-Clause |
gopkg.in/yaml.v2 | Apache-2.0 |
gopkg.in/yaml.v3 | MIT |