Official

Premium

MySQL

The CloudQuery MySQL plugin syncs your MySQL database to any of the supported CloudQuery destinations

Publisher

cloudquery

Latest version

v4.5.0

Type

Source

Platforms

Date Published

November 20, 2024

Price per 1M rows

Starting from $17

monthly free quota

1M rows

Documentation Tables Changelog Destinations

Overview Incremental-Examples Licenses

Set up process #

brew install cloudquery/tap/cloudquery

1. Download CLI and login

See installation options

2. Create source and destination configs

Plugin configuration

cloudquery sync mysql.yml postgresql.yml

3. Run the sync

CloudQuery sync

Overview #

The CloudQuery MySQL plugin syncs your MySQL database to any of the supported CloudQuery destinations (e.g. PostgreSQL, BigQuery, Snowflake, and more).

Supported database versions are >= 8.0.

If you need support for older versions, please contact us

Example Configuration #

kind: source
spec:
  name: mysql
  path: cloudquery/mysql
  registry: cloudquery
  version: "v4.5.0"
  tables: ["*"]
  destinations: ["postgresql"]
  # Learn more about the configuration options at https://cql.ink/mysql_source
  spec:
    connection_string: "user:password@/dbname"
    # Optional parameters
    # rows_per_record: 500

Configuration Reference #

This is the (nested) spec used by the MySQL destination plugin.

connection_string (string) (required)
Connection string to connect to the database. See the Go driver documentation for more details.
- "user:password@tcp(127.0.0.1:3306)/dbname" connect with TCP
- "user:[email protected]:3306/dbname?charset=utf8mb4\u0026parseTime=True\u0026loc=Local" connect and set character set, time parsing, and location
- "user:password@localhost:3306/dbname?timeout=30s\u0026readTimeout=1s\u0026writeTimeout=1s" connect and set various timeouts
- "user:password@/dbname?loc=UTC\u0026allowNativePasswords=true\u0026tls=preferred" connect and set location and native password allowance, and prefer TLS

Make sure you use environment variable expansion in production instead of committing the credentials to the configuration file directly.

rows_per_record (integer) (optional) (default: 500)
Amount of rows to be packed into a single Apache Arrow record to be sent over the wire during sync.

Table Options Spec #

Enables setting table options for specific tables. Map key is the name of the table in untemplated form.

incremental_column (string) (optional)

Name of the incremental column in the table. If empty, no incremental column will be used. See incremental-examples for more information

Incremental-Examples #

Configuration Examples for Incremental Sync

To sync tables incrementally each table needs to have an incremental_column specified in the table_options section of the configuration. The tables without an incremental_column will be synced fully.

Sync a Table Incrementally #

To sync a table incrementally, you need to specify the incremental_column in the table_options section of the configuration. Here's an example source spec to sync a list of tables incrementally:

kind: source
spec:
  name: mysql
  path: cloudquery/mysql
  registry: cloudquery
  version: "v4.5.0"
  tables:
    - "my_table"
    - "another_table"
    - "yet_another_table"
  destinations: ["postgresql"]
  backend_options:
    table_name: "cq_state_bq"
    connection: "@@plugins.postgresql.connection"
  spec:
    connection_string: "user:password@/dbname"
    # Optional parameters
    # rows_per_record: 500
    table_options:
      my_table:
        incremental_column: updated_at
      another_table:
        incremental_column: id

In the example above, the my_table table will be synced incrementally based on the updated_at column. another_table will be synced incrementally based on the id column. yet_another_table will be synced fully.

The incremental column will be used to fetch only the new or updated rows from the source table and the state will be stored in the cq_state_bq table in the PostgreSQL destination.

For more information about managing state for incremental tables, see Managing Incremental Tables.

Licenses #

The following tools / packages are used in this plugin:

Name	License
github.com/adrg/xdg	MIT
github.com/apache/arrow/go/v13	Apache-2.0
github.com/apache/arrow/go/v17	Apache-2.0
github.com/apapsch/go-jsonmerge/v2	MIT
github.com/aws/aws-sdk-go-v2	Apache-2.0
github.com/aws/aws-sdk-go-v2/config	Apache-2.0
github.com/aws/aws-sdk-go-v2/credentials	Apache-2.0
github.com/aws/aws-sdk-go-v2/feature/ec2/imds	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/configsources	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/ini	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/sync/singleflight	BSD-3-Clause
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/licensemanager	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/marketplacemetering	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/sso	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/ssooidc	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/sts	Apache-2.0
github.com/aws/smithy-go	Apache-2.0
github.com/aws/smithy-go/internal/sync/singleflight	BSD-3-Clause
github.com/cenkalti/backoff/v4	MIT
github.com/cloudquery/cloudquery-api-go	MPL-2.0
github.com/cloudquery/plugin-pb-go	MPL-2.0
github.com/cloudquery/plugin-sdk/v2/internal/glob	MIT
github.com/cloudquery/plugin-sdk/v2/schema	MIT
github.com/cloudquery/plugin-sdk/v2/types	MPL-2.0
github.com/cloudquery/plugin-sdk/v4	MPL-2.0
github.com/cloudquery/plugin-sdk/v4/glob	MIT
github.com/cloudquery/plugin-sdk/v4/scalar	MIT
github.com/davecgh/go-spew/spew	ISC
github.com/ghodss/yaml	MIT
github.com/go-logr/logr	Apache-2.0
github.com/go-logr/stdr	Apache-2.0
github.com/go-sql-driver/mysql	MPL-2.0
github.com/goccy/go-json	MIT
github.com/google/flatbuffers/go	Apache-2.0
github.com/google/uuid	BSD-3-Clause
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors	Apache-2.0
github.com/grpc-ecosystem/grpc-gateway/v2	BSD-3-Clause
github.com/hashicorp/go-cleanhttp	MPL-2.0
github.com/hashicorp/go-retryablehttp	MPL-2.0
github.com/klauspost/compress	Apache-2.0
github.com/klauspost/compress/internal/snapref	BSD-3-Clause
github.com/klauspost/compress/zstd/internal/xxhash	MIT
github.com/mattn/go-colorable	MIT
github.com/mattn/go-isatty	MIT
github.com/oapi-codegen/runtime	Apache-2.0
github.com/pierrec/lz4/v4	BSD-3-Clause
github.com/pmezard/go-difflib/difflib	BSD-3-Clause
github.com/rs/zerolog	MIT
github.com/santhosh-tekuri/jsonschema/v6	Apache-2.0
github.com/spf13/cobra	Apache-2.0
github.com/spf13/pflag	BSD-3-Clause
github.com/stretchr/testify	MIT
github.com/thoas/go-funk	MIT
github.com/zeebo/xxh3	BSD-2-Clause
go.opentelemetry.io/otel	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp	Apache-2.0
go.opentelemetry.io/otel/log	Apache-2.0
go.opentelemetry.io/otel/metric	Apache-2.0
go.opentelemetry.io/otel/sdk	Apache-2.0
go.opentelemetry.io/otel/sdk/log	Apache-2.0
go.opentelemetry.io/otel/sdk/metric	Apache-2.0
go.opentelemetry.io/otel/trace	Apache-2.0
go.opentelemetry.io/proto/otlp	Apache-2.0
golang.org/x/exp	BSD-3-Clause
golang.org/x/net	BSD-3-Clause
golang.org/x/sync/errgroup	BSD-3-Clause
golang.org/x/sys	BSD-3-Clause
golang.org/x/text	BSD-3-Clause
golang.org/x/xerrors	BSD-3-Clause
google.golang.org/genproto/googleapis/api/httpbody	Apache-2.0
google.golang.org/genproto/googleapis/rpc/status	Apache-2.0
google.golang.org/grpc	Apache-2.0
google.golang.org/protobuf	BSD-3-Clause
gopkg.in/yaml.v2	Apache-2.0
gopkg.in/yaml.v3	MIT