Back to source list
mysql
Official
Premium

MySQL

The CloudQuery MySQL plugin syncs your MySQL database to any of the supported CloudQuery destinations

Publisher

cloudquery

Latest version

v4.5.3

Type

Source

Platforms
Date Published

Price per 1M rows

Starting from $17

monthly free quota

1M rows

Set up process #


brew install cloudquery/tap/cloudquery

1. Download CLI and login

See installation options

2. Create source and destination configs

Plugin configuration

cloudquery sync mysql.yml postgresql.yml

3. Run the sync

CloudQuery sync

Overview #

The CloudQuery MySQL plugin syncs your MySQL database to any of the supported CloudQuery destinations (e.g. PostgreSQL, BigQuery, Snowflake, and more).
Supported database versions are >= 8.0.
If you need support for older versions, please contact us

Example Configuration #

kind: source
spec:
  name: mysql
  path: cloudquery/mysql
  registry: cloudquery
  version: "v4.5.3"
  tables: ["*"]
  destinations: ["postgresql"]
  # Learn more about the configuration options at https://cql.ink/mysql_source
  spec:
    connection_string: "user:password@/dbname"
    # Optional parameters
    # rows_per_record: 500

Configuration Reference #

This is the (nested) spec used by the MySQL destination plugin.
  • connection_string (string) (required)
    Connection string to connect to the database. See the Go driver documentation for more details.
    • "user:password@tcp(127.0.0.1:3306)/dbname" connect with TCP
    • "user:[email protected]:3306/dbname?charset=utf8mb4\u0026parseTime=True\u0026loc=Local" connect and set character set, time parsing, and location
    • "user:password@localhost:3306/dbname?timeout=30s\u0026readTimeout=1s\u0026writeTimeout=1s" connect and set various timeouts
    • "user:password@/dbname?loc=UTC\u0026allowNativePasswords=true\u0026tls=preferred" connect and set location and native password allowance, and prefer TLS
  • rows_per_record (integer) (optional) (default: 500)
    Amount of rows to be packed into a single Apache Arrow record to be sent over the wire during sync.

Table Options Spec #

Enables setting table options for specific tables. Map key is the name of the table in untemplated form.
  • incremental_column (string) (optional)
Name of the incremental column in the table. If empty, no incremental column will be used. See incremental-examples for more information


Incremental-Examples #

Configuration Examples for Incremental Sync

To sync tables incrementally each table needs to have an incremental_column specified in the table_options section of the configuration. The tables without an incremental_column will be synced fully.

Sync a Table Incrementally #

To sync a table incrementally, you need to specify the incremental_column in the table_options section of the configuration. Here's an example source spec to sync a list of tables incrementally:
kind: source
spec:
  name: mysql
  path: cloudquery/mysql
  registry: cloudquery
  version: "v4.5.3"
  tables:
    - "my_table"
    - "another_table"
    - "yet_another_table"
  destinations: ["postgresql"]
  backend_options:
    table_name: "cq_state_bq"
    connection: "@@plugins.postgresql.connection"
  spec:
    connection_string: "user:password@/dbname"
    # Optional parameters
    # rows_per_record: 500
    table_options:
      my_table:
        incremental_column: updated_at
      another_table:
        incremental_column: id
In the example above, the my_table table will be synced incrementally based on the updated_at column. another_table will be synced incrementally based on the id column. yet_another_table will be synced fully.
The incremental column will be used to fetch only the new or updated rows from the source table and the state will be stored in the cq_state_bq table in the PostgreSQL destination.


Licenses #

The following tools / packages are used in this plugin:
NameLicense
github.com/adrg/xdgMIT
github.com/apache/arrow/go/v13Apache-2.0
github.com/apache/arrow/go/v17Apache-2.0
github.com/apapsch/go-jsonmerge/v2MIT
github.com/aws/aws-sdk-go-v2Apache-2.0
github.com/aws/aws-sdk-go-v2/configApache-2.0
github.com/aws/aws-sdk-go-v2/credentialsApache-2.0
github.com/aws/aws-sdk-go-v2/feature/ec2/imdsApache-2.0
github.com/aws/aws-sdk-go-v2/internal/configsourcesApache-2.0
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/iniApache-2.0
github.com/aws/aws-sdk-go-v2/internal/sync/singleflightBSD-3-Clause
github.com/aws/aws-sdk-go-v2/service/internal/accept-encodingApache-2.0
github.com/aws/aws-sdk-go-v2/service/internal/presigned-urlApache-2.0
github.com/aws/aws-sdk-go-v2/service/licensemanagerApache-2.0
github.com/aws/aws-sdk-go-v2/service/marketplacemeteringApache-2.0
github.com/aws/aws-sdk-go-v2/service/ssoApache-2.0
github.com/aws/aws-sdk-go-v2/service/ssooidcApache-2.0
github.com/aws/aws-sdk-go-v2/service/stsApache-2.0
github.com/aws/smithy-goApache-2.0
github.com/aws/smithy-go/internal/sync/singleflightBSD-3-Clause
github.com/cenkalti/backoff/v4MIT
github.com/cloudquery/cloudquery-api-goMPL-2.0
github.com/cloudquery/plugin-pb-goMPL-2.0
github.com/cloudquery/plugin-sdk/v2/internal/globMIT
github.com/cloudquery/plugin-sdk/v2/schemaMIT
github.com/cloudquery/plugin-sdk/v2/typesMPL-2.0
github.com/cloudquery/plugin-sdk/v4MPL-2.0
github.com/cloudquery/plugin-sdk/v4/globMIT
github.com/cloudquery/plugin-sdk/v4/scalarMIT
github.com/davecgh/go-spew/spewISC
github.com/ghodss/yamlMIT
github.com/go-logr/logrApache-2.0
github.com/go-logr/stdrApache-2.0
github.com/go-sql-driver/mysqlMPL-2.0
github.com/goccy/go-jsonMIT
github.com/google/flatbuffers/goApache-2.0
github.com/google/uuidBSD-3-Clause
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptorsApache-2.0
github.com/grpc-ecosystem/grpc-gateway/v2BSD-3-Clause
github.com/hashicorp/go-cleanhttpMPL-2.0
github.com/hashicorp/go-retryablehttpMPL-2.0
github.com/klauspost/compressApache-2.0
github.com/klauspost/compress/internal/snaprefBSD-3-Clause
github.com/klauspost/compress/zstd/internal/xxhashMIT
github.com/mattn/go-colorableMIT
github.com/mattn/go-isattyMIT
github.com/oapi-codegen/runtimeApache-2.0
github.com/pierrec/lz4/v4BSD-3-Clause
github.com/pmezard/go-difflib/difflibBSD-3-Clause
github.com/rs/zerologMIT
github.com/santhosh-tekuri/jsonschema/v6Apache-2.0
github.com/spf13/cobraApache-2.0
github.com/spf13/pflagBSD-3-Clause
github.com/stretchr/testifyMIT
github.com/thoas/go-funkMIT
github.com/zeebo/xxh3BSD-2-Clause
go.opentelemetry.io/otelApache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttpApache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttpApache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptraceApache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttpApache-2.0
go.opentelemetry.io/otel/logApache-2.0
go.opentelemetry.io/otel/metricApache-2.0
go.opentelemetry.io/otel/sdkApache-2.0
go.opentelemetry.io/otel/sdk/logApache-2.0
go.opentelemetry.io/otel/sdk/metricApache-2.0
go.opentelemetry.io/otel/traceApache-2.0
go.opentelemetry.io/proto/otlpApache-2.0
golang.org/x/expBSD-3-Clause
golang.org/x/netBSD-3-Clause
golang.org/x/sync/errgroupBSD-3-Clause
golang.org/x/sysBSD-3-Clause
golang.org/x/textBSD-3-Clause
golang.org/x/xerrorsBSD-3-Clause
google.golang.org/genproto/googleapis/api/httpbodyApache-2.0
google.golang.org/genproto/googleapis/rpc/statusApache-2.0
google.golang.org/grpcApache-2.0
google.golang.org/protobufBSD-3-Clause
gopkg.in/yaml.v2Apache-2.0
gopkg.in/yaml.v3MIT


Join our mailing list

Subscribe to our newsletter to make sure you don't miss any updates.

Legal

© 2024 CloudQuery, Inc. All rights reserved.

We use tracking cookies to understand how you use the product and help us improve it. Please accept cookies to help us improve. You can always opt out later via the link in the footer.