Back to source list
Official
Premium
MySQL
The CloudQuery MySQL plugin syncs your MySQL database to any of the supported CloudQuery destinations
Publisher
cloudquery
Latest version
v4.5.3
Type
Source
Platforms
Date Published
Price per 1M rows
Starting from $17
monthly free quota
1M rows
Set up process #
brew install cloudquery/tap/cloudquery
1. Download CLI and login
2. Create source and destination configs
Plugin configurationOverview #
The CloudQuery MySQL plugin syncs your MySQL database to any of the supported CloudQuery destinations (e.g. PostgreSQL, BigQuery, Snowflake, and more).
Supported database versions are >= 8.0.
If you need support for older versions, please contact us
Example Configuration #
kind: source
spec:
name: mysql
path: cloudquery/mysql
registry: cloudquery
version: "v4.5.3"
tables: ["*"]
destinations: ["postgresql"]
# Learn more about the configuration options at https://cql.ink/mysql_source
spec:
connection_string: "user:password@/dbname"
# Optional parameters
# rows_per_record: 500
Configuration Reference #
This is the (nested) spec used by the MySQL destination plugin.
connection_string
(string
) (required)Connection string to connect to the database. See the Go driver documentation for more details."user:password@tcp(127.0.0.1:3306)/dbname"
connect with TCP"user:[email protected]:3306/dbname?charset=utf8mb4\u0026parseTime=True\u0026loc=Local"
connect and set character set, time parsing, and location"user:password@localhost:3306/dbname?timeout=30s\u0026readTimeout=1s\u0026writeTimeout=1s"
connect and set various timeouts"user:password@/dbname?loc=UTC\u0026allowNativePasswords=true\u0026tls=preferred"
connect and set location and native password allowance, and prefer TLS
rows_per_record
(integer
) (optional) (default:500
)Amount of rows to be packed into a single Apache Arrow record to be sent over the wire during sync.
Table Options Spec #
Enables setting table options for specific tables. Map key is the name of the table in untemplated form.
incremental_column
(string
) (optional)
Name of the incremental column in the table. If empty, no incremental column will be used. See incremental-examples for more information
Incremental-Examples #
Configuration Examples for Incremental Sync
To sync tables incrementally each table needs to have an
incremental_column
specified in the table_options
section of the configuration. The tables without an incremental_column
will be synced fully.Sync a Table Incrementally #
To sync a table incrementally, you need to specify the
incremental_column
in the table_options
section of the configuration. Here's an example source spec to sync a list of tables incrementally:kind: source
spec:
name: mysql
path: cloudquery/mysql
registry: cloudquery
version: "v4.5.3"
tables:
- "my_table"
- "another_table"
- "yet_another_table"
destinations: ["postgresql"]
backend_options:
table_name: "cq_state_bq"
connection: "@@plugins.postgresql.connection"
spec:
connection_string: "user:password@/dbname"
# Optional parameters
# rows_per_record: 500
table_options:
my_table:
incremental_column: updated_at
another_table:
incremental_column: id
In the example above, the
my_table
table will be synced incrementally based on the updated_at
column. another_table
will be synced incrementally based on the id
column. yet_another_table
will be synced fully.The incremental column will be used to fetch only the new or updated rows from the source table and the state will be stored in the
cq_state_bq
table in the PostgreSQL destination.Licenses #
The following tools / packages are used in this plugin:
Name | License |
---|---|
github.com/adrg/xdg | MIT |
github.com/apache/arrow/go/v13 | Apache-2.0 |
github.com/apache/arrow/go/v17 | Apache-2.0 |
github.com/apapsch/go-jsonmerge/v2 | MIT |
github.com/aws/aws-sdk-go-v2 | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/config | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/credentials | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/feature/ec2/imds | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/internal/configsources | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/internal/ini | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/internal/sync/singleflight | BSD-3-Clause |
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/licensemanager | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/marketplacemetering | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/sso | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/ssooidc | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/sts | Apache-2.0 |
github.com/aws/smithy-go | Apache-2.0 |
github.com/aws/smithy-go/internal/sync/singleflight | BSD-3-Clause |
github.com/cenkalti/backoff/v4 | MIT |
github.com/cloudquery/cloudquery-api-go | MPL-2.0 |
github.com/cloudquery/plugin-pb-go | MPL-2.0 |
github.com/cloudquery/plugin-sdk/v2/internal/glob | MIT |
github.com/cloudquery/plugin-sdk/v2/schema | MIT |
github.com/cloudquery/plugin-sdk/v2/types | MPL-2.0 |
github.com/cloudquery/plugin-sdk/v4 | MPL-2.0 |
github.com/cloudquery/plugin-sdk/v4/glob | MIT |
github.com/cloudquery/plugin-sdk/v4/scalar | MIT |
github.com/davecgh/go-spew/spew | ISC |
github.com/ghodss/yaml | MIT |
github.com/go-logr/logr | Apache-2.0 |
github.com/go-logr/stdr | Apache-2.0 |
github.com/go-sql-driver/mysql | MPL-2.0 |
github.com/goccy/go-json | MIT |
github.com/google/flatbuffers/go | Apache-2.0 |
github.com/google/uuid | BSD-3-Clause |
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors | Apache-2.0 |
github.com/grpc-ecosystem/grpc-gateway/v2 | BSD-3-Clause |
github.com/hashicorp/go-cleanhttp | MPL-2.0 |
github.com/hashicorp/go-retryablehttp | MPL-2.0 |
github.com/klauspost/compress | Apache-2.0 |
github.com/klauspost/compress/internal/snapref | BSD-3-Clause |
github.com/klauspost/compress/zstd/internal/xxhash | MIT |
github.com/mattn/go-colorable | MIT |
github.com/mattn/go-isatty | MIT |
github.com/oapi-codegen/runtime | Apache-2.0 |
github.com/pierrec/lz4/v4 | BSD-3-Clause |
github.com/pmezard/go-difflib/difflib | BSD-3-Clause |
github.com/rs/zerolog | MIT |
github.com/santhosh-tekuri/jsonschema/v6 | Apache-2.0 |
github.com/spf13/cobra | Apache-2.0 |
github.com/spf13/pflag | BSD-3-Clause |
github.com/stretchr/testify | MIT |
github.com/thoas/go-funk | MIT |
github.com/zeebo/xxh3 | BSD-2-Clause |
go.opentelemetry.io/otel | Apache-2.0 |
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp | Apache-2.0 |
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp | Apache-2.0 |
go.opentelemetry.io/otel/exporters/otlp/otlptrace | Apache-2.0 |
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp | Apache-2.0 |
go.opentelemetry.io/otel/log | Apache-2.0 |
go.opentelemetry.io/otel/metric | Apache-2.0 |
go.opentelemetry.io/otel/sdk | Apache-2.0 |
go.opentelemetry.io/otel/sdk/log | Apache-2.0 |
go.opentelemetry.io/otel/sdk/metric | Apache-2.0 |
go.opentelemetry.io/otel/trace | Apache-2.0 |
go.opentelemetry.io/proto/otlp | Apache-2.0 |
golang.org/x/exp | BSD-3-Clause |
golang.org/x/net | BSD-3-Clause |
golang.org/x/sync/errgroup | BSD-3-Clause |
golang.org/x/sys | BSD-3-Clause |
golang.org/x/text | BSD-3-Clause |
golang.org/x/xerrors | BSD-3-Clause |
google.golang.org/genproto/googleapis/api/httpbody | Apache-2.0 |
google.golang.org/genproto/googleapis/rpc/status | Apache-2.0 |
google.golang.org/grpc | Apache-2.0 |
google.golang.org/protobuf | BSD-3-Clause |
gopkg.in/yaml.v2 | Apache-2.0 |
gopkg.in/yaml.v3 | MIT |