Official

DuckDB destination integration documentation

This destination plugin lets you sync data from a CloudQuery source to a DuckDB database

Publisher

cloudquery

Repository

github.com

Latest version

v6.2.1

Type

Destination

Platforms

Date Published

Documentation Changelog

Overview Types Licenses

Overview #

DuckDB Destination Plugin

This destination plugin lets you sync data from a CloudQuery source to a DuckDB database.

Example Config #

This is a basic configuration that will save all your sync resources to ./database.db in the local directory.

kind: destination
spec:
  name: duckdb
  path: cloudquery/duckdb
  registry: cloudquery
  version: "v6.2.1"
  write_mode: "overwrite-delete-stale"
  # Learn more about the configuration options at https://cql.ink/duckdb_destination
  spec:
    connection_string: ./database.db
    # Optional parameters
    # batch_size: 1000
    # batch_size_bytes: 4194304 # 4 MiB
    # debug: false

After running cloudquery sync, you can explore the data locally with the DuckDB CLI: duckdb ./database.db.

The default write_mode is overwrite-delete-stale, but the plugin also supports overwrite or append. Note that overwrite and overwrite-delete-stale modes do not support atomic updates: to update a resource, it is first deleted and then re-inserted. This is due to a current lack of support in DuckDB for upserting list-type columns. If this is an issue for you, consider using the append mode instead. You may then perform a manual cleanup of stale resources after the sync completes.

Note that this plugin does currently not support Windows. See this issue for more details.

DuckDB Spec #

This is the top level spec used by the DuckDB destination Plugin.

connection_string (string) (required)
Absolute or relative path to a file, such as ./example.duckdb.
batch_size (integer) (optional) (default: 1000)
Maximum number of items that may be grouped together to be written in a single write.
batch_size_bytes (integer) (optional) (default: 4194304 (4 MiB))
Maximum size of items that may be grouped together to be written in a single write.
debug (boolean) (optional) (default: false)
Enables debug logging.

Types #

The DuckDB destination supports most Apache Arrow types. The following table shows the supported types and how they are mapped to DuckDB data types.

Unsupported types are always mapped to varchar.

Arrow Column Type	Supported?	DuckDB Type
Binary	✅ Yes	`blob`
Boolean	✅ Yes	`boolean`
Date32	✅ Yes	`timestamp`
Date64	✅ Yes	`timestamp`
Decimal	✅ Yes	`varchar`
Dense Union	✅ Yes	`varchar`
Dictionary	✅ Yes	`varchar`
Duration	✅ Yes	`varchar`
Fixed Size List	✅ Yes	`varchar`
Float16	✅ Yes	`varchar`
Float32	✅ Yes	`float`
Float64	✅ Yes	`double`
Inet	✅ Yes	`varchar`
Int8	✅ Yes	`tinyint`
Int16	✅ Yes	`smallint`
Int32	✅ Yes	`integer`
Int64	✅ Yes	`bigint`
Interval[DayTime]	✅ Yes	`varchar`
Interval[MonthDayNano]	✅ Yes	`varchar`
Interval[Month]	✅ Yes	`varchar`
JSON	✅ Yes	`json`
Large Binary	✅ Yes	`blob`
Large List	✅ Yes	Array of element type
Large String	✅ Yes	`varchar`
List	✅ Yes	Array of element type
MAC	✅ Yes	`varchar`
Map	✅ Yes	`varchar` †
String	✅ Yes	`varchar`
Struct	✅ Yes	`varchar` †
Timestamp	✅ Yes	`timestamp`
UUID	✅ Yes	`uuid`
Uint8	✅ Yes	`uinteger`
Uint16	✅ Yes	`uinteger`
Uint32	✅ Yes	`uinteger`
Uint64	✅ Yes	`ubigint`
Union	✅ Yes	`varchar`

Notes #

DuckDB supports native unsigned integer types (uinteger, ubigint)
List types are converted to DuckDB arrays with the appropriate element type (e.g., integer[])
Date32 and Date64 types are both mapped to timestamp for compatibility
UUID and JSON types have native DuckDB support

† Complex types like Struct and Map are converted to varchar for storage, losing their structured nature

Licenses #

The following tools / packages are used in this plugin:

Name	License
github.com/adrg/xdg	MIT
github.com/andybalholm/brotli	MIT
github.com/apache/arrow-go/v18	Apache-2.0
github.com/apache/arrow/go/v13	Apache-2.0
github.com/apache/thrift/lib/go/thrift	Apache-2.0
github.com/apapsch/go-jsonmerge/v2	MIT
github.com/aws/aws-sdk-go-v2	Apache-2.0
github.com/aws/aws-sdk-go-v2/config	Apache-2.0
github.com/aws/aws-sdk-go-v2/credentials	Apache-2.0
github.com/aws/aws-sdk-go-v2/feature/ec2/imds	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/configsources	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/ini	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/sync/singleflight	BSD-3-Clause
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/licensemanager	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/marketplacemetering	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/sso	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/ssooidc	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/sts	Apache-2.0
github.com/aws/smithy-go	Apache-2.0
github.com/aws/smithy-go/internal/sync/singleflight	BSD-3-Clause
github.com/bahlo/generic-list-go	BSD-3-Clause
github.com/buger/jsonparser	MIT
github.com/cenkalti/backoff/v4	MIT
github.com/cenkalti/backoff/v5	MIT
github.com/cespare/xxhash/v2	MIT
github.com/cloudquery/cloudquery-api-go	MPL-2.0
github.com/cloudquery/codegen/jsonschema/docs	MPL-2.0
github.com/cloudquery/plugin-pb-go	MPL-2.0
github.com/cloudquery/plugin-sdk/v2/internal/glob	MIT
github.com/cloudquery/plugin-sdk/v2/schema	MIT
github.com/cloudquery/plugin-sdk/v2/types	MPL-2.0
github.com/cloudquery/plugin-sdk/v4	MPL-2.0
github.com/cloudquery/plugin-sdk/v4/glob	MIT
github.com/cloudquery/plugin-sdk/v4/scalar	MIT
github.com/davecgh/go-spew/spew	ISC
github.com/duckdb/duckdb-go-bindings/darwin-arm64	MIT
github.com/ghodss/yaml	MIT
github.com/go-logr/logr	Apache-2.0
github.com/go-logr/stdr	Apache-2.0
github.com/go-viper/mapstructure/v2	MIT
github.com/goccy/go-json	MIT
github.com/golang/snappy	BSD-3-Clause
github.com/google/flatbuffers/go	Apache-2.0
github.com/google/uuid	BSD-3-Clause
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors	Apache-2.0
github.com/grpc-ecosystem/grpc-gateway/v2	BSD-3-Clause
github.com/hashicorp/go-cleanhttp	MPL-2.0
github.com/hashicorp/go-retryablehttp	MPL-2.0
github.com/invopop/jsonschema	MIT
github.com/klauspost/compress	Apache-2.0
github.com/klauspost/compress/internal/snapref	BSD-3-Clause
github.com/klauspost/compress/zstd/internal/xxhash	MIT
github.com/klauspost/cpuid/v2	MIT
github.com/mailru/easyjson	MIT
github.com/marcboeker/go-duckdb/mapping	MIT
github.com/marcboeker/go-duckdb/v2	MIT
github.com/mattn/go-colorable	MIT
github.com/mattn/go-isatty	MIT
github.com/oapi-codegen/runtime	Apache-2.0
github.com/pierrec/lz4/v4	BSD-3-Clause
github.com/pmezard/go-difflib/difflib	BSD-3-Clause
github.com/rs/zerolog	MIT
github.com/samber/lo	MIT
github.com/santhosh-tekuri/jsonschema/v6	Apache-2.0
github.com/spf13/cobra	Apache-2.0
github.com/spf13/pflag	BSD-3-Clause
github.com/stoewer/go-strcase	MIT
github.com/stretchr/testify	MIT
github.com/thoas/go-funk	MIT
github.com/wk8/go-ordered-map/v2	Apache-2.0
github.com/zeebo/xxh3	BSD-2-Clause
go.opentelemetry.io/auto/sdk	Apache-2.0
go.opentelemetry.io/otel	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp	Apache-2.0
go.opentelemetry.io/otel/log	Apache-2.0
go.opentelemetry.io/otel/metric	Apache-2.0
go.opentelemetry.io/otel/sdk	Apache-2.0
go.opentelemetry.io/otel/sdk/log	Apache-2.0
go.opentelemetry.io/otel/sdk/metric	Apache-2.0
go.opentelemetry.io/otel/trace	Apache-2.0
go.opentelemetry.io/proto/otlp	Apache-2.0
golang.org/x/exp	BSD-3-Clause
golang.org/x/net	BSD-3-Clause
golang.org/x/sync/errgroup	BSD-3-Clause
golang.org/x/sys	BSD-3-Clause
golang.org/x/text	BSD-3-Clause
golang.org/x/xerrors	BSD-3-Clause
google.golang.org/genproto/googleapis/api/httpbody	Apache-2.0
google.golang.org/genproto/googleapis/rpc/status	Apache-2.0
google.golang.org/grpc	Apache-2.0
google.golang.org/protobuf	BSD-3-Clause
gopkg.in/yaml.v2	Apache-2.0
gopkg.in/yaml.v3	MIT

Loading plugin documentation