Official

SQLite destination integration documentation

This destination plugin lets you sync data from a CloudQuery source to a SQLite database

Publisher

cloudquery

Repository

github.com

Latest version

v2.13.1

Type

Destination

Platforms

Date Published

Documentation Changelog

Overview Authentication Configuration Spec Types Licenses

Overview #

This destination plugin lets you sync data from a CloudQuery source to a SQLite database. This can be useful for local data exploration as no other database or service is required.

Authentication #

CloudQuery needs to know what file to write to. Insert the file name of your SQLite database in the connection_string field in the sync config file: connection_string: ./db.sql

Configuration #

This is a basic configuration that will save all your sync resources to db.sql.

kind: destination
spec:
  name: sqlite
  path: cloudquery/sqlite
  registry: cloudquery
  version: "v2.13.1"
  # Learn more about the configuration options at https://cql.ink/sqlite_destination
  spec:
    connection_string: ./db.sql

After running cloudquery sync, you can explore the data locally with the SQLite CLI: sqlite ./db.sql.

Spec #

This is the top level spec used by the SQLite destination Plugin.

connection_string (string) (required)
Path to a file, such as ./mydb.sql.

Types #

The SQLite destination supports most Apache Arrow types. The following table shows the supported types and how they are mapped to SQLite data types.

Unsupported types are always mapped to text.

Arrow Column Type	Supported?	SQLite Type
Binary	✅ Yes	`blob`
Boolean	✅ Yes	`boolean`
Date32	✅ Yes	`text`
Date64	✅ Yes	`text`
Decimal	✅ Yes	`text`
Dense Union	✅ Yes	`text`
Dictionary	✅ Yes	`text`
Duration	✅ Yes	`text`
Fixed Size List	✅ Yes	`text`
Float16	✅ Yes	`real`
Float32	✅ Yes	`real`
Float64	✅ Yes	`real`
Inet	✅ Yes	`text`
Int8	✅ Yes	`integer`
Int16	✅ Yes	`integer`
Int32	✅ Yes	`integer`
Int64	✅ Yes	`integer`
Interval[DayTime]	✅ Yes	`text`
Interval[MonthDayNano]	✅ Yes	`text`
Interval[Month]	✅ Yes	`text`
JSON	✅ Yes	`text`
Large Binary	✅ Yes	`blob`
Large List	✅ Yes	`text`
Large String	✅ Yes	`text`
List	✅ Yes	`text`
MAC	✅ Yes	`text`
Map	✅ Yes	`text`
String	✅ Yes	`text`
Struct	✅ Yes	`text`
Time32	✅ Yes	`text`
Time64	✅ Yes	`text`
Timestamp	✅ Yes	`timestamp`
UUID	✅ Yes	`text`
Uint8	✅ Yes	`integer`
Uint16	✅ Yes	`integer`
Uint32	✅ Yes	`integer`
Uint64	✅ Yes	`integer`
Union	✅ Yes	`text`

Notes #

SQLite has a simplified type system with only 5 storage classes: NULL, INTEGER, REAL, TEXT, and BLOB
All integer types (signed and unsigned, 8-bit to 64-bit) are stored as SQLite integer
All floating-point types (Float16, Float32, Float64) are stored as SQLite real
Complex data types like JSON, List, and Struct are serialized and stored as text
Binary data uses SQLite's blob storage class
SQLite's dynamic typing system allows flexible data storage regardless of declared column type

Licenses #

The following tools / packages are used in this plugin:

Name	License
github.com/adrg/xdg	MIT
github.com/apache/arrow/go/v13	Apache-2.0
github.com/apache/arrow-go/v18	Apache-2.0
github.com/apapsch/go-jsonmerge/v2	MIT
github.com/aws/aws-sdk-go-v2	Apache-2.0
github.com/aws/aws-sdk-go-v2/config	Apache-2.0
github.com/aws/aws-sdk-go-v2/credentials	Apache-2.0
github.com/aws/aws-sdk-go-v2/feature/ec2/imds	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/configsources	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/ini	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/sync/singleflight	BSD-3-Clause
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/licensemanager	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/marketplacemetering	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/sso	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/ssooidc	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/sts	Apache-2.0
github.com/aws/smithy-go	Apache-2.0
github.com/aws/smithy-go/internal/sync/singleflight	BSD-3-Clause
github.com/cenkalti/backoff/v4	MIT
github.com/cloudquery/cloudquery-api-go	MPL-2.0
github.com/cloudquery/plugin-pb-go	MPL-2.0
github.com/cloudquery/plugin-sdk/v2/internal/glob	MIT
github.com/cloudquery/plugin-sdk/v2/schema	MIT
github.com/cloudquery/plugin-sdk/v2/types	MPL-2.0
github.com/cloudquery/plugin-sdk/v4	MPL-2.0
github.com/cloudquery/plugin-sdk/v4/glob	MIT
github.com/cloudquery/plugin-sdk/v4/scalar	MIT
github.com/davecgh/go-spew/spew	ISC
github.com/ghodss/yaml	MIT
github.com/go-logr/logr	Apache-2.0
github.com/go-logr/stdr	Apache-2.0
github.com/goccy/go-json	MIT
github.com/google/flatbuffers/go	Apache-2.0
github.com/google/uuid	BSD-3-Clause
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors	Apache-2.0
github.com/grpc-ecosystem/grpc-gateway/v2	BSD-3-Clause
github.com/hashicorp/go-cleanhttp	MPL-2.0
github.com/hashicorp/go-retryablehttp	MPL-2.0
github.com/huandu/xstrings	MIT
github.com/klauspost/compress	Apache-2.0
github.com/klauspost/compress/internal/snapref	BSD-3-Clause
github.com/klauspost/compress/zstd/internal/xxhash	MIT
github.com/mattn/go-colorable	MIT
github.com/mattn/go-isatty	MIT
github.com/mattn/go-sqlite3	MIT
github.com/oapi-codegen/runtime	Apache-2.0
github.com/pierrec/lz4/v4	BSD-3-Clause
github.com/pmezard/go-difflib/difflib	BSD-3-Clause
github.com/rs/zerolog	MIT
github.com/santhosh-tekuri/jsonschema/v6	Apache-2.0
github.com/spf13/cobra	Apache-2.0
github.com/spf13/pflag	BSD-3-Clause
github.com/stretchr/testify	MIT
github.com/thoas/go-funk	MIT
github.com/zeebo/xxh3	BSD-2-Clause
go.opentelemetry.io/otel	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp	Apache-2.0
go.opentelemetry.io/otel/log	Apache-2.0
go.opentelemetry.io/otel/metric	Apache-2.0
go.opentelemetry.io/otel/sdk	Apache-2.0
go.opentelemetry.io/otel/sdk/log	Apache-2.0
go.opentelemetry.io/otel/sdk/metric	Apache-2.0
go.opentelemetry.io/otel/trace	Apache-2.0
go.opentelemetry.io/proto/otlp	Apache-2.0
golang.org/x/exp	BSD-3-Clause
golang.org/x/net	BSD-3-Clause
golang.org/x/sync/errgroup	BSD-3-Clause
golang.org/x/sys	BSD-3-Clause
golang.org/x/text	BSD-3-Clause
golang.org/x/xerrors	BSD-3-Clause
google.golang.org/genproto/googleapis/api/httpbody	Apache-2.0
google.golang.org/genproto/googleapis/rpc/status	Apache-2.0
google.golang.org/grpc	Apache-2.0
google.golang.org/protobuf	BSD-3-Clause
gopkg.in/yaml.v2	Apache-2.0
gopkg.in/yaml.v3	MIT

Loading plugin documentation