Back to transformer list
J
Official
JSON Flattener
This plugin is in preview.
The JSON Flattener CloudQuery transformer plugin provides basic JSON flattening capabilities
Price
Free
Overview #
This CloudQuery transformer plugin provides basic JSON flattening capabilities:
- JSON fields on source table will be flattened into separate columns in the destination, while still preserving the original JSON fields.
- Only JSON fields with schema metadata will be flattened. This metadata is present for selected CloudQuery sources.
- Only single-level JSON object fields will be flattened. JSON arrays will not be flattened.
- Multi-level JSON objects will not be flattened recursively.
Example transformation:
If before applying the transformer your sync is producing this table in the destination:
| account_id | region | details |
|---|---|---|
| 012345678901 | us-west-1 | {"field_1": "2021-06-10 07:30:00", "field_2": "value_2", "field_3": true, "field_4": 4} |
| 012345678901 | ca-central-1 | {"field_1": "2021-07-25 12:40:00", "field_2": "value_4", "field_3": false, "field_4": 5} |
| 012345678901 | sa-east-1 | {"field_1": "2021-08-15 15:10:00", "field_2": "value_6", "field_3": null, "field_4": 6} |
After applying the transformer, you will get this table (
details column content is omitted for brevity):| account_id | region | details | field_1 | field_2 | field_3 | field_4 |
|---|---|---|---|---|---|---|
| 012345678901 | us-west-1 | …(unchanged) | 2021-06-10 07:30:00 | value_2 | true | 4 |
| 012345678901 | ca-central-1 | …(unchanged) | 2021-07-25 12:40:00 | value_4 | false | 5 |
| 012345678901 | sa-east-1 | …(unchanged) | 2021-08-15 15:10:00 | value_6 | null | 6 |
Extra fields are typed based on the
TypeSchema metadata:field_1is typed astimestampfield_2is typed asstringfield_3is typed asbooleanfield_4is typed asint64
Configuration #
First, add the transformer to your destination. For example, this will add a jsonflattener transformer to a PostgreSQL destination:
kind: destination
spec:
name: "postgresql"
path: "cloudquery/postgresql"
registry: "cloudquery"
version: "v8.7.5"
write_mode: "overwrite-delete-stale"
migrate_mode: forced # optional
transformers:
- "jsonflattener"
spec:
connection_string: "postgresql://your.user:your.password@localhost:5432/db_name"The
migrate_mode: forced setting might make sense if you plan on modifying the schema from a previous sync.Then, add your transformer spec:
kind: transformer
spec:
name: "jsonflattener"
path: "cloudquery/jsonflattener"
version: v2.0.0
spec:
tables: ["aws_ec2_instances"]Licenses #
The following tools / packages are used in this plugin:
| Name | License |
|---|---|
| github.com/adrg/xdg | MIT |
| github.com/apache/arrow/go/v13 | Apache-2.0 |
| github.com/apache/arrow-go/v18 | Apache-2.0 |
| github.com/apapsch/go-jsonmerge/v2 | MIT |
| github.com/aws/aws-sdk-go-v2 | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/config | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/credentials | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/feature/ec2/imds | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/internal/configsources | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/internal/ini | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/internal/sync/singleflight | BSD-3-Clause |
| github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/service/internal/presigned-url | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/service/licensemanager | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/service/marketplacemetering | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/service/sso | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/service/ssooidc | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/service/sts | Apache-2.0 |
| github.com/aws/smithy-go | Apache-2.0 |
| github.com/aws/smithy-go/internal/sync/singleflight | BSD-3-Clause |
| github.com/cenkalti/backoff/v4 | MIT |
| github.com/cloudquery/cloudquery-api-go | MPL-2.0 |
| github.com/cloudquery/plugin-pb-go | MPL-2.0 |
| github.com/cloudquery/plugin-sdk/v2/internal/glob | MIT |
| github.com/cloudquery/plugin-sdk/v2/schema | MIT |
| github.com/cloudquery/plugin-sdk/v2/types | MPL-2.0 |
| github.com/cloudquery/plugin-sdk/v4 | MPL-2.0 |
| github.com/cloudquery/plugin-sdk/v4/glob | MIT |
| github.com/cloudquery/plugin-sdk/v4/scalar | MIT |
| github.com/davecgh/go-spew/spew | ISC |
| github.com/ghodss/yaml | MIT |
| github.com/go-logr/logr | Apache-2.0 |
| github.com/go-logr/stdr | Apache-2.0 |
| github.com/goccy/go-json | MIT |
| github.com/google/flatbuffers/go | Apache-2.0 |
| github.com/google/uuid | BSD-3-Clause |
| github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors | Apache-2.0 |
| github.com/grpc-ecosystem/grpc-gateway/v2 | BSD-3-Clause |
| github.com/hashicorp/go-cleanhttp | MPL-2.0 |
| github.com/hashicorp/go-retryablehttp | MPL-2.0 |
| github.com/klauspost/compress | Apache-2.0 |
| github.com/klauspost/compress/internal/snapref | BSD-3-Clause |
| github.com/klauspost/compress/zstd/internal/xxhash | MIT |
| github.com/mattn/go-colorable | MIT |
| github.com/mattn/go-isatty | MIT |
| github.com/oapi-codegen/runtime | Apache-2.0 |
| github.com/pierrec/lz4/v4 | BSD-3-Clause |
| github.com/pmezard/go-difflib/difflib | BSD-3-Clause |
| github.com/rs/zerolog | MIT |
| github.com/santhosh-tekuri/jsonschema/v6 | Apache-2.0 |
| github.com/spf13/cobra | Apache-2.0 |
| github.com/spf13/pflag | BSD-3-Clause |
| github.com/stretchr/testify | MIT |
| github.com/thoas/go-funk | MIT |
| github.com/zeebo/xxh3 | BSD-2-Clause |
| go.opentelemetry.io/otel | Apache-2.0 |
| go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp | Apache-2.0 |
| go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp | Apache-2.0 |
| go.opentelemetry.io/otel/exporters/otlp/otlptrace | Apache-2.0 |
| go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp | Apache-2.0 |
| go.opentelemetry.io/otel/log | Apache-2.0 |
| go.opentelemetry.io/otel/metric | Apache-2.0 |
| go.opentelemetry.io/otel/sdk | Apache-2.0 |
| go.opentelemetry.io/otel/sdk/log | Apache-2.0 |
| go.opentelemetry.io/otel/sdk/metric | Apache-2.0 |
| go.opentelemetry.io/otel/trace | Apache-2.0 |
| go.opentelemetry.io/proto/otlp | Apache-2.0 |
| golang.org/x/exp | BSD-3-Clause |
| golang.org/x/net | BSD-3-Clause |
| golang.org/x/sync/errgroup | BSD-3-Clause |
| golang.org/x/sys | BSD-3-Clause |
| golang.org/x/text | BSD-3-Clause |
| golang.org/x/xerrors | BSD-3-Clause |
| google.golang.org/genproto/googleapis/api/httpbody | Apache-2.0 |
| google.golang.org/genproto/googleapis/rpc/status | Apache-2.0 |
| google.golang.org/grpc | Apache-2.0 |
| google.golang.org/protobuf | BSD-3-Clause |
| gopkg.in/yaml.v2 | Apache-2.0 |
| gopkg.in/yaml.v3 | MIT |