JSON Flattener transformer documentation

The JSON Flattener CloudQuery transformer plugin provides basic JSON flattening capabilities

Publisher

cloudquery

Repository

github.com

Latest version

v2.1.12

Type

Transformer

Platforms

Date Published

Start Syncing

Overview Licenses

Overview #

This CloudQuery transformer plugin provides basic JSON flattening capabilities:

JSON fields on source table will be flattened into separate columns in the destination, while still preserving the original JSON fields.
Only JSON fields with schema metadata will be flattened. This metadata is present for selected CloudQuery sources.
Only single-level JSON object fields will be flattened. JSON arrays will not be flattened.
Multi-level JSON objects will not be flattened recursively.

Example transformation:

If before applying the transformer your sync is producing this table in the destination:

account_id	region	details
012345678901	us-west-1	{"field_1": "2021-06-10 07:30:00", "field_2": "value_2", "field_3": true, "field_4": 4}
012345678901	ca-central-1	{"field_1": "2021-07-25 12:40:00", "field_2": "value_4", "field_3": false, "field_4": 5}
012345678901	sa-east-1	{"field_1": "2021-08-15 15:10:00", "field_2": "value_6", "field_3": null, "field_4": 6}

After applying the transformer, you will get this table (details column content is omitted for brevity):

account_id	region	details	field_1	field_2	field_3	field_4
012345678901	us-west-1	…(unchanged)	2021-06-10 07:30:00	value_2	true	4
012345678901	ca-central-1	…(unchanged)	2021-07-25 12:40:00	value_4	false	5
012345678901	sa-east-1	…(unchanged)	2021-08-15 15:10:00	value_6	null	6

Extra fields are typed based on the TypeSchema metadata:

field_1 is typed as timestamp
field_2 is typed as string
field_3 is typed as boolean
field_4 is typed as int64

Configuration #

First, add the transformer to your destination. For example, this will add a jsonflattener transformer to a PostgreSQL destination:

kind: destination
spec:
  name: "postgresql"
  path: "cloudquery/postgresql"
  registry: "cloudquery"
  version: "v8.14.1"
  write_mode: "overwrite-delete-stale"
  migrate_mode: forced # optional
  transformers:
    - "jsonflattener"

  spec:
    connection_string: "postgresql://your.user:your.password@localhost:5432/db_name"

The migrate_mode: forced setting might make sense if you plan on modifying the schema from a previous sync.

Then, add your transformer spec:

kind: transformer
spec:
  name: "jsonflattener"
  path: "cloudquery/jsonflattener"
  version: v2.1.12
  spec:
    tables: ["aws_ec2_instances"]

Licenses #

The following tools / packages are used in this plugin:

Name	License
github.com/adrg/xdg	MIT
github.com/apache/arrow/go/v13	Apache-2.0
github.com/apache/arrow-go/v18	Apache-2.0
github.com/apapsch/go-jsonmerge/v2	MIT
github.com/aws/aws-sdk-go-v2	Apache-2.0
github.com/aws/aws-sdk-go-v2/config	Apache-2.0
github.com/aws/aws-sdk-go-v2/credentials	Apache-2.0
github.com/aws/aws-sdk-go-v2/feature/ec2/imds	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/configsources	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/ini	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/sync/singleflight	BSD-3-Clause
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/licensemanager	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/marketplacemetering	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/sso	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/ssooidc	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/sts	Apache-2.0
github.com/aws/smithy-go	Apache-2.0
github.com/aws/smithy-go/internal/sync/singleflight	BSD-3-Clause
github.com/cenkalti/backoff/v4	MIT
github.com/cloudquery/cloudquery-api-go	MPL-2.0
github.com/cloudquery/plugin-pb-go	MPL-2.0
github.com/cloudquery/plugin-sdk/v2/internal/glob	MIT
github.com/cloudquery/plugin-sdk/v2/schema	MIT
github.com/cloudquery/plugin-sdk/v2/types	MPL-2.0
github.com/cloudquery/plugin-sdk/v4	MPL-2.0
github.com/cloudquery/plugin-sdk/v4/glob	MIT
github.com/cloudquery/plugin-sdk/v4/scalar	MIT
github.com/davecgh/go-spew/spew	ISC
github.com/ghodss/yaml	MIT
github.com/go-logr/logr	Apache-2.0
github.com/go-logr/stdr	Apache-2.0
github.com/goccy/go-json	MIT
github.com/google/flatbuffers/go	Apache-2.0
github.com/google/uuid	BSD-3-Clause
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors	Apache-2.0
github.com/grpc-ecosystem/grpc-gateway/v2	BSD-3-Clause
github.com/hashicorp/go-cleanhttp	MPL-2.0
github.com/hashicorp/go-retryablehttp	MPL-2.0
github.com/klauspost/compress	Apache-2.0
github.com/klauspost/compress/internal/snapref	BSD-3-Clause
github.com/klauspost/compress/zstd/internal/xxhash	MIT
github.com/mattn/go-colorable	MIT
github.com/mattn/go-isatty	MIT
github.com/oapi-codegen/runtime	Apache-2.0
github.com/pierrec/lz4/v4	BSD-3-Clause
github.com/pmezard/go-difflib/difflib	BSD-3-Clause
github.com/rs/zerolog	MIT
github.com/santhosh-tekuri/jsonschema/v6	Apache-2.0
github.com/spf13/cobra	Apache-2.0
github.com/spf13/pflag	BSD-3-Clause
github.com/stretchr/testify	MIT
github.com/thoas/go-funk	MIT
github.com/zeebo/xxh3	BSD-2-Clause
go.opentelemetry.io/otel	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp	Apache-2.0
go.opentelemetry.io/otel/log	Apache-2.0
go.opentelemetry.io/otel/metric	Apache-2.0
go.opentelemetry.io/otel/sdk	Apache-2.0
go.opentelemetry.io/otel/sdk/log	Apache-2.0
go.opentelemetry.io/otel/sdk/metric	Apache-2.0
go.opentelemetry.io/otel/trace	Apache-2.0
go.opentelemetry.io/proto/otlp	Apache-2.0
golang.org/x/exp	BSD-3-Clause
golang.org/x/net	BSD-3-Clause
golang.org/x/sync/errgroup	BSD-3-Clause
golang.org/x/sys	BSD-3-Clause
golang.org/x/text	BSD-3-Clause
golang.org/x/xerrors	BSD-3-Clause
google.golang.org/genproto/googleapis/api/httpbody	Apache-2.0
google.golang.org/genproto/googleapis/rpc/status	Apache-2.0
google.golang.org/grpc	Apache-2.0
google.golang.org/protobuf	BSD-3-Clause
gopkg.in/yaml.v2	Apache-2.0
gopkg.in/yaml.v3	MIT

Loading plugin documentation

CloudOps

J

JSON Flattener transformer documentation

Overview #

Example transformation:

Configuration #

Overview #

Example transformation:

Configuration #