Back to destination list
Official
Gremlin
This destination plugin lets you sync data from any CloudQuery source to a Gremlin compatible graph database such as AWS Neptune
Price
Free
Overview #
Gremlin Destination Plugin
This destination plugin lets you sync data from any CloudQuery source to a Gremlin compatible graph database such as AWS Neptune.
Supported database (tested) versions (We use the official Go driver):
- Gremlin Server >= 3.6.2
- AWS Neptune >= 1.2
As a side note graph databases can be quite useful for various networking use-cases, visualization, for read-teams, blue-teams and more.
Configuration #
Example #
This example configures a Gremlin destination, located at
ws://localhost:8182
. The username and password are stored in environment variables.kind: destination
spec:
name: "gremlin"
path: "cloudquery/gremlin"
registry: "cloudquery"
version: "v2.6.3"
spec:
endpoint: "ws://localhost:8182"
# Optional parameters
# auth_mode: none
# username: ""
# password: ""
# aws_region: ""
# aws_neptune_host: ""
# max_retries: 5
# max_concurrent_connections: 5 # default: number of CPUs
# batch_size: 200
# batch_size_bytes: 4194304 # 4 MiB
The (top level) spec section is described in the Destination Spec Reference.
The Gremlin destination utilizes batching, and supports
batch_size
and batch_size_bytes
.Connecting to AWS Neptune #
For AWS Neptune, you don't need to specify any credentials if IAM authentication is not enabled. Keep
auth_mode
at none
.If IAM authentication is enabled, you need to set
auth_mode
to aws
and aws_region
to the region of the database. The plugin will use the default AWS credentials chain to authenticate.Plugin Spec #
This is the (nested) spec used by the Gremlin destination Plugin.
endpoint
(string
) (required)Endpoint for the database. Supported schemes arewss://
andws://
, the default port is8182
."localhost"
(defaults towss://localhost:8182
)"ws://localhost:8182"
"wss://your-endpoint.cluster-id.your-region.neptune.amazonaws.com"
insecure
(boolean
) (optional)Whether to skip TLS verification. Defaults tofalse
. This should be set on a macOS environment when connecting to an AWS Neptune endpoint.auth_mode
(string
) (optional) (default:none
)Authentication mode to use.basic
uses static credentials,aws
uses AWS IAM authentication. Supported values arenone
,basic
oraws
.username
(string
) (optional)Username to connect to the database.password
(string
) (optional)Password to connect to the database.aws_region
(string
) (required whenauth_mode
isaws
)AWS region to use for AWS IAM authentication. Example:us-east-1
.aws_neptune_host
(string
) (optional, used whenauth_mode
isaws
)AWS Neptune host header to use with AWS IAM authentication. Use if you're not accessing Neptune directly, whenauth_mode
isaws
. Example:my-neptune.cluster.us-east-1.neptune.amazonaws.com
max_retries
(integer
) (optional) (default:5
)Number of retries onConcurrentModificationException
before giving up for each batch. Retries are exponentially backed off.max_concurrent_connections
(integer
) (optional) (default: number of CPUs)Maximum number of concurrent connections to the database.complete_types
(boolean
) (optional) (default:false
)Whether to use all Gremlin-supported types or just a basic set. Should remainfalse
for Amazon Neptune compatibility.batch_size
(integer
) (optional) (default:200
)Number of records to batch together before sending to the database.batch_size_bytes
(integer
) (optional) (default:4194304
(4 MiB))Number of bytes (as Arrow buffer size) to batch together before sending to the database.
Types #
Gremlin Types
The Gremlin destination (
v2.0.0
and later) supports most Apache Arrow types. The following table shows the supported types and how they are mapped to Gremlin data types.Arrow Column Type | Supported? | Gremlin Type |
---|---|---|
Binary | ✅ Yes | Bytes |
Boolean | ✅ Yes | Boolean |
Date32 | ✅ Yes | String |
Date64 | ✅ Yes | String |
Decimal | ✅ Yes | String |
Dense Union | ✅ Yes | String |
Dictionary | ✅ Yes | String |
Duration[ms] | ✅ Yes | String |
Duration[ns] | ✅ Yes | String |
Duration[s] | ✅ Yes | String |
Duration[us] | ✅ Yes | String |
Fixed Size List | ✅ Yes | String |
Float16 | ✅ Yes | String |
Float32 | ✅ Yes | Float |
Float64 | ✅ Yes | Float |
Inet | ✅ Yes | String |
Int8 | ✅ Yes | Integer |
Int16 | ✅ Yes | Integer |
Int32 | ✅ Yes | Integer |
Int64 | ✅ Yes | Integer |
Interval[DayTime] | ✅ Yes | String |
Interval[MonthDayNano] | ✅ Yes | String |
Interval[Month] | ✅ Yes | String |
JSON | ✅ Yes | String |
Large Binary | ✅ Yes | Bytes |
Large List | ✅ Yes | String |
Large String | ✅ Yes | String |
List | ✅ Yes | String or List † |
MAC | ✅ Yes | String |
Map | ✅ Yes | String |
String | ✅ Yes | String |
Struct | ✅ Yes | String |
Timestamp[ms] | ✅ Yes | String * |
Timestamp[ns] | ✅ Yes | String |
Timestamp[s] | ✅ Yes | String |
Timestamp[us] | ✅ Yes | String |
UUID | ✅ Yes | String |
Uint8 | ✅ Yes | String |
Uint16 | ✅ Yes | Integer |
Uint32 | ✅ Yes | Integer |
Uint64 | ✅ Yes | Integer |
Union | ✅ Yes | String |
String-persisted data types are encoded according to the Arrow String Representation specification.
Notes #
* Timestamps are converted to strings in the format
yyyy-MM-dd HH:mm:ss.SSSSSSSSS
(UTC timezone) (e.g. 2021-01-01 00:00:00.000000000
). _cq_sync_time
column is persisted in native Timestamp
type.† List types are persisted as-is only if
complete_types
option is enabled. Otherwise, they are converted to strings.NUL
bytes are stripped from strings.Licenses #
The following tools / packages are used in this plugin:
Name | License |
---|---|
github.com/adrg/xdg | MIT |
github.com/apache/arrow/go/v13 | Apache-2.0 |
github.com/apache/arrow-go/v18 | Apache-2.0 |
github.com/apache/tinkerpop/gremlin-go/v3/driver | Apache-2.0 |
github.com/apapsch/go-jsonmerge/v2 | MIT |
github.com/aws/aws-sdk-go-v2 | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/config | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/credentials | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/feature/ec2/imds | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/internal/configsources | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/internal/ini | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/internal/sync/singleflight | BSD-3-Clause |
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/licensemanager | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/marketplacemetering | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/sso | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/ssooidc | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/sts | Apache-2.0 |
github.com/aws/smithy-go | Apache-2.0 |
github.com/aws/smithy-go/internal/sync/singleflight | BSD-3-Clause |
github.com/bahlo/generic-list-go | BSD-3-Clause |
github.com/buger/jsonparser | MIT |
github.com/cenkalti/backoff/v4 | MIT |
github.com/cloudquery/cloudquery-api-go | MPL-2.0 |
github.com/cloudquery/plugin-pb-go | MPL-2.0 |
github.com/cloudquery/plugin-sdk/v2/internal/glob | MIT |
github.com/cloudquery/plugin-sdk/v2/schema | MIT |
github.com/cloudquery/plugin-sdk/v2/types | MPL-2.0 |
github.com/cloudquery/plugin-sdk/v4 | MPL-2.0 |
github.com/cloudquery/plugin-sdk/v4/glob | MIT |
github.com/cloudquery/plugin-sdk/v4/scalar | MIT |
github.com/davecgh/go-spew/spew | ISC |
github.com/ghodss/yaml | MIT |
github.com/go-logr/logr | Apache-2.0 |
github.com/go-logr/stdr | Apache-2.0 |
github.com/goccy/go-json | MIT |
github.com/google/flatbuffers/go | Apache-2.0 |
github.com/google/uuid | BSD-3-Clause |
github.com/gorilla/websocket | BSD-3-Clause |
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors | Apache-2.0 |
github.com/grpc-ecosystem/grpc-gateway/v2 | BSD-3-Clause |
github.com/hashicorp/go-cleanhttp | MPL-2.0 |
github.com/hashicorp/go-retryablehttp | MPL-2.0 |
github.com/huandu/xstrings | MIT |
github.com/invopop/jsonschema | MIT |
github.com/klauspost/compress | Apache-2.0 |
github.com/klauspost/compress/internal/snapref | BSD-3-Clause |
github.com/klauspost/compress/zstd/internal/xxhash | MIT |
github.com/mailru/easyjson | MIT |
github.com/mattn/go-colorable | MIT |
github.com/mattn/go-isatty | MIT |
github.com/nicksnyder/go-i18n/v2 | MIT |
github.com/oapi-codegen/runtime | Apache-2.0 |
github.com/pierrec/lz4/v4 | BSD-3-Clause |
github.com/pmezard/go-difflib/difflib | BSD-3-Clause |
github.com/rs/zerolog | MIT |
github.com/santhosh-tekuri/jsonschema/v6 | Apache-2.0 |
github.com/spf13/cobra | Apache-2.0 |
github.com/spf13/pflag | BSD-3-Clause |
github.com/stretchr/testify | MIT |
github.com/thoas/go-funk | MIT |
github.com/wk8/go-ordered-map/v2 | Apache-2.0 |
github.com/zeebo/xxh3 | BSD-2-Clause |
go.opentelemetry.io/otel | Apache-2.0 |
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp | Apache-2.0 |
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp | Apache-2.0 |
go.opentelemetry.io/otel/exporters/otlp/otlptrace | Apache-2.0 |
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp | Apache-2.0 |
go.opentelemetry.io/otel/log | Apache-2.0 |
go.opentelemetry.io/otel/metric | Apache-2.0 |
go.opentelemetry.io/otel/sdk | Apache-2.0 |
go.opentelemetry.io/otel/sdk/log | Apache-2.0 |
go.opentelemetry.io/otel/sdk/metric | Apache-2.0 |
go.opentelemetry.io/otel/trace | Apache-2.0 |
go.opentelemetry.io/proto/otlp | Apache-2.0 |
golang.org/x/exp | BSD-3-Clause |
golang.org/x/net | BSD-3-Clause |
golang.org/x/sync/errgroup | BSD-3-Clause |
golang.org/x/sys | BSD-3-Clause |
golang.org/x/text | BSD-3-Clause |
golang.org/x/xerrors | BSD-3-Clause |
google.golang.org/genproto/googleapis/api/httpbody | Apache-2.0 |
google.golang.org/genproto/googleapis/rpc/status | Apache-2.0 |
google.golang.org/grpc | Apache-2.0 |
google.golang.org/protobuf | BSD-3-Clause |
gopkg.in/yaml.v2 | Apache-2.0 |
gopkg.in/yaml.v3 | MIT |