Back to destination list
elasticsearch
Official

Elasticsearch

The Elasticsearch plugin syncs data from any CloudQuery source plugin(s) to an Elasticsearch cluster

Publisher

cloudquery

Repositorygithub.com
Latest version

v3.5.2

Type

Destination

Platforms
Date Published

Price

Free

Overview #

Elasticsearch Destination Plugin

The Elasticsearch plugin syncs data from any CloudQuery source plugin(s) to an Elasticsearch cluster.

Example config #

The following config will sync data to an Elasticsearch cluster running on localhost:9200:
kind: destination
spec:
  name: elasticsearch
  path: cloudquery/elasticsearch
  registry: cloudquery
  version: "v3.5.2"
  write_mode: "overwrite-delete-stale"
  spec:
    # Elastic Cloud configuration parameters
    cloud_id: "${ELASTICSEARCH_CLOUD_ID}"
    api_key: "${ELASTICSEARCH_API_KEY}"

    # Self-hosted Elasticsearch configuration parameters
    # addresses: ["http://localhost:9200"]
    # username: ""
    # password: ""
    # service_token: ""
    # certificate_fingerprint: ""
    # ca_cert: ""

    # Optional parameters
    # concurrency: 5 # default: number of CPUs
    # batch_size: 1000
    # batch_size_bytes: 5242880 # 5 MiB
The Elasticsearch destination utilizes batching, and supports batch_size and batch_size_bytes.
It supports append, overwrite and overwrite-delete-stale write modes. The default write mode is overwrite-delete-stale.

Elasticsearch Spec #

This is the spec used by the Elasticsearch destination plugin.
  • addresses ([]string) (optional) (default: ["http://localhost:9200"])
    A list of Elasticsearch nodes to use. Mutually exclusive with cloud_id.
  • username (string) (optional)
    Username for HTTP Basic Authentication.
  • password (string) (optional)
    Password for HTTP Basic Authentication.
  • cloud_id (string) (optional) (example: MyDeployment:abcdefgh)
    Endpoint for the Elasticsearch Service (https://elastic.co/cloud). Mutually exclusive with addresses.
  • api_key (string) (optional)
    Base64-encoded token for authorization; if set, overrides username/password and service token.
  • service_token (string) (optional)
    Service token for authorization; if set, overrides username/password.
  • certificate_fingerprint (string) (optional)
    SHA256 hex fingerprint given by Elasticsearch on first launch.
  • ca_cert (string) (optional)
    PEM-encoded certificate authorities. When set, an empty certificate pool will be created, and the certificates will be appended to it. See file variable substitution for how to read this value from a file.
  • concurrency (string) (optional) (default: number of CPUs)
    Number of concurrent worker goroutines to use for indexing.
  • batch_size (integer) (optional) (default: 1000)
    Maximum number of items that may be grouped together to be written in a single write.
  • batch_size_bytes (integer) (optional) (default: 5242880 (5 MiB))
    Maximum size of items that may be grouped together to be written in a single write.

Index Template Creation #

The Elasticsearch destination will create an index template for every table during the migration step. It is recommended that you use the generated index templates, as it will automatically create indexes with the correct mappings for the table. However, to skip index template creation (or use your own), you may use the --no-migrate option when running cloudquery sync.

Index Naming #

Index names will be formatted according to the selected write mode:
  • append: indexes will be named using the format <table_name>-<YYYY-MM-DD>. In other words, a new index will be created every day the table is synced. Entries will never be overwritten.
  • overwrite: indexes will be named using the format <table_name>. Objects with duplicate primary keys will be overwritten.
  • overwrite-delete-stale: indexes will be named using the format <table_name>. Objects with duplicate primary keys will be overwritten, and any objects that are not present in the current sync will be deleted.
Index templates will also be created such that they match the index names generated by the selected write mode.

Querying From Kibana #

To query data from Kibana, you will need to create data views (previously also known as "index patterns"). To query a specific table, the data view's index pattern should be in the format <table_name>-*. For example, if you have a table named aws_ec2_instances, you should create a data view with index pattern named aws_ec2_instances-*. One useful feature of Elasticsearch and Kibana, however, is the ability to query across all data. To do this for the aws source plugin, for example, you may use an index pattern named aws_*. This will then allow queries across all tables synced by the aws source plugin.

Underlying library #

We use the official go-elasticsearch package. It is tested against Elasticsearch 8.6.0. Please open an issue if you encounter any problems with this (or another) version.


Types #

Elasticsearch Types

The Elasticsearch destination (v2.0.0 and later) supports most Apache Arrow types. The following table shows the supported types and how they are mapped to Elasticsearch field data types.
Arrow Column TypeSupported?Elasticsearch Type
Binary✅ Yesbinary
Boolean✅ Yesboolean
Date32✅ Yesdate with format yyyy-MM-dd
Date64✅ Yesdate with format yyyy-MM-dd
Decimal✅ Yestext
Dense Union✅ Yestext
Dictionary✅ Yestext
Duration[ms]✅ Yestext
Duration[ns]✅ Yestext
Duration[s]✅ Yestext
Duration[us]✅ Yestext
Fixed Size List✅ YesUses type from list elements
Float16✅ Yeshalf_float
Float32✅ Yesfloat
Float64✅ Yesdouble
Inet✅ Yestext
Int8✅ Yesbyte
Int16✅ Yesshort
Int32✅ Yesinteger
Int64✅ Yeslong
Interval[DayTime]✅ Yesobject
Interval[MonthDayNano]✅ Yesobject
Interval[Month]✅ Yesobject
JSON✅ Yestext
Large Binary✅ Yesbyte
Large List✅ YesUses type from list elements
Large String✅ Yestext
List✅ YesUses type from list elements
MAC✅ Yestext
Map✅ Yesobject with key and value fields
String✅ Yestext
Struct✅ Yesobject
Time32[s]✅ Yesdate with format HH:mm:ss
Time32[ms]✅ Yesdate with format HH:mm:ss.SSS
Time64[us]✅ Yestext
Time64[ns]✅ Yestext
Timestamp[s]✅ Yesdate with format 2006-01-02T15:04:05Z
Timestamp[ms]✅ Yesdate with format 2006-01-02T15:04:05.999Z
Timestamp[us]✅ Yesdate with format 2006-01-02T15:04:05.999999Z"
Timestamp[ns]✅ Yesdate_nanos with format 2006-01-02T15:04:05.99999999Z
UUID✅ Yestext
Uint8✅ Yesunsigned_long
Uint16✅ Yesunsigned_long
Uint32✅ Yesunsigned_long
Uint64✅ Yesunsigned_long
Union✅ Yestext


Licenses #

The following tools / packages are used in this plugin:
NameLicense
github.com/adrg/xdgMIT
github.com/apache/arrow/go/v13Apache-2.0
github.com/apache/arrow/go/v17Apache-2.0
github.com/apapsch/go-jsonmerge/v2MIT
github.com/aws/aws-sdk-go-v2Apache-2.0
github.com/aws/aws-sdk-go-v2/configApache-2.0
github.com/aws/aws-sdk-go-v2/credentialsApache-2.0
github.com/aws/aws-sdk-go-v2/feature/ec2/imdsApache-2.0
github.com/aws/aws-sdk-go-v2/internal/configsourcesApache-2.0
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/iniApache-2.0
github.com/aws/aws-sdk-go-v2/internal/sync/singleflightBSD-3-Clause
github.com/aws/aws-sdk-go-v2/service/internal/accept-encodingApache-2.0
github.com/aws/aws-sdk-go-v2/service/internal/presigned-urlApache-2.0
github.com/aws/aws-sdk-go-v2/service/licensemanagerApache-2.0
github.com/aws/aws-sdk-go-v2/service/marketplacemeteringApache-2.0
github.com/aws/aws-sdk-go-v2/service/ssoApache-2.0
github.com/aws/aws-sdk-go-v2/service/ssooidcApache-2.0
github.com/aws/aws-sdk-go-v2/service/stsApache-2.0
github.com/aws/smithy-goApache-2.0
github.com/aws/smithy-go/internal/sync/singleflightBSD-3-Clause
github.com/bahlo/generic-list-goBSD-3-Clause
github.com/buger/jsonparserMIT
github.com/cenkalti/backoff/v4MIT
github.com/cloudquery/cloudquery-api-goMPL-2.0
github.com/cloudquery/plugin-pb-goMPL-2.0
github.com/cloudquery/plugin-sdk/v2/internal/globMIT
github.com/cloudquery/plugin-sdk/v2/schemaMIT
github.com/cloudquery/plugin-sdk/v2/typesMPL-2.0
github.com/cloudquery/plugin-sdk/v4MPL-2.0
github.com/cloudquery/plugin-sdk/v4/globMIT
github.com/cloudquery/plugin-sdk/v4/scalarMIT
github.com/davecgh/go-spew/spewISC
github.com/elastic/elastic-transport-go/v8/elastictransportApache-2.0
github.com/elastic/go-elasticsearch/v8Apache-2.0
github.com/elastic/go-elasticsearch/v8/typedapi/typesApache-2.0
github.com/elastic/go-elasticsearch/v8/typedapi/types/enums/licensestatusApache-2.0
github.com/elastic/go-elasticsearch/v8/typedapi/types/enums/licensetypeApache-2.0
github.com/ghodss/yamlMIT
github.com/go-logr/logrApache-2.0
github.com/go-logr/stdrApache-2.0
github.com/goccy/go-jsonMIT
github.com/google/flatbuffers/goApache-2.0
github.com/google/uuidBSD-3-Clause
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptorsApache-2.0
github.com/grpc-ecosystem/grpc-gateway/v2BSD-3-Clause
github.com/hashicorp/go-cleanhttpMPL-2.0
github.com/hashicorp/go-retryablehttpMPL-2.0
github.com/huandu/xstringsMIT
github.com/invopop/jsonschemaMIT
github.com/klauspost/compressApache-2.0
github.com/klauspost/compress/internal/snaprefBSD-3-Clause
github.com/klauspost/compress/zstd/internal/xxhashMIT
github.com/mailru/easyjsonMIT
github.com/mattn/go-colorableMIT
github.com/mattn/go-isattyMIT
github.com/oapi-codegen/runtimeApache-2.0
github.com/pierrec/lz4/v4BSD-3-Clause
github.com/pmezard/go-difflib/difflibBSD-3-Clause
github.com/rs/zerologMIT
github.com/santhosh-tekuri/jsonschema/v6Apache-2.0
github.com/segmentio/fasthash/fnv1aMIT
github.com/spf13/cobraApache-2.0
github.com/spf13/pflagBSD-3-Clause
github.com/stretchr/testifyMIT
github.com/thoas/go-funkMIT
github.com/wk8/go-ordered-map/v2Apache-2.0
github.com/zeebo/xxh3BSD-2-Clause
go.opentelemetry.io/otelApache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttpApache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttpApache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptraceApache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttpApache-2.0
go.opentelemetry.io/otel/logApache-2.0
go.opentelemetry.io/otel/metricApache-2.0
go.opentelemetry.io/otel/sdkApache-2.0
go.opentelemetry.io/otel/sdk/logApache-2.0
go.opentelemetry.io/otel/sdk/metricApache-2.0
go.opentelemetry.io/otel/traceApache-2.0
go.opentelemetry.io/proto/otlpApache-2.0
golang.org/x/expBSD-3-Clause
golang.org/x/netBSD-3-Clause
golang.org/x/sync/errgroupBSD-3-Clause
golang.org/x/sysBSD-3-Clause
golang.org/x/textBSD-3-Clause
golang.org/x/xerrorsBSD-3-Clause
google.golang.org/genproto/googleapis/api/httpbodyApache-2.0
google.golang.org/genproto/googleapis/rpc/statusApache-2.0
google.golang.org/grpcApache-2.0
google.golang.org/protobufBSD-3-Clause
gopkg.in/yaml.v2Apache-2.0
gopkg.in/yaml.v3MIT


Join our mailing list

Subscribe to our newsletter to make sure you don't miss any updates.

Legal

© 2024 CloudQuery, Inc. All rights reserved.

We use tracking cookies to understand how you use the product and help us improve it. Please accept cookies to help us improve. You can always opt out later via the link in the footer.