Elasticsearch Destination Plugin

Example config #

kind: destination
spec:
  name: elasticsearch
  path: cloudquery/elasticsearch
  registry: cloudquery
  version: "v3.5.1"
  write_mode: "overwrite-delete-stale"
  spec:
    # Elastic Cloud configuration parameters
    cloud_id: "${ELASTICSEARCH_CLOUD_ID}"
    api_key: "${ELASTICSEARCH_API_KEY}"

    # Self-hosted Elasticsearch configuration parameters
    # addresses: ["http://localhost:9200"]
    # username: ""
    # password: ""
    # service_token: ""
    # certificate_fingerprint: ""
    # ca_cert: ""

    # Optional parameters
    # concurrency: 5 # default: number of CPUs
    # batch_size: 1000
    # batch_size_bytes: 5242880 # 5 MiB

Elasticsearch Spec #

• addresses ([]string) (optional) (default: ["http://localhost:9200"])

  A list of Elasticsearch nodes to use. Mutually exclusive with cloud_id.
• username (string) (optional)

  Username for HTTP Basic Authentication.
• password (string) (optional)

  Password for HTTP Basic Authentication.
• cloud_id (string) (optional) (example: MyDeployment:abcdefgh)

  Endpoint for the Elasticsearch Service (https://elastic.co/cloud). Mutually exclusive with addresses.
• api_key (string) (optional)

  Base64-encoded token for authorization; if set, overrides username/password and service token.
• service_token (string) (optional)

  Service token for authorization; if set, overrides username/password.
• certificate_fingerprint (string) (optional)

  SHA256 hex fingerprint given by Elasticsearch on first launch.
• ca_cert (string) (optional)

  PEM-encoded certificate authorities. When set, an empty certificate pool will be created, and the certificates will be appended to it. See file variable substitution for how to read this value from a file.
• concurrency (string) (optional) (default: number of CPUs)

  Number of concurrent worker goroutines to use for indexing.
• batch_size (integer) (optional) (default: 1000)

  Maximum number of items that may be grouped together to be written in a single write.
• batch_size_bytes (integer) (optional) (default: 5242880 (5 MiB))

  Maximum size of items that may be grouped together to be written in a single write.

Index Template Creation #

Index Naming #

• append: indexes will be named using the format <table_name>-<YYYY-MM-DD>. In other words, a new index will be created every day the table is synced. Entries will never be overwritten.
• overwrite: indexes will be named using the format <table_name>. Objects with duplicate primary keys will be overwritten.
• overwrite-delete-stale: indexes will be named using the format <table_name>. Objects with duplicate primary keys will be overwritten, and any objects that are not present in the current sync will be deleted.

Querying From Kibana #

Underlying library #