Opensearch Destination Plugin

Example config #

kind: destination
spec:
  name: opensearch
  path: cloudquery/opensearch
  registry: cloudquery
  version: "v1.2.4"
  write_mode: "overwrite-delete-stale"
  spec:
    # Optional parameters
    # addresses: ["http://localhost:9200"]
    # username: ""
    # password: ""
    # ca_cert: ""
    # concurrency: 5 # default: number of CPUs
    # batch_size: 1000
    # batch_size_bytes: 5242880 # 5 MiB
    # aws_signing:
    #  region: "us-west-2"

Opensearch Spec #

• addresses ([]string) (optional) (default: ["http://localhost:9200"])

  A list of Opensearch nodes to use.
• username (string) (optional)

  Username for HTTP Basic Authentication.
• password (string) (optional)

  Password for HTTP Basic Authentication.
• ca_cert (string) (optional)

  PEM-encoded certificate authorities. When set, an empty certificate pool will be created, and the certificates will be appended to it. See file variable substitution for how to read this value from a file.
• concurrency (string) (optional) (default: number of CPUs)

  Number of concurrent worker goroutines to use for indexing.
• batch_size (integer) (optional) (default: 1000)

  Maximum number of items that may be grouped together to be written in a single write.
• batch_size_bytes (integer) (optional) (default: 5242880 (5 MiB))

  Maximum size of items that may be grouped together to be written in a single write.
• aws_signing (aws_signing_spec) (optional)

  AWS signing configuration used to enable AWS request signing for requests to the AWS Opensearch Service.

aws_signing_spec #

• region (string) (required)

  AWS region to use for signing.

Index Template Creation #

Index Naming #

• append: indexes will be named using the format <table_name>-<YYYY-MM-DD>. In other words, a new index will be created every day the table is synced. Entries will never be overwritten.
• overwrite: indexes will be named using the format <table_name>. Objects with duplicate primary keys will be overwritten.
• overwrite-delete-stale: indexes will be named using the format <table_name>. Objects with duplicate primary keys will be overwritten, and any objects that are not present in the current sync will be deleted.

Querying From Opensearch Dashboard #

Underlying library #

Authentication #

export AWS_ACCESS_KEY_ID={Your AWS Access Key ID}
export AWS_SECRET_ACCESS_KEY={Your AWS secret access key}
export AWS_SESSION_TOKEN={Your AWS session token}

[default]
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY

[myprofile]
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY

export AWS_PROFILE=myprofile

aws sts get-session-token --serial-number <YOUR_MFA_SERIAL_NUMBER> --token-code <YOUR_MFA_TOKEN_CODE> --duration-seconds 3600

export AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY_ID>
export AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_ACCESS_KEY>
export AWS_SESSION_TOKEN=<YOUR_SESSION_TOKEN>

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/CloudquerySyncRole"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-west-2:123456789012:domain/my-domain/*"
    }
  ]
}