We use tracking cookies to understand how you use the product and help us improve it. Please accept cookies to help us improve. You can always opt out later via the link in the footer.
localhost:9200
:kind: destination
spec:
name: opensearch
path: cloudquery/opensearch
registry: cloudquery
version: "v1.2.28"
write_mode: "overwrite-delete-stale"
spec:
# Optional parameters
# addresses: ["http://localhost:9200"]
# username: ""
# password: ""
# ca_cert: ""
# concurrency: 5 # default: number of CPUs
# batch_size: 1000
# batch_size_bytes: 5242880 # 5 MiB
# aws_signing:
# region: "us-west-2"
batch_size
and batch_size_bytes
.append
, overwrite
and overwrite-delete-stale
write modes. The default write mode is overwrite-delete-stale
.addresses
([]string
) (optional) (default: ["http://localhost:9200"]
)username
(string
) (optional)password
(string
) (optional)ca_cert
(string
) (optional)concurrency
(string
) (optional) (default: number of CPUs)batch_size
(integer
) (optional) (default: 1000
)batch_size_bytes
(integer
) (optional) (default: 5242880
(5 MiB))aws_signing
(aws_signing_spec) (optional)region
(string
) (required)--no-migrate
option when running cloudquery sync
.append
: indexes will be named using the format <table_name>-<YYYY-MM-DD>
. In other words, a new index will be created every day the table is synced. Entries will never be overwritten.overwrite
: indexes will be named using the format <table_name>
. Objects with duplicate primary keys will be overwritten.overwrite-delete-stale
: indexes will be named using the format <table_name>
. Objects with duplicate primary keys will be overwritten, and any objects that are not present in the current sync will be deleted.<table_name>-*
. For example, if you have a table named aws_ec2_instances
, you should create a data view with index pattern named aws_ec2_instances-*
. One useful feature of Opensearch, however, is the ability to query across all data. To do this for the aws
source plugin, for example, you may use an index pattern named aws_*
. This will then allow queries across all tables synced by the aws
source plugin.AWS_ACCESS_KEY_ID
, AWS_SECRET_ACCESS_KEY
, AWS_SESSION_TOKEN
environment variables.credentials
and config
files in ~/.aws
(the credentials
file takes priority).aws sso
to authenticate cloudquery - you can read more about it here.AWS_ACCESS_KEY_ID
, AWS_SECRET_ACCESS_KEY
, and
AWS_SESSION_TOKEN
environment variables (AWS_SESSION_TOKEN
can be optional for some accounts). For information on obtaining credentials, see the
AWS guide.export AWS_ACCESS_KEY_ID={Your AWS Access Key ID}
export AWS_SECRET_ACCESS_KEY={Your AWS secret access key}
export AWS_SESSION_TOKEN={Your AWS session token}
credentials
and config
files in the .aws
directory in your home folder.
The contents of these files are practically interchangeable, but CloudQuery will prioritize credentials in the credentials
file.credentials
file:[default]
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY
[myprofile]
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY
AWS_PROFILE
environment variable (On Linux/Mac, similar for Windows):export AWS_PROFILE=myprofile
aws sts get-session-token --serial-number <YOUR_MFA_SERIAL_NUMBER> --token-code <YOUR_MFA_TOKEN_CODE> --duration-seconds 3600
export AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY_ID>
export AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_ACCESS_KEY>
export AWS_SESSION_TOKEN=<YOUR_SESSION_TOKEN>
arn:aws:iam::123456789012:role/CloudquerySyncRole
to the OpenSearch role cloudquery-sync-role
, configured with the required permissions for a Cloudquery sync.arn:aws:iam::123456789012:role/CloudquerySyncRole
to access the domain. The following is an example of an OpenSearch domain access policy that allows the IAM role arn:aws:iam::123456789012:role/CloudquerySyncRole
to access the domain:{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789012:role/CloudquerySyncRole"
},
"Action": "es:*",
"Resource": "arn:aws:es:us-west-2:123456789012:domain/my-domain/*"
}
]
}
cloudquery-sync-role
.cluster_monitor
cluster_composite_ops
indices:admin/index_template/put
*
:indices_all
crud
arn:aws:iam::123456789012:role/CloudquerySyncRole
to the backend roles and click "Map".arn:aws:iam::123456789012:role/CloudquerySyncRole
will now be authorized by the OpenSearch role cloudquery-sync-role
.