This destination plugin lets you sync data from a CloudQuery source to an Opensearch compatible service.
Publisher
cloudquery
Latest version
v1.3.1
Type
Destination
Platforms
Date Published
Loading plugin documentation
CloudQuery's use of cookies
We use tracking cookies to understand how you use the product and help us improve it. Please accept cookies to help us improve. You can always opt out later via the link in the footer.
localhost:9200:kind: destination
spec:
name: opensearch
path: cloudquery/opensearch
registry: cloudquery
version: "v1.3.1"
write_mode: "overwrite-delete-stale"
spec:
# Optional parameters
# addresses: ["http://localhost:9200"]
# username: ""
# password: ""
# ca_cert: ""
# concurrency: 5 # default: number of CPUs
# batch_size: 1000
# batch_size_bytes: 5242880 # 5 MiB
# aws_signing:
# region: "us-west-2"
batch_size and batch_size_bytes.append, overwrite and overwrite-delete-stale write modes. The default write mode is overwrite-delete-stale.addresses ([]string) (optional) (default: ["http://localhost:9200"])username (string) (optional)password (string) (optional)ca_cert (string) (optional)concurrency (string) (optional) (default: number of CPUs)batch_size (integer) (optional) (default: 1000)batch_size_bytes (integer) (optional) (default: 5242880 (5 MiB))aws_signing (aws_signing_spec) (optional)region (string) (required)--no-migrate option when running cloudquery sync.append: indexes will be named using the format <table_name>-<YYYY-MM-DD>. In other words, a new index will be created every day the table is synced. Entries will never be overwritten.overwrite: indexes will be named using the format <table_name>. Objects with duplicate primary keys will be overwritten.overwrite-delete-stale: indexes will be named using the format <table_name>. Objects with duplicate primary keys will be overwritten, and any objects that are not present in the current sync will be deleted.<table_name>-*. For example, if you have a table named aws_ec2_instances, you should create a data view with index pattern named aws_ec2_instances-*. One useful feature of Opensearch, however, is the ability to query across all data. To do this for the aws source plugin, for example, you may use an index pattern named aws_*. This will then allow queries across all tables synced by the aws source plugin.AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN environment variables.credentials and config files in ~/.aws (the credentials file takes priority).aws sso to authenticate cloudquery - you can read more about it here.AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and
AWS_SESSION_TOKEN environment variables (AWS_SESSION_TOKEN can be optional for some accounts). For information on obtaining credentials, see the
AWS guide.export AWS_ACCESS_KEY_ID={Your AWS Access Key ID}
export AWS_SECRET_ACCESS_KEY={Your AWS secret access key}
export AWS_SESSION_TOKEN={Your AWS session token}
credentials and config files in the .aws directory in your home folder.
The contents of these files are practically interchangeable, but CloudQuery will prioritize credentials in the credentials file.credentials file:[default]
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY
[myprofile]
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY
AWS_PROFILE environment variable (On Linux/Mac, similar for Windows):export AWS_PROFILE=myprofile
aws sts get-session-token --serial-number <YOUR_MFA_SERIAL_NUMBER> --token-code <YOUR_MFA_TOKEN_CODE> --duration-seconds 3600
export AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY_ID>
export AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_ACCESS_KEY>
export AWS_SESSION_TOKEN=<YOUR_SESSION_TOKEN>
arn:aws:iam::123456789012:role/CloudquerySyncRole to the OpenSearch role cloudquery-sync-role, configured with the required permissions for a Cloudquery sync.arn:aws:iam::123456789012:role/CloudquerySyncRole to access the domain. The following is an example of an OpenSearch domain access policy that allows the IAM role arn:aws:iam::123456789012:role/CloudquerySyncRole to access the domain:{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789012:role/CloudquerySyncRole"
},
"Action": "es:*",
"Resource": "arn:aws:es:us-west-2:123456789012:domain/my-domain/*"
}
]
}
cloudquery-sync-role.cluster_monitorcluster_composite_opsindices:admin/index_template/put*:indices_allcrudarn:aws:iam::123456789012:role/CloudquerySyncRole to the backend roles and click "Map".arn:aws:iam::123456789012:role/CloudquerySyncRole will now be authorized by the OpenSearch role cloudquery-sync-role.