We use tracking cookies to understand how you use the product and help us improve it. Please accept cookies to help us improve. You can always opt out later via the link in the footer.
s3://<bucket>/datafile_0.parquet
s3://<bucket>/datafile_1.parquet
s3://<bucket>/data/2024/datafile_1.parquet
s3://<bucket>/data/2024/02/14/14/15/datafile_3.parquet
s3://<bucket>/data/2024/02/14/14/15/datafile_4.parquet
datafile_0.parquet --> datafile_0
datafile_1.parquet --> datafile_1
data/2024/datafile_1.parquet --> data_2024
data/2024/02/14/14/15/datafile_3.parquet --> data_2024_02_14_14_15
data/2024/02/14/14/15/datafile_4.parquet --> data_2024_02_14_14_15
s3:GetObject
and s3:ListBucket
permissions on the bucket and objects that you are trying to sync.AWS_ACCESS_KEY_ID
, AWS_SECRET_ACCESS_KEY
, AWS_SESSION_TOKEN
environment variables.credentials
and config
files in ~/.aws
(the credentials
file takes priority).aws sso
to authenticate cloudquery - you can read more about it here.AWS_ACCESS_KEY_ID
, AWS_SECRET_ACCESS_KEY
, and AWS_SESSION_TOKEN
environment variables (AWS_SESSION_TOKEN
can be optional for some accounts).
For information on obtaining credentials, see the AWS guide.export AWS_ACCESS_KEY_ID='{Your AWS Access Key ID}'
export AWS_SECRET_ACCESS_KEY='{Your AWS secret access key}'
export AWS_SESSION_TOKEN='{Your AWS session token}'
credentials
and config
files in the .aws
directory in your home folder.
The contents of these files are practically interchangeable, but CloudQuery will prioritize credentials in the credentials
file.credentials
file:[default]
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY
[myprofile]
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY
AWS_PROFILE
environment variable (On Linux/Mac, similar for Windows):export AWS_PROFILE=myprofile
aws sts get-session-token --serial-number <YOUR_MFA_SERIAL_NUMBER> --token-code <YOUR_MFA_TOKEN_CODE> --duration-seconds 3600
export AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY_ID>
export AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_ACCESS_KEY>
export AWS_SESSION_TOKEN=<YOUR_SESSION_TOKEN>
backend_options
must be set in the spec (as shown below). This is documented in the Managing Incremental Tables section.kind: source
spec:
name: s3
path: cloudquery/s3
registry: cloudquery
version: "v1.8.15"
tables: ["*"]
destinations: ["postgresql"]
backend_options:
table_name: "cq_state_s3"
connection: "@@plugins.postgresql.connection"
# Learn more about the configuration options at https://cql.ink/s3_source
spec:
# TODO: Update it with the actual spec
bucket: "<BUCKET_NAME>"
region: "<REGION>"
# Optional parameters
# path_prefix: ""
# rows_per_record: 500
# concurrency: 50
bucket
(string
) (required)region
(string
) (required)local_profile
(string
) (optional) (default: will use current credentials)[default]
aws_access_key_id=xxxx
aws_secret_access_key=xxxx
[user1]
aws_access_key_id=xxxx
aws_secret_access_key=xxxx
local_profile
should be set to either default
or user1
.path_prefix
(string
) (optional) (default: ""
)filetype
(string
) (optional) (default: parquet
)parquet
is supported.rows_per_record
(integer
) (optional) (default: 500
)concurrency
(integer
) (optional) (default: 50
)