The CloudQuery S3 source plugin reads parquet files and loads them into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more)
Publisher
cloudquery
Latest version
v1.8.16
Type
Source
Platforms
Date Published
Loading plugin documentation
We use tracking cookies to understand how you use the product and help us improve it. Please accept cookies to help us improve. You can always opt out later via the link in the footer.
s3://<bucket>/datafile_0.parquet
s3://<bucket>/datafile_1.parquet
s3://<bucket>/data/2024/datafile_1.parquet
s3://<bucket>/data/2024/02/14/14/15/datafile_3.parquet
s3://<bucket>/data/2024/02/14/14/15/datafile_4.parquet
datafile_0.parquet --> datafile_0
datafile_1.parquet --> datafile_1
data/2024/datafile_1.parquet --> data_2024
data/2024/02/14/14/15/datafile_3.parquet --> data_2024_02_14_14_15
data/2024/02/14/14/15/datafile_4.parquet --> data_2024_02_14_14_15
s3:GetObject and s3:ListBucket permissions on the bucket and objects that you are trying to sync.AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN environment variables.credentials and config files in ~/.aws (the credentials file takes priority).aws sso to authenticate cloudquery - you can read more about it here.AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN environment variables (AWS_SESSION_TOKEN can be optional for some accounts).
For information on obtaining credentials, see the AWS guide.export AWS_ACCESS_KEY_ID='{Your AWS Access Key ID}'
export AWS_SECRET_ACCESS_KEY='{Your AWS secret access key}'
export AWS_SESSION_TOKEN='{Your AWS session token}'
credentials and config files in the .aws directory in your home folder.
The contents of these files are practically interchangeable, but CloudQuery will prioritize credentials in the credentials file.credentials file:[default]
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY
[myprofile]
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY
AWS_PROFILE environment variable (On Linux/Mac, similar for Windows):export AWS_PROFILE=myprofile
aws sts get-session-token --serial-number <YOUR_MFA_SERIAL_NUMBER> --token-code <YOUR_MFA_TOKEN_CODE> --duration-seconds 3600
export AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY_ID>
export AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_ACCESS_KEY>
export AWS_SESSION_TOKEN=<YOUR_SESSION_TOKEN>
backend_options must be set in the spec (as shown below). This is documented in the Managing Incremental Tables section.kind: source
spec:
name: s3
path: cloudquery/s3
registry: cloudquery
version: "v1.8.16"
tables: ["*"]
destinations: ["postgresql"]
backend_options:
table_name: "cq_state_s3"
connection: "@@plugins.postgresql.connection"
# Learn more about the configuration options at https://cql.ink/s3_source
spec:
# TODO: Update it with the actual spec
bucket: "<BUCKET_NAME>"
region: "<REGION>"
# Optional parameters
# path_prefix: ""
# rows_per_record: 500
# concurrency: 50
bucket (string) (required)region (string) (required)local_profile (string) (optional) (default: will use current credentials)[default]
aws_access_key_id=xxxx
aws_secret_access_key=xxxx
[user1]
aws_access_key_id=xxxx
aws_secret_access_key=xxxx
local_profile should be set to either default or user1.path_prefix (string) (optional) (default: "")filetype (string) (optional) (default: parquet)parquet is supported.rows_per_record (integer) (optional) (default: 500)concurrency (integer) (optional) (default: 50)