Set up process #

s3://<bucket>/datafile_0.parquet
s3://<bucket>/datafile_1.parquet
s3://<bucket>/data/2024/datafile_1.parquet
s3://<bucket>/data/2024/02/14/14/15/datafile_3.parquet
s3://<bucket>/data/2024/02/14/14/15/datafile_4.parquet

datafile_0.parquet --> datafile_0
datafile_1.parquet --> datafile_1
data/2024/datafile_1.parquet --> data_2024
data/2024/02/14/14/15/datafile_3.parquet --> data_2024_02_14_14_15
data/2024/02/14/14/15/datafile_4.parquet --> data_2024_02_14_14_15

Configuration #

kind: source
spec:
  name: s3

  path: cloudquery/s3
  registry: cloudquery
  version: "v1.5.0"
  tables: ["*"]
  destinations: ["postgresql"]

  # Learn more about the configuration options at https://cql.ink/s3_source
  spec:
    # TODO: Update it with the actual spec 
    bucket: "<BUCKET_NAME>"
    region: "<REGION>"
    # Optional parameters
    # path_prefix: ""
    # rows_per_record: 500
    # concurrency: 50

S3 spec #

• bucket (string) (required)

  The name of the S3 bucket that will be synced.
• region (string) (required)

  The AWS region of the S3 bucket.
• local_profile (string) (optional) (default: will use current credentials)

  Local profile to use to authenticate this account with. Please note this should be set to the name of the profile.

  For example, with the following credentials file:

  [default]
  aws_access_key_id=xxxx
  aws_secret_access_key=xxxx
  
  [user1]
  aws_access_key_id=xxxx
  aws_secret_access_key=xxxx

  local_profile should be set to either default or user1.
• path_prefix (string) (optional) (default: "")

  The path prefix that will limit the files to sync.
• filetype (string) (optional) (default: parquet)

  Type of file that will be synced. Currently only parquet is supported.
• rows_per_record (integer) (optional) (default: 500)

  Amount of rows to be packed into a single Apache Arrow record to be sent over the wire during sync.
• concurrency (integer) (optional) (default: 50)

  Number of objects to sync in parallel. Negative values mean no limit.