The CloudQuery File plugin syncs parquet files to any of the supported CloudQuery destinations
Publisher
cloudquery
Latest version
v1.7.1
Type
Source
Platforms
Date Published
Loading plugin documentation
CloudQuery's use of cookies
We use tracking cookies to understand how you use the product and help us improve it. Your consent is required before we can enable these cookies. You can opt out via the link in the footer.
kind: source
spec:
name: file
path: cloudquery/file
registry: cloudquery
version: "v1.7.1"
tables: ["*"]
destinations: ["postgresql"]
# Learn more about the configuration options at https://cql.ink/file_source
spec:
files_dir: "/path/to/files-to-sync" # required. Path to the directory with files to sync
# Optional parameters
# rows_per_record: 500
# concurrency: 50
files_dir (string) (required).parquet extension will be synced.rows_per_record (integer) (optional) (default: 500)concurrency (integer) (optional) (default: 50)kind: source
spec:
name: file
path: cloudquery/file
registry: cloudquery
version: "v1.7.1"
destinations: [postgresql]
tables: ["*"]
spec:
files_dir: "/path/to/cost_and_usage_reports" # Update this value to the local directory with your AWS Cost and Usage Reports
---
kind: source
spec:
name: aws
path: cloudquery/aws
registry: cloudquery
version: "v33.7.0"
destinations: [postgresql]
tables: ["*"]
skip_tables:
- aws_ec2_vpc_endpoint_services
- aws_cloudtrail_events
- aws_docdb_cluster_parameter_groups
- aws_docdb_engine_versions
- aws_ec2_instance_types
- aws_elasticache_engine_versions
- aws_elasticache_parameter_groups
- aws_elasticache_reserved_cache_nodes_offerings
- aws_elasticache_service_updates
- aws_iam_group_last_accessed_details
- aws_iam_policy_last_accessed_details
- aws_iam_role_last_accessed_details
- aws_iam_user_last_accessed_details
- aws_neptune_cluster_parameter_groups
- aws_neptune_db_parameter_groups
- aws_rds_cluster_parameter_groups
- aws_rds_db_parameter_groups
- aws_rds_engine_versions
- aws_servicequotas_services
---
kind: destination
spec:
name: postgresql
path: cloudquery/postgresql
registry: cloudquery
version: "v8.14.1"
spec:
connection_string: postgresql://postgres:pass@localhost:5432/postgres