We use tracking cookies to understand how you use the product and help us improve it. Please accept cookies to help us improve. You can always opt out later via the link in the footer.
cloudquery
needs to be authenticated. Credentials can be inserted in the connection_string
field.localhost:5432
. The (top level) spec section is described in the Source Spec Reference.kind: source
spec:
name: "postgresql"
registry: "cloudquery"
path: "cloudquery/postgresql"
version: "v6.11.2"
tables: ["*"]
destinations: ["postgresql"]
# Learn more about the configuration options at https://cql.ink/postgresql_source
spec:
connection_string: "${POSTGRESQL_CONNECTION_STRING}" # set the environment variable in a format like postgres://postgres:pass@localhost:5432/postgres?sslmode=disable
# Optional parameters:
# cdc_id: "postgresql" # Set to a unique string per source to enable Change Data Capture mode (logical replication, or CDC)
# pgx_log_level: error
# rows_per_record: 500
connection_string
(string
) (required)pgxpool
"postgres://jack:secret@localhost:5432/mydb?sslmode=prefer"
connect with tcp and prefer TLS"postgres://jack:secret@localhost:5432/mydb?sslmode=disable&application_name=pgxtest&search_path=myschema&connect_timeout=5"
be explicit with all options"postgres://localhost:5432/mydb?sslmode=disable"
connect with os username cloudquery is being run as"postgres:///mydb?host=/tmp"
connect over unix socket"dbname=mydb"
unix domain socket, just specifying the db name - useful if you want to use peer authentication"user=jack password=jack\\'ssooper\\\\secret host=localhost port=5432 dbname=mydb sslmode=disable"
DSN with escaped backslash and single quotepgx_log_level
(string
) (optional) (default: error
)cdc_id
(string
) (optional)rows_per_record
(integer
) (optional) (default: 500
)destination_table_name
(string
) (optional) (default: {{TABLE}}
){{TABLE}}
will be replaced with the table name{{UUID}}
will be replaced with a random UUID in raw format to uniquely identify each table (*){{YEAR}}
will be replaced with the current year in YYYY
format (*){{MONTH}}
will be replaced with the current month in MM
format (*){{DAY}}
will be replaced with the current day in DD
format (*){{HOUR}}
will be replaced with the current hour in HH
format (*){{MINUTE}}
will be replaced with the current minute in mm
format (*){{TABLE}}
or {{UUID}}
is required. Dynamic placeholders (denoted with *
) are not allowed in CDC mode (when cdc_id
is set)UTC
and will be the time sync started.SHOW wal_level;
SELECT setting FROM pg_settings WHERE name='wal_level'
replication
but for CDC to work we need to set it to logical
(This can only be done on the database startup).docker run -p 5432:5432 -e POSTGRES_PASSWORD=pass -d postgres:11 -c "wal_level=logical"
wal_level
to logical
in the postgresql.conf
file and restart the database.rds.logical_replication
to 1
and restart the database.rds.logical_replication
to 1
and restart the database.cloudsql.logical_deconging
configuration to on. See full documentation on GCP documentationCREATE TABLE
, ALTER TABLE
, etc)TRUNCATE
statements (e.g. TRUNCATE TABLE table_name
)DELETE
statements (e.g. DELETE FROM table_name
)user:pass
in the connection string, the password needs to be longer than 112 bits.kind: source
spec:
name: postgresql
path: cloudquery/postgresql
registry: cloudquery
version: "v6.11.2-fips"
...