Set up process #

Authentication #

Configuration #

kind: source
spec:
  name: "postgresql"
  registry: "cloudquery"
  path: "cloudquery/postgresql"
  version: "v6.10.0"
  tables: ["*"]
  destinations: ["postgresql"]
  # Learn more about the configuration options at https://cql.ink/postgresql_source
  spec:
    connection_string: "${POSTGRESQL_CONNECTION_STRING}" # set the environment variable in a format like postgres://postgres:pass@localhost:5432/postgres?sslmode=disable
    # Optional parameters:
    # cdc_id: "postgresql" # Set to a unique string per source to enable Change Data Capture mode (logical replication, or CDC)
    # pgx_log_level: error
    # rows_per_record: 500

PostgreSQL Spec #

• connection_string (string) (required)

  Connection string to connect to the database. This can be a URL or a DSN, as per pgxpool

  • "postgres://jack:secret@localhost:5432/mydb?sslmode=prefer" connect with tcp and prefer TLS
  • "postgres://jack:secret@localhost:5432/mydb?sslmode=disable&application_name=pgxtest&search_path=myschema&connect_timeout=5" be explicit with all options
  • "postgres://localhost:5432/mydb?sslmode=disable" connect with os username cloudquery is being run as
  • "postgres:///mydb?host=/tmp" connect over unix socket
  • "dbname=mydb" unix domain socket, just specifying the db name - useful if you want to use peer authentication
  • "user=jack password=jack\\'ssooper\\\\secret host=localhost port=5432 dbname=mydb sslmode=disable" DSN with escaped backslash and single quote
• pgx_log_level (string) (optional) (default: error)

  Available: error, warn, info, debug, trace. Defines what pgx call events should be logged.
• cdc_id (string) (optional)

  If set to a non-empty string the source plugin will start syncing CDC via PostgreSQL logical replication in real-time. The value should be unique across all sources.
• rows_per_record (integer) (optional) (default: 500)

  Amount of rows to be packed into a single Apache Arrow record to be sent over the wire during sync (or initial sync in the CDC mode).
• destination_table_name (string) (optional) (default: {{TABLE}})

  The destination table name to write the data to. (Preview Feature)

  Supports the following placeholder variables:

  • {{TABLE}} will be replaced with the table name
  • {{UUID}} will be replaced with a random UUID in raw format to uniquely identify each table (*)
  • {{YEAR}} will be replaced with the current year in YYYY format (*)
  • {{MONTH}} will be replaced with the current month in MM format (*)
  • {{DAY}} will be replaced with the current day in DD format (*)
  • {{HOUR}} will be replaced with the current hour in HH format (*)
  • {{MINUTE}} will be replaced with the current minute in mm format (*)

  At least one of {{TABLE}} or {{UUID}} is required. Dynamic placeholders (denoted with *) are not allowed in CDC mode (when cdc_id is set)

  Note that timestamps are in UTC and will be the time sync started.