Example Configuration #

kind: source
spec:
  name: google-search-console
  path: cloudquery/google-search-console
  registry: cloudquery
  version: "v1.2.4"
  tables:
    - google_search_console_search_analytics_by_country
    - google_search_console_search_analytics_by_device
    - google_search_console_search_analytics_by_page
    - google_search_console_search_analytics_by_query
    - google_search_console_search_analytics_by_type
  destinations: ["postgresql"]
  backend_options:
    table_name: "cq_state_google_search_console"
    connection: "@@plugins.postgresql.connection"
  spec:
    # OAuth spec for authorization in Google Search Console.
    oauth:
      # Specify either access_token OR client_id and client_secret:
      access_token: "${GOOGLE_SEARCH_CONSOLE_ACCESS_TOKEN}"
      # client_id: "${GOOGLE_SEARCH_CONSOLE_CLIENT_ID}"
      # client_secret: "${GOOGLE_SEARCH_CONSOLE_CLIENT_SECRET}"
    # Optional fields:
    # site_urls:  # Site URLs to use, e.g. cloudquery.io. If empty, all available sites are used.
    #   - "cloudquery.io"
    # data_state: "" # "FINAL" or "ALL"; "FINAL" is recommended for incremental syncs
    # start_date: "" # in YYYY-MM-DD format, in PST (UTC - 8:00). Must be less than or equal to the end date. This value is included in the range. Defaults to one year ago.
    # end_date: "" # in YYYY-MM-DD format, in PST (UTC - 8:00). Must be greater than or equal to the start date. This value is included in the range. Defaults to the current date.

Authentication #

• Using an existing access token

  This token should be authorized for https://www.googleapis.com/auth/webmasters.readonly scope. For a quick local setup, you can use the OAuth 2.0 Playground with a Google Cloud project that has the Search Console API enabled, a web OAuth client ID and secret, to generate a valid access token.
• Using OAuth client ID & client secret

  You can get your own OAuth credentials using this guide.

Configuration Reference #

• oauth (OAuthSpec) (optional)

  OAuth spec. See OAuth spec for more details.
• data_state (string) (optional)

  Data state to be fetched. Possible values are "FINAL" and "ALL". "FINAL" is recommended for incremental syncs.
• start_date (string) (optional)

  Start date of the search analytics data, in YYYY-MM-DD format, in PST (UTC - 8:00). Must be less than or equal to the end date. This value is included in the range. Defaults to one year ago.
• end_date (string) (optional)

  End date of the search analytics data, in YYYY-MM-DD format, in PST (UTC - 8:00). Must be greater than or equal to the start date. This value is included in the range. Defaults to the current date.
• concurrency (integer) (optional) (default: 50000) A best effort maximum number of Go routines to use. Lower this number to reduce memory usage.
• scheduler (string) (optional) (default: dfs) The scheduler to use when determining the priority of resources to sync. Supported values are dfs (depth-first search), round-robin, shuffle and shuffle-queue.

  For more information about this, see performance tuning.

OAuth spec #

• access_token (string) (optional)

  An access token that you generated authorizing for https://www.googleapis.com/auth/webmasters.readonly scope (e.g., by using OAuth 2.0 Playground).
• client_id (string) (optional)

  OAuth 2.0 Client ID.
• client_secret (string) (optional)

  OAuth 2.0 Client secret.