Official

Premium

Salesforce

The CloudQuery Salesforce plugin extracts information from your Salesforce organization(s) and loads it into any supported CloudQuery destination

Publisher

cloudquery

Latest version

v5.1.0

Type

Source

Platforms

Date Published

October 22, 2024

Price per 1M rows

Starting from $15

monthly free quota

1M rows

Documentation Tables Changelog Destinations

Overview

Set up process #

brew install cloudquery/tap/cloudquery

1. Download CLI and login

See installation options

2. Create source and destination configs

Plugin configuration

cloudquery sync salesforce.yml postgresql.yml

3. Run the sync

CloudQuery sync

Overview #

The CloudQuery Salesforce plugin extracts information from your Salesforce organization(s) and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more).

Authentication #

You will need to create a Salesforce Connected App using the OAuth2.0 Client Credentials flow. This is a Salesforce app that will be used to authenticate with Salesforce. You can create a Connected App in the Salesforce Setup UI. Checkout the Salesforce documentation for more information. Or check out our latest guide to set up a connected app.

Configuration #

This example syncs from Salesforce to a Postgres destination:

kind: source
spec:
  name: slack
  path: cloudquery/salesforce
  registry: cloudquery
  version: "v5.1.0"
  tables: ["*"]
  destinations: ["postgresql"]
  # Salesforce specific configuration
  # Learn more about the configuration options at https://cql.ink/salesforce_source
  spec:
    domain_name: "${SF_DOMAIN_NAME}" // e.g. my-domain
    client_id: "${SF_CLIENT_ID}"
    client_secret: "${SF_CLIENT_SECRET}"

It assumes that the SF_DOMAIN_NAME, SF_CLIENT_ID and SF_CLIENT_SECRET environment variables are set.

Make sure you use environment variable expansion in production instead of committing the credentials to the configuration file directly.

Configuration #

This is the (nested) spec used by the Slack source plugin.

domain_name (string) (required)
This is the domain name of your Salesforce instance. It will be used to build the instance endpoint to which calls will be sent. For example, if your Salesforce instance is https://my-domain.my.salesforce.com, then the domain name is my-domain.
client_id (string) (required)
This is the consumer key of the connected app. This can be obtained by Creating a Salesforce connected app.
client_secret (string) (required)
This is the consumer secreted of the connected app. This can be obtained by Creating a Salesforce connected app.
include_objects ([]string) (optional)
By default the plugin will sync all Salesforce objects. If you want to limit the objects that are synced you can specify the objects you want to sync in this array.
exclude_objects ([]string) (optional)
By default the plugin will sync all Salesforce objects. If you want to exclude some objects from being synced you can specify the objects you want to exclude in this array.
concurrency (integer) (optional) (default: 10000)
A best effort maximum number of Go routines to use. Lower this number to reduce memory usage.
scheduler (string) (optional) (default: dfs) The scheduler to use when determining the priority of resources to sync. Supported values are dfs (depth-first search), round-robin, shuffle and shuffle-queue.
For more information about this, see performance tuning.

Example Queries #

List all Salesforce objects #

select distinct(object_type) from salesforce_objects;

List all Accounts #

select * from salesforce_objects where object_type = 'Account';

Creating Connected Guide #

Full documentation is available here. This guide will walk you through the steps to create a connected app.

Pre-requisites #

Salesforce developer account to be able to access the REST APIs
A Salesforce user with the permissions to create connected a

Step 1: Go To Connected App #

Go to Setup Screen and search for "App Manager" and click on "New Connected App".

Step 2: Create Connected App #

Fill in the fields as the following and click save:

Step 3: Set up an execution user for the Client Credentials Flow #

Salesforce uses the OAuth 2.0 Client Credentials Flow to authenticate the connected app. This flow requires a user to execute the flow. You can use the same user that created the connected app or create a new user (this approach is more secure as you have the possibility to create an API-only user).

Go to "Apps" > "Connected Apps" > "Manage Connected Apps" > "Edit Policies" and under the "Client Credentials flow" section pick the "Run As" user of your choice.

Step 4: Copy Consumer Key and Consumer Secret #

Now that this is configured click Manage Consumer Details to view your Consumer Key and Consumer Secret (those will be used as client_id and client_secret to configure your CloudQuery Salesforce source plugin).

And copy the Consumer Key and Consumer Secret somewhere safe so you can provide them as environment variables to the Salesforce plugin.

Step 5: IP Relaxation (optional) #

Depending on where you run CloudQuery your might need to change IP Relxation to Relax IP restrictions or configure the IP address of the machine you run CloudQuery on.