Official

Premium

Microsoft Entra ID (Azure AD)

The CloudQuery Microsoft Entra ID (Azure AD) source plugin extracts your Microsoft Entra ID information and loads it into any supported CloudQuery destination

Publisher

cloudquery

Latest version

v1.6.0

Type

Source

Platforms

Date Published

October 22, 2024

Price per 1M rows

Starting from $15

monthly free quota

1M rows

Documentation Tables Changelog Destinations

Overview Incremental syncing

Set up process #

brew install cloudquery/tap/cloudquery

1. Download CLI and login

See installation options

2. Create source and destination configs

Plugin configuration

cloudquery sync entraid.yml postgresql.yml

3. Run the sync

CloudQuery sync

Overview #

The CloudQuery Microsoft Entra ID (Azure AD) source plugin extracts your Microsoft Entra ID information and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more).

Authentication #

The Microsoft Entra ID source plugin uses the Microsoft Graph API. Authentication is done using a service principal, you can create one using the Azure CLI. First, install the Azure CLI (az).

Then, login with the Azure CLI:

az login

Then, create the service principal the plugin will use to access Entra ID. WARNING: The output of az ad sp create-for-rbac contains credentials that you must protect - Make sure to handle with appropriate care. This example uses bash - The commands for CMD and PowerShell are similar.

az ad sp create-for-rbac --name CloudQuerySP

You can choose any name you'd like for your service-principal, CloudQuerySP is an example. If the service principal doesn't exist it will create a new one, otherwise it will update the existing one

The output of az ad sp create-for-rbac should look like this:

{
  "appId": "YOUR AZURE_CLIENT_ID",
  "displayName": "CloudQuerySP",
  "password": "YOUR AZURE_CLIENT_SECRET",
  "tenant": "YOUR AZURE_TENANT_ID"
}

Exporting environment variables #

Next, you need to export the environment variables that the plugin will use to sync Entra ID resources. Copy them from the output of az ad sp create-for-rbac. The example shows how to export environment variables for Linux - exporting for CMD and PowerShell is similar.

AZURE_TENANT_ID is tenant in the JSON.
AZURE_CLIENT_ID is appId in the JSON.
AZURE_CLIENT_SECRET is password in the JSON.

export AZURE_TENANT_ID=<YOUR AZURE_TENANT_ID>
export AZURE_CLIENT_ID=<YOUR AZURE_CLIENT_ID>
export AZURE_CLIENT_SECRET=<YOUR AZURE_CLIENT_SECRET>

Adding permissions to the service principal #

The Microsoft Graph API requires the service principal to have specific permissions to access the data. We can cover most tables in the plugin with the Global Reader role, but some tables require additional roles. Below is a list of permissions required for each table (some permissions cover more than one table).

Table	Required Permission
`entraid_auditlogs_directoryaudits`	`AuditLog.Read.All`
`entraid_auditlogs_signins`	`AuditLog.Read.All`
`entraid_devicemanagement_configurations`	`DeviceManagementConfiguration.Read.All`
`entraid_devicemanagement_manageddevices`	`DeviceManagementManagedDevices.Read.All`
`entraid_group_lifecyclepolicies`	`Policy.Read.All`
`entraid_identity_conditional_access_policies`	`Policy.Read.All`
`entraid_identityprotection_riskdetections`	`IdentityRiskEvent.Read.All`
`entraid_identityprotection_riskyserviceprincipals`	`IdentityRiskyServicePrincipal.Read.All`
`entraid_identityprotection_riskyusers`	`IdentityRiskyUser.Read.All`
`entraid_identityprotection_serviceprincipalriskdetections`	`IdentityRiskEvent.Read.All`
`entraid_policies_identity_security_defaults_enforcement_policy`	`Policy.Read.All`,`Organization.Read.All`
`entraid_rolemanagement_roleassignmentscheduleinstances`	`RoleAssignmentSchedule.Read.Directory`
`entraid_rolemanagement_roleassignmentschedulerequests`	`RoleAssignmentSchedule.ReadWrite.Directory`
`entraid_rolemanagement_roleassignmentschedules`	`RoleAssignmentSchedule.Read.Directory`

To add the permissions open the Azure portal, search for Microsoft Entra ID and click the Microsoft Entra ID service to open the overview page. Then, click on Roles and administrators as shown in the image below.

Under Roles and administrators, search for the Global Reader role and click on it.

Under the Global Reader role, under the Manage->Assignments sidebar location, click on Add assignments.

Under Add assignments, click no members selected to open a search box for members.

Search for the service principal you created earlier and click on it.

Approve the next screens to finalize the assignment.

To add specific permissions open App registrations from the Microsoft Entra ID overview page and click on the service principal you created earlier.

Under manage->API permissions click on Add a permission.

Chose Microsoft Graph.

Then click on Application permissions.

Search for the permission you need to add, for example AuditLog.Read.All, select it and click on Add permissions.

Repeat the process for all permissions required by the tables you want to sync. After adding all permissions, click on Grant admin consent for <your tenant> to finalize the process.

After granting the permissions, you should see a message like the one below.

That is it! You have successfully added the required permissions to the service principal.

Configuration #

This example syncs from Entra ID to a PostgreSQL database destination. The (top level) source spec section is described in the Source Spec Reference.

kind: source
# Common source-plugin configuration
spec:
  name: entraid
  path: cloudquery/entraid
  registry: cloudquery
  version: "v1.6.0"
  tables: ["*"]
  destinations: ["postgresql"]

  # Entra ID specific configuration
  # Learn more about the configuration options at https://cql.ink/entraid_source
  spec:
    # Optional parameters
    # concurrency: 50000

Entra ID Spec #

concurrency (integer) (optional) (default: 50000)
A best effort maximum number of Go routines to use. Lower this number to reduce memory usage.
scheduler (string) (optional) (default: dfs)
The scheduler to use when determining the priority of resources to sync. Supported values are dfs (depth-first search), round-robin, shuffle and shuffle-queue.
For more information about this, see performance tuning.

Incremental syncing #

The Entra ID plugin support incremental syncing via MS Graph API delta queries. The following tables support incremental syncing:

entraid_applications using https://learn.microsoft.com/en-us/graph/api/application-delta
entraid_directoryroles using https://learn.microsoft.com/en-us/graph/api/directoryrole-delta
entraid_groups using https://learn.microsoft.com/en-us/graph/api/group-delta
entraid_serviceprincipals using https://learn.microsoft.com/en-us/graph/api/serviceprincipal-delta
entraid_users using https://learn.microsoft.com/en-us/graph/api/user-delta

To enable incremental syncing for the above tables, set the backend_options configuration as shown in the example below. This is documented in the Managing Incremental Tables section.

kind: source
spec:
  name: "entraid"
  path: "cloudquery/entraid"
  version: "v1.6.0"
  destinations: ["postgresql"]
  tables: ["entraid_applications", "entraid_directoryroles", "entraid_groups", "entraid_serviceprincipals", "entraid_users"]
  backend_options:
    table_name: "cq_state_entraid"
    connection: "@@plugins.postgresql.connection"
---
kind: destination
spec:
  name: "postgresql"
  path: "cloudquery/postgresql"
  version: "v8.6.4"
  spec:
    connection_string: "postgresql://postgres:pass@localhost:5432/postgres?sslmode=disable"