Report an issue
Back to source list
entraid
Official
Premium

Microsoft Entra ID (Azure AD)

The CloudQuery Microsoft Entra ID (Azure AD) source plugin extracts your Microsoft Entra ID information and loads it into any supported CloudQuery destination

Publisher

cloudquery

Latest version

v1.8.1

Type

Source

Platforms
Date Published

Price per 1M rows

Starting from $17

monthly free quota

1M rows

Set up process #


brew install cloudquery/tap/cloudquery

1. Download CLI and login

See installation options

2. Create source and destination configs

Plugin configuration

cloudquery sync entraid.yml postgresql.yml

3. Run the sync

CloudQuery sync

Overview #

The CloudQuery Microsoft Entra ID (Azure AD) source plugin extracts your Microsoft Entra ID information and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more).

Authentication #

The Microsoft Entra ID source plugin uses the Microsoft Graph API. Authentication is done using a service principal, you can create one using the Azure CLI. First, install the Azure CLI (az).
Then, login with the Azure CLI:
az login
Then, create the service principal the plugin will use to access Entra ID. WARNING: The output of az ad sp create-for-rbac contains credentials that you must protect - Make sure to handle with appropriate care. This example uses bash - The commands for CMD and PowerShell are similar.
az ad sp create-for-rbac --name CloudQuerySP
You can choose any name you'd like for your service-principal, CloudQuerySP is an example. If the service principal doesn't exist it will create a new one, otherwise it will update the existing one
The output of az ad sp create-for-rbac should look like this:
{
  "appId": "YOUR AZURE_CLIENT_ID",
  "displayName": "CloudQuerySP",
  "password": "YOUR AZURE_CLIENT_SECRET",
  "tenant": "YOUR AZURE_TENANT_ID"
}

Exporting environment variables #

Next, you need to export the environment variables that the plugin will use to sync Entra ID resources. Copy them from the output of az ad sp create-for-rbac. The example shows how to export environment variables for Linux - exporting for CMD and PowerShell is similar.
  • AZURE_TENANT_ID is tenant in the JSON.
  • AZURE_CLIENT_ID is appId in the JSON.
  • AZURE_CLIENT_SECRET is password in the JSON.
export AZURE_TENANT_ID=<YOUR AZURE_TENANT_ID>
export AZURE_CLIENT_ID=<YOUR AZURE_CLIENT_ID>
export AZURE_CLIENT_SECRET=<YOUR AZURE_CLIENT_SECRET>

Adding permissions to the service principal #

The Microsoft Graph API requires the service principal to have specific permissions to access the data. We can cover most tables in the plugin with the Global Reader role, but some tables require additional roles. Below is a list of permissions required for each table (some permissions cover more than one table).
TableRequired Permission
entraid_auditlogs_directoryauditsAuditLog.Read.All
entraid_auditlogs_signinsAuditLog.Read.All
entraid_devicemanagement_configurationsDeviceManagementConfiguration.Read.All
entraid_devicemanagement_manageddevicesDeviceManagementManagedDevices.Read.All
entraid_group_lifecyclepoliciesPolicy.Read.All
entraid_identity_conditional_access_policiesPolicy.Read.All
entraid_identityprotection_riskdetectionsIdentityRiskEvent.Read.All
entraid_identityprotection_riskyserviceprincipalsIdentityRiskyServicePrincipal.Read.All
entraid_identityprotection_riskyusersIdentityRiskyUser.Read.All
entraid_identityprotection_serviceprincipalriskdetectionsIdentityRiskEvent.Read.All
entraid_policies_identity_security_defaults_enforcement_policyPolicy.Read.All,Organization.Read.All
entraid_reports_user_registration_detailsAuditLog.Read.All
entraid_rolemanagement_roleassignmentscheduleinstancesRoleAssignmentSchedule.Read.Directory
entraid_rolemanagement_roleassignmentschedulerequestsRoleAssignmentSchedule.ReadWrite.Directory
entraid_rolemanagement_roleassignmentschedulesRoleAssignmentSchedule.Read.Directory
To add the permissions open the Azure portal, search for Microsoft Entra ID and click the Microsoft Entra ID service to open the overview page. Then, click on Roles and administrators as shown in the image below.
Under Roles and administrators, search for the Global Reader role and click on it.
Under the Global Reader role, under the Manage->Assignments sidebar location, click on Add assignments.
Under Add assignments, click no members selected to open a search box for members.
Search for the service principal you created earlier and click on it.
Approve the next screens to finalize the assignment.
To add specific permissions open App registrations from the Microsoft Entra ID overview page and click on the service principal you created earlier.
Under manage->API permissions click on Add a permission.
Chose Microsoft Graph.
Then click on Application permissions.
Search for the permission you need to add, for example AuditLog.Read.All, select it and click on Add permissions.
Repeat the process for all permissions required by the tables you want to sync. After adding all permissions, click on Grant admin consent for <your tenant> to finalize the process.
After granting the permissions, you should see a message like the one below.
That is it! You have successfully added the required permissions to the service principal.

Configuration #

This example syncs from Entra ID to a PostgreSQL database destination. The (top level) source spec section is described in the Source Spec Reference.
kind: source
# Common source-plugin configuration
spec:
  name: entraid
  path: cloudquery/entraid
  registry: cloudquery
  version: "v1.8.1"
  tables: ["*"]
  destinations: ["postgresql"]

  # Entra ID specific configuration
  # Learn more about the configuration options at https://cql.ink/entraid_source
  spec:
    # Optional parameters
    # concurrency: 50000

Entra ID Spec #

  • concurrency (integer) (optional) (default: 50000)
    A best effort maximum number of Go routines to use. Lower this number to reduce memory usage.
  • scheduler (string) (optional) (default: dfs)
    The scheduler to use when determining the priority of resources to sync. Supported values are dfs (depth-first search), round-robin, shuffle and shuffle-queue.
    For more information about this, see performance tuning.


Incremental syncing #

The Entra ID plugin support incremental syncing via MS Graph API delta queries. The following tables support incremental syncing:
To enable incremental syncing for the above tables, set the backend_options configuration as shown in the example below. This is documented in the Managing Incremental Tables section.
kind: source
spec:
  name: "entraid"
  path: "cloudquery/entraid"
  version: "v1.8.1"
  destinations: ["postgresql"]
  tables: ["entraid_applications", "entraid_directoryroles", "entraid_groups", "entraid_serviceprincipals", "entraid_users"]
  backend_options:
    table_name: "cq_state_entraid"
    connection: "@@plugins.postgresql.connection"
---
kind: destination
spec:
  name: "postgresql"
  path: "cloudquery/postgresql"
  version: "v8.7.2"
  spec:
    connection_string: "postgresql://postgres:pass@localhost:5432/postgres?sslmode=disable"


Licenses #

The following tools / packages are used in this plugin:
NameLicense
github.com/Azure/azure-sdk-for-go/sdk/azcoreMIT
github.com/Azure/azure-sdk-for-go/sdk/azidentityMIT
github.com/Azure/azure-sdk-for-go/sdk/internalMIT
github.com/AzureAD/microsoft-authentication-library-for-go/appsMIT
github.com/adrg/xdgMIT
github.com/apache/arrow/go/v13Apache-2.0
github.com/apache/arrow/go/v17Apache-2.0
github.com/apapsch/go-jsonmerge/v2MIT
github.com/aws/aws-sdk-go-v2Apache-2.0
github.com/aws/aws-sdk-go-v2/configApache-2.0
github.com/aws/aws-sdk-go-v2/credentialsApache-2.0
github.com/aws/aws-sdk-go-v2/feature/ec2/imdsApache-2.0
github.com/aws/aws-sdk-go-v2/internal/configsourcesApache-2.0
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/iniApache-2.0
github.com/aws/aws-sdk-go-v2/internal/sync/singleflightBSD-3-Clause
github.com/aws/aws-sdk-go-v2/service/internal/accept-encodingApache-2.0
github.com/aws/aws-sdk-go-v2/service/internal/presigned-urlApache-2.0
github.com/aws/aws-sdk-go-v2/service/licensemanagerApache-2.0
github.com/aws/aws-sdk-go-v2/service/marketplacemeteringApache-2.0
github.com/aws/aws-sdk-go-v2/service/ssoApache-2.0
github.com/aws/aws-sdk-go-v2/service/ssooidcApache-2.0
github.com/aws/aws-sdk-go-v2/service/stsApache-2.0
github.com/aws/smithy-goApache-2.0
github.com/aws/smithy-go/internal/sync/singleflightBSD-3-Clause
github.com/bahlo/generic-list-goBSD-3-Clause
github.com/buger/jsonparserMIT
github.com/cenkalti/backoff/v4MIT
github.com/cjlapao/common-go/durationMIT
github.com/cloudquery/cloudquery-api-goMPL-2.0
github.com/cloudquery/plugin-pb-goMPL-2.0
github.com/cloudquery/plugin-sdk/v2/internal/globMIT
github.com/cloudquery/plugin-sdk/v2/schemaMIT
github.com/cloudquery/plugin-sdk/v2/typesMPL-2.0
github.com/cloudquery/plugin-sdk/v4MPL-2.0
github.com/cloudquery/plugin-sdk/v4/globMIT
github.com/cloudquery/plugin-sdk/v4/scalarMIT
github.com/davecgh/go-spew/spewISC
github.com/ghodss/yamlMIT
github.com/go-logr/logrApache-2.0
github.com/go-logr/stdrApache-2.0
github.com/goccy/go-jsonMIT
github.com/golang-jwt/jwt/v5MIT
github.com/google/flatbuffers/goApache-2.0
github.com/google/uuidBSD-3-Clause
github.com/gorilla/muxBSD-3-Clause
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptorsApache-2.0
github.com/grpc-ecosystem/grpc-gateway/v2BSD-3-Clause
github.com/hashicorp/go-cleanhttpMPL-2.0
github.com/hashicorp/go-retryablehttpMPL-2.0
github.com/invopop/jsonschemaMIT
github.com/klauspost/compressApache-2.0
github.com/klauspost/compress/internal/snaprefBSD-3-Clause
github.com/klauspost/compress/zstd/internal/xxhashMIT
github.com/kylelemons/godebugApache-2.0
github.com/mailru/easyjsonMIT
github.com/mattn/go-colorableMIT
github.com/mattn/go-isattyMIT
github.com/microsoft/kiota-abstractions-goMIT
github.com/microsoft/kiota-authentication-azure-goMIT
github.com/microsoft/kiota-http-goMIT
github.com/microsoft/kiota-serialization-form-goMIT
github.com/microsoft/kiota-serialization-json-goMIT
github.com/microsoft/kiota-serialization-multipart-goMIT
github.com/microsoft/kiota-serialization-text-goMIT
github.com/microsoftgraph/msgraph-beta-sdk-goMIT
github.com/microsoftgraph/msgraph-sdk-goMIT
github.com/microsoftgraph/msgraph-sdk-go-coreMIT
github.com/oapi-codegen/runtimeApache-2.0
github.com/pierrec/lz4/v4BSD-3-Clause
github.com/pkg/browserBSD-2-Clause
github.com/pmezard/go-difflib/difflibBSD-3-Clause
github.com/rs/zerologMIT
github.com/samber/loMIT
github.com/santhosh-tekuri/jsonschema/v6Apache-2.0
github.com/spf13/cobraApache-2.0
github.com/spf13/pflagBSD-3-Clause
github.com/std-uritemplate/std-uritemplate/goApache-2.0
github.com/stretchr/testifyMIT
github.com/thoas/go-funkMIT
github.com/wk8/go-ordered-map/v2Apache-2.0
github.com/zeebo/xxh3BSD-2-Clause
go.opentelemetry.io/otelApache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttpApache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttpApache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptraceApache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttpApache-2.0
go.opentelemetry.io/otel/logApache-2.0
go.opentelemetry.io/otel/metricApache-2.0
go.opentelemetry.io/otel/sdkApache-2.0
go.opentelemetry.io/otel/sdk/logApache-2.0
go.opentelemetry.io/otel/sdk/metricApache-2.0
go.opentelemetry.io/otel/traceApache-2.0
go.opentelemetry.io/proto/otlpApache-2.0
golang.org/x/crypto/pkcs12BSD-3-Clause
golang.org/x/expBSD-3-Clause
golang.org/x/netBSD-3-Clause
golang.org/x/syncBSD-3-Clause
golang.org/x/sysBSD-3-Clause
golang.org/x/textBSD-3-Clause
golang.org/x/xerrorsBSD-3-Clause
google.golang.org/genproto/googleapis/api/httpbodyApache-2.0
google.golang.org/genproto/googleapis/rpc/statusApache-2.0
google.golang.org/grpcApache-2.0
google.golang.org/protobufBSD-3-Clause
gopkg.in/yaml.v2Apache-2.0
gopkg.in/yaml.v3MIT


Join our mailing list

Subscribe to our newsletter to make sure you don't miss any updates.

Legal

© 2024 CloudQuery, Inc. All rights reserved.

We use tracking cookies to understand how you use the product and help us improve it. Please accept cookies to help us improve. You can always opt out later via the link in the footer.