The CloudQuery Azure source plugin extracts information from many of the supported services by Microsoft Azure and loads it into any supported CloudQuery destination
Publisher
cloudquery
Latest version
v17.12.0
Type
Source
Platforms
Date Published
Loading plugin documentation
We use tracking cookies to understand how you use the product and help us improve it. Please accept cookies to help us improve. You can always opt out later via the link in the footer.
DefaultAzureCredential to authenticate.DefaultAzureCredential will attempt to authenticate via different mechanisms in order, stopping when one succeeds. The order is described in detail in the Azure SDK documentation.az login. The latter is highly discouraged for production use as it requires spawning a new Azure CLI process each time an authentication token is needed and causes memory and performance issues.az).az login
az ad sp create-for-rbac contains credentials that you must protect - Make sure to handle with appropriate care.
This example uses bash - The commands for CMD and PowerShell are similar.az provider register --namespace 'Microsoft.Security'
# Create a service-principal for the plugin and grant it access to your Azure resources in a subscription
az ad sp create-for-rbac --name cloudquery-sp --scopes /subscriptions/{subscription-id} --role Reader
You can choose any name you'd like for your service-principal,cloudquery-spis an example. If the service principal doesn't exist it will create a new one. If the principal already exists, you will seeInsufficient privileges to complete the operation.
az ad sp create-for-rbac should look like this:{
"appId": "YOUR AZURE_CLIENT_ID",
"displayName": "cloudquery-sp",
"password": "YOUR AZURE_CLIENT_SECRET",
"tenant": "YOUR AZURE_TENANT_ID"
}
az provider register --namespace 'Microsoft.Security'
# Create a service-principal for the plugin and grant it access to all resources under the specified Management Group
az ad sp create-for-rbac --name cloudquery-sp-root-1 --scopes /providers/Microsoft.Management/managementGroups/{management-group-name} --role Reader
--scopes parameter. For example, to grant access to all subscriptions that a user has access to at that specific time, you can use the following command:az provider register --namespace 'Microsoft.Security'
# Create a service-principal for the plugin and grant it access to all subscriptions that the user has access to at that specific time
az ad sp create-for-rbac --name cloudquery-sp --scopes $(az account subscription list --query "[].id" -o tsv --only-show-errors | xargs) --role Reader
az ad sp create-for-rbac.
The example shows how to export environment variables for Linux - exporting for CMD and PowerShell is similar.AZURE_TENANT_ID is tenant in the JSON.AZURE_CLIENT_ID is appId in the JSON.AZURE_CLIENT_SECRET is password in the JSON.export AZURE_TENANT_ID=<YOUR AZURE_TENANT_ID>
export AZURE_CLIENT_ID=<YOUR AZURE_CLIENT_ID>
export AZURE_CLIENT_SECRET=<YOUR AZURE_CLIENT_SECRET>
az login #az). Then, login with the Azure CLI:az login
SELECT * FROM azure_mysql_servers;
SELECT * from azure_storage_accounts where enable_https_traffic_only = false;
SELECT * from azure_keyvault_vault_keys where attributes_expires >= extract(epoch from now()) * 1000;
SELECT
distinct(vm.name),
vcpus.capability_value AS "vCPUs",
memory.capability_value AS "Memory"
FROM
azure_compute_skus vm
CROSS JOIN LATERAL (
SELECT (caps ->> 'value') AS capability_value
FROM jsonb_array_elements(vm.capabilities) caps
WHERE (caps ->> 'name') = 'vCPUs'
) vcpus
CROSS JOIN LATERAL (
SELECT (caps ->> 'value') AS capability_value
FROM jsonb_array_elements(vm.capabilities) caps
WHERE (caps ->> 'name') = 'MemoryGB'
) memory
WHERE
vm.resource_type = 'virtualMachines' order by name;
+---------------------------+-------+--------+
| name | vCPUs | Memory |
|---------------------------+-------+--------|
| Basic_A0 | 1 | 0.75 |
| Basic_A1 | 1 | 1.75 |
| Basic_A2 | 2 | 3.5 |
| Basic_A3 | 4 | 7 |
| Basic_A4 | 8 | 14 |
| Standard_A0 | 1 | 0.75 |
| Standard_A1 | 1 | 1.75 |
| Standard_A1_v2 | 1 | 2 |
... (truncated)
kind: source
spec:
# Source spec section
name: "azure"
path: "cloudquery/azure"
registry: "cloudquery"
version: "v17.12.0"
destinations: ["postgresql"]
tables: ["azure_compute_virtual_machines"]
# Learn more about the configuration options at https://cql.ink/azure_source
spec:
# Optional parameters
# subscriptions: []
# cloud_name: ""
# concurrency: 50000
# discovery_concurrency: 400
# skip_subscriptions: []
# normalize_ids: false
# oidc_token: ""
# retry_options:
# max_retries: 3
# try_timeout_seconds: 0
# retry_delay_seconds: 4
# max_retry_delay_seconds: 60
subscriptions ([]string) (default: empty. Will use all visible subscriptions)cloud_name (string) (default: empty)AzurePublic, AzureGovernment, AzureChina.concurrency (integer) (default: 50000):discovery_concurrency (integer) (default: 400)scheduler (string) (default: dfs):dfs (depth-first search), round-robin, shuffle and shuffle-queue.skip_subscriptions ([]string) (default: empty)normalize_ids (bool) (default: false)id column values to be lowercase. This is useful to avoid case sensitivity and uniqueness issues around the id primary keysoidc_token (string) (default: empty)AZURE_CLIENT_SECRET.
This is useful for Azure AD workload identity federation.
When using this option, the AZURE_CLIENT_ID and AZURE_TENANT_ID environment variables must be set.retry_options (RetryOptions) (default: empty)retry_options #max_retries (integer) (default: 10)3 to 10.try_timeout_seconds (integer) (default: 0)retry_delay_seconds (integer) (default: 4)max_retry_delay_seconds (integer) (default: 60)status_codes ([]integer) (default: null)null uses the default status codes.
An empty value disables retries for HTTP status codes.* is necessary for tables, below is a reference configuration of skip tables, where certain tables are skipped.kind: source
spec:
# Source spec section
name: aws
path: cloudquery/azure
registry: cloudquery
version: "v17.12.0"
tables: ["*"]
skip_tables:
- "azure_*_definitions"
- "azure_authorization_role_assignments"
- "azure_compute_skus"
- "azure_consumption_subscription_price_sheets"
- "azure_logic_workflow_run_action_request_histories"
- "azure_quota_quotas"
- "azure_quota_usages"
- "azure_security_solutions"
# free tables
- "azure_appservice_top_level_domains"
- "azure_authorization_provider_operations_metadata"
- "azure_cognitiveservices_resource_skus"
- "azure_cosmos_locations"
- "azure_network_bgp_service_communities"
- "azure_peering_service_locations"
- "azure_security_assessments_metadata"
destinations: ["postgresql"]
spec:
# Azure Spec section described below
table_options object is as follows:table_options:
<table_name>:
<service_name>:
- <input_object>
<input_object> objects should be provided.
The plugin will iterate through these to make multiple API calls. table_options:
azure_compute_virtual_machines:
virtual_machines_options:
- status_only: "true"
- status_only: "false"
expand: "instanceView"
StatusOnly is represented as status_only.table_options:
azure_compute_virtual_machines:
virtual_machines_options:
- <VirtualMachines.ListAllOptions>
Table Options section of each table in the Azure plugin tables documentation.