Come and Meet CloudQuery at RSA Conference 2025 in San Francisco Learn more ❯

CloudQuery

Back to source list
k8s
Official
Premium

Kubernetes

The K8s Source plugin for CloudQuery extracts configuration from a variety of K8s APIs

Publisher

cloudquery

Latest version

v7.7.0

Type

Source

Platforms

Date Published

Overview #

The K8s Source plugin for CloudQuery extracts configuration from a variety of K8s APIs.

Libraries in Use #

Authentication #

Similar to how kubectl works, cloudquery depends on a Kubernetes configuration file to connect to a Kubernetes cluster and sync its information. By default, cloudquery uses the default Kubernetes configuration file (~/.kube/config). You can also specify a different configuration by setting the KUBECONFIG environment variable before running cloudquery sync.
export KUBECONFIG="<PATH_TO_YOUR_CONFIG_FILE>"

Kubernetes Service Account #

If cloudquery is running in a pod of the Kubernetes cluster, the Kubernetes Service Account can be used for direct authentication. To use the Kubernetes Service Account for direct authentication, a cluster role with all get and list privileges will need to be used.
The below command creates a new cluster role with get and list privileges.
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind:       ClusterRole
metadata:
  name: cloudquery-cluster-read
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - get
  - list
- nonResourceURLs:
  - '*'
  verbs:
  - get
  - list
EOF
Next, the cluster role and service account will need to be linked via a cluster role binding. The following creates a cluster role binding for the role we created above and the service account for the cloudquery pod.
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind:       ClusterRoleBinding
metadata:
  name: cloudquery-cluster-read-binding
subjects:
- kind: ServiceAccount
  name: cloudquery-sa
roleRef:
  kind: ClusterRole
  name: cloudquery-cluster-read
EOF


Configuration #

K8s Source Plugin Configuration Reference

The K8s source plugin connects to a Kubernetes cluster, fetches resources and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more).

Example #

This example connects a single k8s context to a Postgres destination. The (top level) source spec section is described in the Source Spec Reference.
kind: source
spec:
  # Source spec section
  name: k8s
  path: cloudquery/k8s
  registry: cloudquery
  version: "v7.7.0"
  tables: ["*"]
  destinations: ["postgresql"]
  # Learn more about the configuration options at https://cql.ink/k8s_source
  spec:
    contexts: ["context"]
This example connects to an EKS cluster using an IAM role. Provider spec is used to generate a kube config file for the provider. The (top level) source spec section is described in the Source Spec Reference.
kind: source
spec:
  name: k8s
  path: cloudquery/k8s
  version: v7.3.8
  tables:
    - "*"
  spec:
    providers:  
     - cluster: eks-cluster-name
       aws: 
         region: us-east-1
         role_arn: arn:aws:iam::111111111111:role/cross-account-readonly-role
This example connects to an AKS cluster using a service principal. For more details on creating the service principal see our Azure source plugin docs. The (top level) source spec section is described in the Source Spec Reference.
kind: source
spec:
  name: k8s
  path: cloudquery/k8s
  version: "v7.7.0"
  tables:
    - "*"
  spec:
    providers:
      - cluster: cluster-name
        azure:
          # service principal details:
          client_id: service-pricipal-app-id
          tenant_id: service-pricipal-tenant
          client_secret: service-pricipal-password
          # k8s cluster details:
          subscription_id: subscription-id
          resource_group_name: resource-group-name
This example connects to a GKE cluster using a Service Account JSON key.
kind: source
spec:
  name: k8s
  path: cloudquery/k8s
  version: "v7.7.0"
  tables:
    - "*"
  spec:
    providers:  
     - cluster: gcp-cluster-name
       gcp: 
         project_id: project-id
         location: us-central1
         service_account_key_json: |
          ${SERVICE_ACCOUNT_JSON_KEY}
         authenticate_gcloud: true

K8s Spec #

This is the (nested) spec used by K8s Source Plugin
  • contexts ([]string) (optional) (default: empty. Will use the default context from K8s's config file)
    Specify K8s contexts to connect to. Specifying * will connect to all contexts available in the K8s config file (usually ~/.kube/config).
  • concurrency (integer) (optional) (default: 5000)
    A best effort maximum number of Go routines to use. Lower this number to reduce memory usage.
  • scheduler (string) (optional) (default: dfs) The scheduler to use when determining the priority of resources to sync. Supported values are dfs (depth-first search), round-robin, shuffle and shuffle-queue.
    For more information about this, see performance tuning.
  • providers ([]Provider) (optional) (default: empty.)
    List of providers to connect to. This is used to generate a kube config file for the provider. Each entry in the list represents a context in the K8s config file and first entry is the default context. Cluster name is the name of the context in the K8s config file also.

Provider Spec #

  • cluster (string) (required)
    Name of the cluster.
  • aws (AWSSpec) (optional)
    AWS specific configuration for EKS cluster access
  • gcp (GCPSpec) (optional)
    GCP specific configuration for GKE cluster access
  • azure (AzureSpec) (optional)
    Azure specific configuration for AKS cluster access

AWSSpec #

  • region (string) (required)
    Region of the EKS cluster.
  • role_arn (string) (optional)
    IAM Role ARN to assume to access the EKS cluster.
  • external_id (string) (optional)
    ExternalID to use when assuming the IAM Role.

GCPSpec #

  • project_id (string) (required)
    Project of the GKE cluster.
  • location (string) (required)
    Location of the GKE cluster.
  • service_account_key_json (string) (required)
    Service account key, generated from GCP
  • authenticate_gcloud (bool) (optional)
    If set, the plugin will use the provided Service Account JSON to authenticate. Analogical to running gcloud auth activate-service-account --key-file=<JSON_FILE>

AzureSpec #

  • subscription_id (string) (required)
    Azure subscription ID where the cluster is defined.
  • resource_group_name (string) (required)
    Azure resource group name of the cluster.
  • client_id (string) (required)
    Azure client ID or the service principal app ID.
  • tenant_id (string) (required)
    Azure tenant ID.
  • client_secret (string) (optional)
    Azure secret or the service principal password.
  • oidc_token (string) (optional)
    Azure OIDC token. Use this option when not using the client_secret.
  • cloud_name (string) (optional)
    The name of the cloud environment to use. Possible values are AzureCloud, AzureChinaCloud, AzureGovernment.

Example #

This example connects a single k8s context to a Postgres destination. The (top level) source spec section is described in the Source Spec Reference.
kind: source
spec:
  # Source spec section
  name: k8s
  path: cloudquery/k8s
  registry: cloudquery
  version: "v7.7.0"
  tables: ["*"]
  destinations: ["postgresql"]
  # Learn more about the configuration options at https://cql.ink/k8s_source
  spec:
    contexts: ["context"]
This example connects to an EKS cluster using an IAM role. Provider spec is used to generate a kube config file for the provider. The (top level) source spec section is described in the Source Spec Reference.
kind: source
spec:
  name: k8s
  path: cloudquery/k8s
  version: v7.3.8
  tables:
    - "*"
  spec:
    providers:  
     - cluster: eks-cluster-name
       aws: 
         region: us-east-1
         role_arn: arn:aws:iam::111111111111:role/cross-account-readonly-role
This example connects to an AKS cluster using a service principal. For more details on creating the service principal see our Azure source plugin docs. The (top level) source spec section is described in the Source Spec Reference.
kind: source
spec:
  name: k8s
  path: cloudquery/k8s
  version: "v7.7.0"
  tables:
    - "*"
  spec:
    providers:
      - cluster: cluster-name
        azure:
          # service principal details:
          client_id: service-pricipal-app-id
          tenant_id: service-pricipal-tenant
          client_secret: service-pricipal-password
          # k8s cluster details:
          subscription_id: subscription-id
          resource_group_name: resource-group-name
This example connects to a GKE cluster using a Service Account JSON key.
kind: source
spec:
  name: k8s
  path: cloudquery/k8s
  version: "v7.7.0"
  tables:
    - "*"
  spec:
    providers:  
     - cluster: gcp-cluster-name
       gcp: 
         project_id: project-id
         location: us-central1
         service_account_key_json: |
          ${SERVICE_ACCOUNT_JSON_KEY}
         authenticate_gcloud: true


Licenses #

The following tools / packages are used in this plugin:
NameLicense
cloud.google.com/go/authApache-2.0
cloud.google.com/go/auth/oauth2adaptApache-2.0
cloud.google.com/go/compute/metadataApache-2.0
cloud.google.com/go/containerApache-2.0
github.com/Azure/azure-sdk-for-go/sdk/azcoreMIT
github.com/Azure/azure-sdk-for-go/sdk/azidentityMIT
github.com/Azure/azure-sdk-for-go/sdk/internalMIT
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v5MIT
github.com/AzureAD/microsoft-authentication-library-for-go/appsMIT
github.com/adrg/xdgMIT
github.com/apache/arrow-go/v18Apache-2.0
github.com/apache/arrow/go/v13Apache-2.0
github.com/apapsch/go-jsonmerge/v2MIT
github.com/aws/aws-sdk-go-v2Apache-2.0
github.com/aws/aws-sdk-go-v2/configApache-2.0
github.com/aws/aws-sdk-go-v2/credentialsApache-2.0
github.com/aws/aws-sdk-go-v2/feature/ec2/imdsApache-2.0
github.com/aws/aws-sdk-go-v2/internal/configsourcesApache-2.0
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/iniApache-2.0
github.com/aws/aws-sdk-go-v2/internal/sync/singleflightBSD-3-Clause
github.com/aws/aws-sdk-go-v2/service/eksApache-2.0
github.com/aws/aws-sdk-go-v2/service/internal/accept-encodingApache-2.0
github.com/aws/aws-sdk-go-v2/service/internal/presigned-urlApache-2.0
github.com/aws/aws-sdk-go-v2/service/licensemanagerApache-2.0
github.com/aws/aws-sdk-go-v2/service/marketplacemeteringApache-2.0
github.com/aws/aws-sdk-go-v2/service/ssoApache-2.0
github.com/aws/aws-sdk-go-v2/service/ssooidcApache-2.0
github.com/aws/aws-sdk-go-v2/service/stsApache-2.0
github.com/aws/smithy-goApache-2.0
github.com/aws/smithy-go/internal/sync/singleflightBSD-3-Clause
github.com/bahlo/generic-list-goBSD-3-Clause
github.com/buger/jsonparserMIT
github.com/cenkalti/backoff/v4MIT
github.com/cloudquery/cloudquery-api-goMPL-2.0
github.com/cloudquery/plugin-pb-goMPL-2.0
github.com/cloudquery/plugin-sdk/v2/internal/globMIT
github.com/cloudquery/plugin-sdk/v2/schemaMIT
github.com/cloudquery/plugin-sdk/v2/typesMPL-2.0
github.com/cloudquery/plugin-sdk/v4MPL-2.0
github.com/cloudquery/plugin-sdk/v4/globMIT
github.com/cloudquery/plugin-sdk/v4/scalarMIT
github.com/davecgh/go-spew/spewISC
github.com/emicklei/go-restful/v3MIT
github.com/felixge/httpsnoopMIT
github.com/ghodss/yamlMIT
github.com/go-logr/logrApache-2.0
github.com/go-logr/stdrApache-2.0
github.com/go-openapi/jsonpointerApache-2.0
github.com/go-openapi/jsonreferenceApache-2.0
github.com/go-openapi/swagApache-2.0
github.com/goccy/go-jsonMIT
github.com/gogo/protobufBSD-3-Clause
github.com/golang-jwt/jwt/v5MIT
github.com/golang/mock/gomockApache-2.0
github.com/golang/protobuf/protoBSD-3-Clause
github.com/google/flatbuffers/goApache-2.0
github.com/google/gnostic-modelsApache-2.0
github.com/google/gofuzzApache-2.0
github.com/google/s2a-goApache-2.0
github.com/google/uuidBSD-3-Clause
github.com/googleapis/enterprise-certificate-proxy/clientApache-2.0
github.com/googleapis/gax-go/v2BSD-3-Clause
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptorsApache-2.0
github.com/grpc-ecosystem/grpc-gateway/v2BSD-3-Clause
github.com/hashicorp/go-cleanhttpMPL-2.0
github.com/hashicorp/go-retryablehttpMPL-2.0
github.com/imdario/mergoBSD-3-Clause
github.com/invopop/jsonschemaMIT
github.com/josharian/internMIT
github.com/json-iterator/goMIT
github.com/klauspost/compressApache-2.0
github.com/klauspost/compress/internal/snaprefBSD-3-Clause
github.com/klauspost/compress/zstd/internal/xxhashMIT
github.com/kylelemons/godebugApache-2.0
github.com/mailru/easyjsonMIT
github.com/mattn/go-colorableMIT
github.com/mattn/go-isattyMIT
github.com/modern-go/concurrentApache-2.0
github.com/modern-go/reflect2Apache-2.0
github.com/munnerz/goautonegBSD-3-Clause
github.com/oapi-codegen/runtimeApache-2.0
github.com/pierrec/lz4/v4BSD-3-Clause
github.com/pkg/browserBSD-2-Clause
github.com/pmezard/go-difflib/difflibBSD-3-Clause
github.com/rs/zerologMIT
github.com/samber/loMIT
github.com/santhosh-tekuri/jsonschema/v6Apache-2.0
github.com/spf13/cobraApache-2.0
github.com/spf13/pflagBSD-3-Clause
github.com/stretchr/testifyMIT
github.com/thoas/go-funkMIT
github.com/wk8/go-ordered-map/v2Apache-2.0
github.com/zeebo/xxh3BSD-2-Clause
go.opentelemetry.io/auto/sdkApache-2.0
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpcApache-2.0
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttpApache-2.0
go.opentelemetry.io/otelApache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttpApache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttpApache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptraceApache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttpApache-2.0
go.opentelemetry.io/otel/logApache-2.0
go.opentelemetry.io/otel/metricApache-2.0
go.opentelemetry.io/otel/sdkApache-2.0
go.opentelemetry.io/otel/sdk/logApache-2.0
go.opentelemetry.io/otel/sdk/metricApache-2.0
go.opentelemetry.io/otel/traceApache-2.0
go.opentelemetry.io/proto/otlpApache-2.0
golang.org/x/cryptoBSD-3-Clause
golang.org/x/expBSD-3-Clause
golang.org/x/netBSD-3-Clause
golang.org/x/oauth2BSD-3-Clause
golang.org/x/syncBSD-3-Clause
golang.org/x/sysBSD-3-Clause
golang.org/x/termBSD-3-Clause
golang.org/x/textBSD-3-Clause
golang.org/x/time/rateBSD-3-Clause
golang.org/x/xerrorsBSD-3-Clause
google.golang.org/apiBSD-3-Clause
google.golang.org/api/internal/third_party/uritemplatesBSD-3-Clause
google.golang.org/genproto/googleapis/apiApache-2.0
google.golang.org/genproto/googleapis/rpcApache-2.0
google.golang.org/grpcApache-2.0
google.golang.org/protobufBSD-3-Clause
gopkg.in/inf.v0BSD-3-Clause
gopkg.in/yaml.v2Apache-2.0
gopkg.in/yaml.v3MIT
k8s.io/apiApache-2.0
k8s.io/apiextensions-apiserver/pkgApache-2.0
k8s.io/apimachinery/pkgApache-2.0
k8s.io/apimachinery/third_party/forked/golang/reflectBSD-3-Clause
k8s.io/client-goApache-2.0
k8s.io/klog/v2Apache-2.0
k8s.io/kube-openapi/pkgApache-2.0
k8s.io/kube-openapi/pkg/internal/third_party/go-json-experiment/jsonBSD-3-Clause
k8s.io/kube-openapi/pkg/validation/specApache-2.0
k8s.io/utilsApache-2.0
k8s.io/utils/internal/third_party/forked/golang/netBSD-3-Clause
sigs.k8s.io/jsonApache-2.0
sigs.k8s.io/structured-merge-diff/v4Apache-2.0
sigs.k8s.io/yamlApache-2.0
sigs.k8s.io/yaml/goyaml.v2Apache-2.0



© 2025 CloudQuery, Inc. All rights reserved.