Official

Premium

Kubernetes

The K8s Source plugin for CloudQuery extracts configuration from a variety of K8s APIs

Publisher

cloudquery

Latest version

v7.1.1

Type

Source

Platforms

Date Published

October 8, 2024

Price per 1M rows

Starting from $15

monthly free quota

1M rows

Documentation Tables Changelog Destinations

Overview Configuration

Set up process #

brew install cloudquery/tap/cloudquery

1. Download CLI and login

See installation options

2. Create source and destination configs

Plugin configuration

cloudquery sync k8s.yml postgresql.yml

3. Run the sync

CloudQuery sync

Overview #

The K8s Source plugin for CloudQuery extracts configuration from a variety of K8s APIs.

Libraries in Use #

https://pkg.go.dev/k8s.io/api

Authentication #

Similar to how kubectl works, cloudquery depends on a Kubernetes configuration file to connect to a Kubernetes cluster and sync its information. By default, cloudquery uses the default Kubernetes configuration file (~/.kube/config). You can also specify a different configuration by setting the KUBECONFIG environment variable before running cloudquery sync.

export KUBECONFIG="<PATH_TO_YOUR_CONFIG_FILE>"

Kubernetes Service Account #

If cloudquery is running in a pod of the Kubernetes cluster, the Kubernetes Service Account can be used for direct authentication. To use the Kubernetes Service Account for direct authentication, a cluster role with all get and list privileges will need to be used.

The below command creates a new cluster role with get and list privileges.

kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind:       ClusterRole
metadata:
  name: cloudquery-cluster-read
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - get
  - list
- nonResourceURLs:
  - '*'
  verbs:
  - get
  - list
EOF

Next, the cluster role and service account will need to be linked via a cluster role binding. The following creates a cluster role binding for the role we created above and the service account for the cloudquery pod.

kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind:       ClusterRoleBinding
metadata:
  name: cloudquery-cluster-read-binding
subjects:
- kind: ServiceAccount
  name: cloudquery-sa
roleRef:
  kind: ClusterRole
  name: cloudquery-cluster-read
EOF

Configuration #

K8s Source Plugin Configuration Reference

The K8s source plugin connects to a Kubernetes cluster, fetches resources and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more).

Example #

This example connects a single k8s context to a Postgres destination. The (top level) source spec section is described in the Source Spec Reference.

kind: source
spec:
  # Source spec section
  name: k8s
  path: cloudquery/k8s
  registry: cloudquery
  version: "v7.1.1"
  tables: ["*"]
  destinations: ["postgresql"]
  # Learn more about the configuration options at https://cql.ink/k8s_source
  spec:
    contexts: ["context"]

K8s Spec #

This is the (nested) spec used by K8s Source Plugin

contexts ([]string) (optional) (default: empty. Will use the default context from K8s's config file)
Specify K8s contexts to connect to. Specifying * will connect to all contexts available in the K8s config file (usually ~/.kube/config).
concurrency (integer) (optional) (default: 50000):
The best effort maximum number of Go routines to use. Lower this number to reduce memory usage.