Back to source list
Official
Premium
Kubernetes
The K8s Source plugin for CloudQuery extracts configuration from a variety of K8s APIs
Publisher
cloudquery
Latest version
v7.3.1
Type
Source
Platforms
Date Published
Price per 1M rows
Starting from $17
monthly free quota
1M rows
Set up process #
brew install cloudquery/tap/cloudquery
1. Download CLI and login
2. Create source and destination configs
Plugin configurationOverview #
The K8s Source plugin for CloudQuery extracts configuration from a variety of K8s APIs.
Libraries in Use #
Authentication #
Similar to how
kubectl
works, cloudquery
depends on a Kubernetes configuration file to connect to a Kubernetes cluster and sync
its information.
By default, cloudquery
uses the default Kubernetes configuration file (~/.kube/config
).
You can also specify a different configuration by setting the KUBECONFIG
environment variable before running cloudquery sync
.export KUBECONFIG="<PATH_TO_YOUR_CONFIG_FILE>"
Kubernetes Service Account #
If
cloudquery
is running in a pod of the Kubernetes cluster, the Kubernetes Service Account can be used for direct authentication. To use the Kubernetes Service Account for direct authentication, a cluster role with all get and list privileges will need to be used.The below command creates a new cluster role with
get
and list
privileges.kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cloudquery-cluster-read
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- get
- list
- nonResourceURLs:
- '*'
verbs:
- get
- list
EOF
Next, the cluster role and service account will need to be linked via a cluster role binding.
The following creates a cluster role binding for the role we created above and the service account for the
cloudquery
pod.kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cloudquery-cluster-read-binding
subjects:
- kind: ServiceAccount
name: cloudquery-sa
roleRef:
kind: ClusterRole
name: cloudquery-cluster-read
EOF
Configuration #
K8s Source Plugin Configuration Reference
The K8s source plugin connects to a Kubernetes cluster, fetches resources and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more).
Example #
This example connects a single k8s context to a Postgres destination. The (top level) source spec section is described in the Source Spec Reference.
kind: source
spec:
# Source spec section
name: k8s
path: cloudquery/k8s
registry: cloudquery
version: "v7.3.1"
tables: ["*"]
destinations: ["postgresql"]
# Learn more about the configuration options at https://cql.ink/k8s_source
spec:
contexts: ["context"]
K8s Spec #
This is the (nested) spec used by K8s Source Plugin
contexts
([]string
) (optional) (default: empty. Will use the default context from K8s's config file)Specify K8s contexts to connect to. Specifying*
will connect to all contexts available in the K8s config file (usually~/.kube/config
).concurrency
(integer
) (optional) (default:5000
)A best effort maximum number of Go routines to use. Lower this number to reduce memory usage.scheduler
(string
) (optional) (default:dfs
) The scheduler to use when determining the priority of resources to sync. Supported values aredfs
(depth-first search),round-robin
,shuffle
andshuffle-queue
.For more information about this, see performance tuning.
Licenses #
The following tools / packages are used in this plugin:
Name | License |
---|---|
github.com/adrg/xdg | MIT |
github.com/apache/arrow/go/v13 | Apache-2.0 |
github.com/apache/arrow/go/v17 | Apache-2.0 |
github.com/apapsch/go-jsonmerge/v2 | MIT |
github.com/aws/aws-sdk-go-v2 | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/config | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/credentials | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/feature/ec2/imds | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/internal/configsources | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/internal/ini | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/internal/sync/singleflight | BSD-3-Clause |
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/licensemanager | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/marketplacemetering | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/sso | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/ssooidc | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/sts | Apache-2.0 |
github.com/aws/smithy-go | Apache-2.0 |
github.com/aws/smithy-go/internal/sync/singleflight | BSD-3-Clause |
github.com/bahlo/generic-list-go | BSD-3-Clause |
github.com/buger/jsonparser | MIT |
github.com/cenkalti/backoff/v4 | MIT |
github.com/cloudquery/cloudquery-api-go | MPL-2.0 |
github.com/cloudquery/plugin-pb-go | MPL-2.0 |
github.com/cloudquery/plugin-sdk/v2/internal/glob | MIT |
github.com/cloudquery/plugin-sdk/v2/schema | MIT |
github.com/cloudquery/plugin-sdk/v2/types | MPL-2.0 |
github.com/cloudquery/plugin-sdk/v4 | MPL-2.0 |
github.com/cloudquery/plugin-sdk/v4/glob | MIT |
github.com/cloudquery/plugin-sdk/v4/scalar | MIT |
github.com/davecgh/go-spew/spew | ISC |
github.com/emicklei/go-restful/v3 | MIT |
github.com/ghodss/yaml | MIT |
github.com/go-logr/logr | Apache-2.0 |
github.com/go-logr/stdr | Apache-2.0 |
github.com/go-openapi/jsonpointer | Apache-2.0 |
github.com/go-openapi/jsonreference | Apache-2.0 |
github.com/go-openapi/swag | Apache-2.0 |
github.com/goccy/go-json | MIT |
github.com/gogo/protobuf | BSD-3-Clause |
github.com/golang/mock/gomock | Apache-2.0 |
github.com/golang/protobuf/proto | BSD-3-Clause |
github.com/google/flatbuffers/go | Apache-2.0 |
github.com/google/gnostic-models | Apache-2.0 |
github.com/google/gofuzz | Apache-2.0 |
github.com/google/uuid | BSD-3-Clause |
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors | Apache-2.0 |
github.com/grpc-ecosystem/grpc-gateway/v2 | BSD-3-Clause |
github.com/hashicorp/go-cleanhttp | MPL-2.0 |
github.com/hashicorp/go-retryablehttp | MPL-2.0 |
github.com/imdario/mergo | BSD-3-Clause |
github.com/invopop/jsonschema | MIT |
github.com/josharian/intern | MIT |
github.com/json-iterator/go | MIT |
github.com/klauspost/compress | Apache-2.0 |
github.com/klauspost/compress/internal/snapref | BSD-3-Clause |
github.com/klauspost/compress/zstd/internal/xxhash | MIT |
github.com/mailru/easyjson | MIT |
github.com/mattn/go-colorable | MIT |
github.com/mattn/go-isatty | MIT |
github.com/modern-go/concurrent | Apache-2.0 |
github.com/modern-go/reflect2 | Apache-2.0 |
github.com/munnerz/goautoneg | BSD-3-Clause |
github.com/oapi-codegen/runtime | Apache-2.0 |
github.com/pierrec/lz4/v4 | BSD-3-Clause |
github.com/pmezard/go-difflib/difflib | BSD-3-Clause |
github.com/rs/zerolog | MIT |
github.com/samber/lo | MIT |
github.com/santhosh-tekuri/jsonschema/v6 | Apache-2.0 |
github.com/spf13/cobra | Apache-2.0 |
github.com/spf13/pflag | BSD-3-Clause |
github.com/stretchr/testify | MIT |
github.com/thoas/go-funk | MIT |
github.com/wk8/go-ordered-map/v2 | Apache-2.0 |
github.com/zeebo/xxh3 | BSD-2-Clause |
go.opentelemetry.io/otel | Apache-2.0 |
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp | Apache-2.0 |
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp | Apache-2.0 |
go.opentelemetry.io/otel/exporters/otlp/otlptrace | Apache-2.0 |
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp | Apache-2.0 |
go.opentelemetry.io/otel/log | Apache-2.0 |
go.opentelemetry.io/otel/metric | Apache-2.0 |
go.opentelemetry.io/otel/sdk | Apache-2.0 |
go.opentelemetry.io/otel/sdk/log | Apache-2.0 |
go.opentelemetry.io/otel/sdk/metric | Apache-2.0 |
go.opentelemetry.io/otel/trace | Apache-2.0 |
go.opentelemetry.io/proto/otlp | Apache-2.0 |
golang.org/x/exp | BSD-3-Clause |
golang.org/x/net | BSD-3-Clause |
golang.org/x/oauth2 | BSD-3-Clause |
golang.org/x/sync | BSD-3-Clause |
golang.org/x/sys | BSD-3-Clause |
golang.org/x/term | BSD-3-Clause |
golang.org/x/text | BSD-3-Clause |
golang.org/x/time/rate | BSD-3-Clause |
golang.org/x/xerrors | BSD-3-Clause |
google.golang.org/genproto/googleapis/api/httpbody | Apache-2.0 |
google.golang.org/genproto/googleapis/rpc/status | Apache-2.0 |
google.golang.org/grpc | Apache-2.0 |
google.golang.org/protobuf | BSD-3-Clause |
gopkg.in/inf.v0 | BSD-3-Clause |
gopkg.in/yaml.v2 | Apache-2.0 |
gopkg.in/yaml.v3 | MIT |
k8s.io/api | Apache-2.0 |
k8s.io/apiextensions-apiserver/pkg | Apache-2.0 |
k8s.io/apimachinery/pkg | Apache-2.0 |
k8s.io/apimachinery/third_party/forked/golang/reflect | BSD-3-Clause |
k8s.io/client-go | Apache-2.0 |
k8s.io/klog/v2 | Apache-2.0 |
k8s.io/kube-openapi/pkg | Apache-2.0 |
k8s.io/kube-openapi/pkg/internal/third_party/go-json-experiment/json | BSD-3-Clause |
k8s.io/kube-openapi/pkg/validation/spec | Apache-2.0 |
k8s.io/utils | Apache-2.0 |
k8s.io/utils/internal/third_party/forked/golang/net | BSD-3-Clause |
sigs.k8s.io/json | Apache-2.0 |
sigs.k8s.io/structured-merge-diff/v4 | Apache-2.0 |
sigs.k8s.io/yaml | Apache-2.0 |
sigs.k8s.io/yaml/goyaml.v2 | Apache-2.0 |