Back to source list![buildkite](/_next/image?url=https%3A%2F%2Fstorage.googleapis.com%2Fcq-cloud-images%2F79884f35-4b30-4d3a-a59d-ca360e11314f&w=96&q=75)
Official
Premium
Buildkite
This plugin is in preview.
Sync from Buildkite to any destination
Publisher
cloudquery
Latest version
v1.0.0
Type
Source
Platforms
Date Published
Overview #
CloudQuery Buildkite plugin reads information from Buildkite and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more).
Authentication #
The Buildkite plugin requires an API access token to authenticate.
API Access tokens are scoped to a single organization, in order to sync data from multiple organizations at once, multiple tokens can be provided in the
access_tokens
configuration.
You can generate the API Access Token by following the Buildkite documentation.
This integration syncs data using Buildkite's REST API, GraphQL API access should not be required.Once you have an access token, you can export it as an environment variable:
export BUILDKITE_ACCESS_TOKEN=<your_api_token>
(MacOS / Linux) or
set BUILDKITE_ACCESS_TOKEN=<your_api_token>
(Windows)
Example Config #
This example syncs from Buildkite to a Postgres destination. The (top level) source spec section is described in the Source Spec Reference.
kind: source
# Common source-plugin configuration
spec:
name: buildkite
path: cloudquery/buildkite
version: "v1.0.0"
tables: ["*"]
destinations: ["postgresql"]
# buildkite specific configuration
spec:
access_tokens: ["${BUILDKITE_ACCESS_TOKEN}"]
# optional parameters
# base_url: "https://api.buildkite.com/"
# concurrency: 100
# scheduler: "dfs"
# requests_per_minute: 160
Configuration #
This is the (nested) spec used by the Buildkite source plugin.
access_tokens
([]string
) (required)Buildkite API Acces token. Can be obtained by generating an access token.base_url
(string
) (optional)The base URL to use for the Buildkite API. Defaults to https://api.buildkite.com/concurrency
(integer
) (optional) (default:100
)A best effort maximum number of Go routines to use. Lower this number to reduce memory usage.scheduler
(string
, optional, default:dfs
)The scheduler to use when determining the priority of resources to sync. Supported values aredfs
(depth-first search),round-robin
,shuffle
andshuffle-queue
.For more information about this, see performance tuning.requests_per_minute
(integer
) (optional)The maximum number of requests per minute that can be made to the Buildkite API. BuildKite API rate limit default is 200 requests per minute for each organization. CloudQuery by default uses 80% of the rate limit, to allow for other requests to be made to the API.table_options
(map
) (default:null
)A set of options to override the defaults for certain tables. The full list of supported options are documented under the Table Options section of each tableThe format of thetable_options object
is as follows:table_options: <table_name>: <input_object>
table_options: buildkite_agents: include_access_token: true # default: false buildkite_pipelines: include_env_vars: true # default: false buildkite_builds: ListBuilds: - include_env_vars: true # default: false created_from: "2 days ago" created_to: "1 day ago" - include_env_vars: false branch: ["dev", "feature1"]
Time in table options
Our
Time
type allows for defining timestamps in both absolute and relative formats.Absolute timestamps must be RFC3339 formatted. Example:
2024-01-01T12:00:00+00:00
.Relative timestamps can take this format:
now
x seconds [ago|from now]
x minutes [ago|from now]
x hours [ago|from now]
x days [ago|from now]
Here are a few valid values for the
until
field:until: now
until: 2 days ago
until: 10 months 3 days 4h20m from now
until: 2024-01-01T12:00:00+00:00
Licenses #
The following tools / packages are used in this plugin:
Name | License |
---|---|
github.com/adrg/xdg | MIT |
github.com/apache/arrow-go/v18 | Apache-2.0 |
github.com/apache/arrow/go/v13 | Apache-2.0 |
github.com/apapsch/go-jsonmerge/v2 | MIT |
github.com/aws/aws-sdk-go-v2 | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/config | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/credentials | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/feature/ec2/imds | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/internal/configsources | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/internal/ini | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/internal/sync/singleflight | BSD-3-Clause |
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/licensemanager | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/marketplacemetering | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/sso | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/ssooidc | Apache-2.0 |
github.com/aws/aws-sdk-go-v2/service/sts | Apache-2.0 |
github.com/aws/smithy-go | Apache-2.0 |
github.com/aws/smithy-go/internal/sync/singleflight | BSD-3-Clause |
github.com/bahlo/generic-list-go | BSD-3-Clause |
github.com/buger/jsonparser | MIT |
github.com/buildkite/go-buildkite/v4 | BSD-2-Clause |
github.com/cenkalti/backoff | MIT |
github.com/cenkalti/backoff/v4 | MIT |
github.com/cloudquery/cloudquery-api-go | MPL-2.0 |
github.com/cloudquery/codegen/jsonschema/docs | MPL-2.0 |
github.com/cloudquery/plugin-pb-go | MPL-2.0 |
github.com/cloudquery/plugin-sdk/v2/internal/glob | MIT |
github.com/cloudquery/plugin-sdk/v2/schema | MIT |
github.com/cloudquery/plugin-sdk/v2/types | MPL-2.0 |
github.com/cloudquery/plugin-sdk/v4 | MPL-2.0 |
github.com/cloudquery/plugin-sdk/v4/glob | MIT |
github.com/cloudquery/plugin-sdk/v4/scalar | MIT |
github.com/davecgh/go-spew/spew | ISC |
github.com/ghodss/yaml | MIT |
github.com/go-logr/logr | Apache-2.0 |
github.com/go-logr/stdr | Apache-2.0 |
github.com/goccy/go-json | MIT |
github.com/golang/mock/gomock | Apache-2.0 |
github.com/google/flatbuffers/go | Apache-2.0 |
github.com/google/go-querystring/query | BSD-3-Clause |
github.com/google/uuid | BSD-3-Clause |
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors | Apache-2.0 |
github.com/grpc-ecosystem/grpc-gateway/v2 | BSD-3-Clause |
github.com/hashicorp/go-cleanhttp | MPL-2.0 |
github.com/hashicorp/go-retryablehttp | MPL-2.0 |
github.com/invopop/jsonschema | MIT |
github.com/klauspost/compress | Apache-2.0 |
github.com/klauspost/compress/internal/snapref | BSD-3-Clause |
github.com/klauspost/compress/zstd/internal/xxhash | MIT |
github.com/mailru/easyjson | MIT |
github.com/mattn/go-colorable | MIT |
github.com/mattn/go-isatty | MIT |
github.com/mitchellh/hashstructure/v2 | MIT |
github.com/oapi-codegen/runtime | Apache-2.0 |
github.com/pierrec/lz4/v4 | BSD-3-Clause |
github.com/pmezard/go-difflib/difflib | BSD-3-Clause |
github.com/rs/zerolog | MIT |
github.com/samber/lo | MIT |
github.com/santhosh-tekuri/jsonschema/v6 | Apache-2.0 |
github.com/spf13/cobra | Apache-2.0 |
github.com/spf13/pflag | BSD-3-Clause |
github.com/stretchr/testify | MIT |
github.com/thoas/go-funk | MIT |
github.com/wk8/go-ordered-map/v2 | Apache-2.0 |
github.com/zeebo/xxh3 | BSD-2-Clause |
go.opentelemetry.io/auto/sdk | Apache-2.0 |
go.opentelemetry.io/otel | Apache-2.0 |
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp | Apache-2.0 |
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp | Apache-2.0 |
go.opentelemetry.io/otel/exporters/otlp/otlptrace | Apache-2.0 |
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp | Apache-2.0 |
go.opentelemetry.io/otel/log | Apache-2.0 |
go.opentelemetry.io/otel/metric | Apache-2.0 |
go.opentelemetry.io/otel/sdk | Apache-2.0 |
go.opentelemetry.io/otel/sdk/log | Apache-2.0 |
go.opentelemetry.io/otel/sdk/metric | Apache-2.0 |
go.opentelemetry.io/otel/trace | Apache-2.0 |
go.opentelemetry.io/proto/otlp | Apache-2.0 |
golang.org/x/exp | BSD-3-Clause |
golang.org/x/net | BSD-3-Clause |
golang.org/x/sync | BSD-3-Clause |
golang.org/x/sys | BSD-3-Clause |
golang.org/x/text | BSD-3-Clause |
golang.org/x/time/rate | BSD-3-Clause |
golang.org/x/xerrors | BSD-3-Clause |
google.golang.org/genproto/googleapis/api/httpbody | Apache-2.0 |
google.golang.org/genproto/googleapis/rpc/status | Apache-2.0 |
google.golang.org/grpc | Apache-2.0 |
google.golang.org/protobuf | BSD-3-Clause |
gopkg.in/yaml.v2 | Apache-2.0 |
gopkg.in/yaml.v3 | MIT |