Back to source list
Official
Premium
OpenAI
Sync from OpenAI to any destination
Publisher
cloudquery
Latest version
v1.3.4
Type
Source
Platforms
Date Published
Price per 1M rows
Starting from $17
monthly free quota
1M rows
Overview #
The CloudQuery OpenAI source plugin pulls data out of the OpenAI platform and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more).
Authentication #
In order to fetch information from OpenAI,
cloudquery needs to be authenticated. An organization admin key is required for authentication.
See Administration API documentation for more information.OpenAI Source Plugin Configuration Reference
Example #
This example syncs from OpenAI to a Postgres destination.
The (top level) source spec section is described in the Source Spec Reference.
kind: source
# Common source-plugin configuration
spec:
name: openai
path: cloudquery/openai
registry: cloudquery
version: "v1.3.4"
tables: ["*"]
destinations: ["postgresql"]
spec:
# required parameters
admin_api_key: "${OPENAI_ADMIN_API_KEY}"
# optional parameters
# concurrency: 10000OpenAI Spec #
This is the (nested) spec used by the OpenAI source plugin.
The API Key can be generated from the OpenAI Cloud platform, under the
My Account section.admin_api_key(string) (required)An organization admin key is required for authentication. See Administration API page for more information.concurrency(integer) (optional) (default:10000)A best effort maximum number of Go routines to use. Lower this number to reduce memory usage.scheduler(string) (optional) (default:dfs) The scheduler to use when determining the priority of resources to sync. Supported values aredfs(depth-first search),round-robin,shuffleandshuffle-queue.For more information about this, see performance tuning.
Licenses #
The following tools / packages are used in this plugin:
| Name | License |
|---|---|
| github.com/adrg/xdg | MIT |
| github.com/apache/arrow/go/v13 | Apache-2.0 |
| github.com/apache/arrow-go/v18 | Apache-2.0 |
| github.com/apapsch/go-jsonmerge/v2 | MIT |
| github.com/aws/aws-sdk-go-v2 | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/config | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/credentials | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/feature/ec2/imds | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/internal/configsources | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/internal/ini | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/internal/sync/singleflight | BSD-3-Clause |
| github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/service/internal/presigned-url | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/service/licensemanager | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/service/marketplacemetering | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/service/sso | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/service/ssooidc | Apache-2.0 |
| github.com/aws/aws-sdk-go-v2/service/sts | Apache-2.0 |
| github.com/aws/smithy-go | Apache-2.0 |
| github.com/aws/smithy-go/internal/sync/singleflight | BSD-3-Clause |
| github.com/bahlo/generic-list-go | BSD-3-Clause |
| github.com/buger/jsonparser | MIT |
| github.com/cenkalti/backoff/v4 | MIT |
| github.com/cloudquery/cloudquery-api-go | MPL-2.0 |
| github.com/cloudquery/plugin-pb-go | MPL-2.0 |
| github.com/cloudquery/plugin-sdk/v2/internal/glob | MIT |
| github.com/cloudquery/plugin-sdk/v2/schema | MIT |
| github.com/cloudquery/plugin-sdk/v2/types | MPL-2.0 |
| github.com/cloudquery/plugin-sdk/v4 | MPL-2.0 |
| github.com/cloudquery/plugin-sdk/v4/glob | MIT |
| github.com/cloudquery/plugin-sdk/v4/scalar | MIT |
| github.com/davecgh/go-spew/spew | ISC |
| github.com/ghodss/yaml | MIT |
| github.com/go-logr/logr | Apache-2.0 |
| github.com/go-logr/stdr | Apache-2.0 |
| github.com/goccy/go-json | MIT |
| github.com/google/flatbuffers/go | Apache-2.0 |
| github.com/google/uuid | BSD-3-Clause |
| github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors | Apache-2.0 |
| github.com/grpc-ecosystem/grpc-gateway/v2 | BSD-3-Clause |
| github.com/hashicorp/go-cleanhttp | MPL-2.0 |
| github.com/hashicorp/go-retryablehttp | MPL-2.0 |
| github.com/invopop/jsonschema | MIT |
| github.com/klauspost/compress | Apache-2.0 |
| github.com/klauspost/compress/internal/snapref | BSD-3-Clause |
| github.com/klauspost/compress/zstd/internal/xxhash | MIT |
| github.com/mailru/easyjson | MIT |
| github.com/mattn/go-colorable | MIT |
| github.com/mattn/go-isatty | MIT |
| github.com/oapi-codegen/oapi-codegen/v2/pkg/securityprovider | Apache-2.0 |
| github.com/oapi-codegen/runtime | Apache-2.0 |
| github.com/pierrec/lz4/v4 | BSD-3-Clause |
| github.com/pmezard/go-difflib/difflib | BSD-3-Clause |
| github.com/rs/zerolog | MIT |
| github.com/samber/lo | MIT |
| github.com/santhosh-tekuri/jsonschema/v6 | Apache-2.0 |
| github.com/spf13/cobra | Apache-2.0 |
| github.com/spf13/pflag | BSD-3-Clause |
| github.com/stretchr/testify | MIT |
| github.com/thoas/go-funk | MIT |
| github.com/wk8/go-ordered-map/v2 | Apache-2.0 |
| github.com/zeebo/xxh3 | BSD-2-Clause |
| go.opentelemetry.io/otel | Apache-2.0 |
| go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp | Apache-2.0 |
| go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp | Apache-2.0 |
| go.opentelemetry.io/otel/exporters/otlp/otlptrace | Apache-2.0 |
| go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp | Apache-2.0 |
| go.opentelemetry.io/otel/log | Apache-2.0 |
| go.opentelemetry.io/otel/metric | Apache-2.0 |
| go.opentelemetry.io/otel/sdk | Apache-2.0 |
| go.opentelemetry.io/otel/sdk/log | Apache-2.0 |
| go.opentelemetry.io/otel/sdk/metric | Apache-2.0 |
| go.opentelemetry.io/otel/trace | Apache-2.0 |
| go.opentelemetry.io/proto/otlp | Apache-2.0 |
| golang.org/x/exp | BSD-3-Clause |
| golang.org/x/net | BSD-3-Clause |
| golang.org/x/sync | BSD-3-Clause |
| golang.org/x/sys | BSD-3-Clause |
| golang.org/x/text | BSD-3-Clause |
| golang.org/x/xerrors | BSD-3-Clause |
| google.golang.org/genproto/googleapis/api/httpbody | Apache-2.0 |
| google.golang.org/genproto/googleapis/rpc/status | Apache-2.0 |
| google.golang.org/grpc | Apache-2.0 |
| google.golang.org/protobuf | BSD-3-Clause |
| gopkg.in/yaml.v2 | Apache-2.0 |
| gopkg.in/yaml.v3 | MIT |