Official

Premium

Firestore source integration documentation

The CloudQuery Firestore plugin reads information from your Firestore database and loads it into any supported CloudQuery destinationf

Publisher

cloudquery

Latest version

v4.5.2

Type

Source

Platforms

Date Published

Start Syncing

Overview Licenses

Overview #

Firestore Source Plugin

The CloudQuery Firestore plugin reads information from your Firestore database and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more).

Authentication #

The Firestore plugin authenticates using your Application Default Credentials. Available options are all the same options described here in detail:

Local Environment:

gcloud auth application-default login (recommended when running locally)

Google Cloud cloud-based development environment:

When you run on Cloud Shell or Cloud Code credentials are already available.

Google Cloud containerized environment:

When running on GKE use workload identity.

Google Cloud services that support attaching a service account:

Services such as Compute Engine, App Engine and functions supporting attaching a user-managed service account which will CloudQuery will be able to utilize.

On-premises or another cloud provider

The suggested way is to use Workload identity federation
If not available you can always use service account keys and export the location of the key via GOOGLE_APPLICATION_CREDENTIALS. Highly not recommended as long-lived keys are a security risk

Configuration #

kind: source
spec:
  # Source spec section
  name: firestore
  path: cloudquery/firestore
  registry: cloudquery
  version: "v4.5.2"
  tables: ["*"]
  destinations: ["postgresql"]
  # Learn more about the configuration options at https://cql.ink/firestore_source
  spec:
    # Firestore specific configuration goes here

For more information on downloading, installing and running the CloudQuery CLI, see the Quickstart guide.

Firestore Source Plugin Configuration Reference

Example #

This example connects a single Firestore project to a PostgreSQL destination. The (top level) source spec section is described in the Source Spec Reference.

kind: source
spec:
  # Source spec section
  name: "firestore"
  path: "cloudquery/firestore"
  version: "v4.5.2"
  destinations: ["postgresql"]
  tables: ["*"]
  spec:
    # Firestore Spec section described below
    project_id: "my-project"

Firestore Spec #

This is the (nested) spec used by Firestore Source Plugin

project_id ([]string) (optional) (default: auto detect. Will detect the project id from the credentials)
Specify project id to use when syncing Firestore resources. If not specified, the project id will be auto detected from the credentials. See more about auto detection here
service_account_json (string) (optional) (default: empty).
Firestore service account JSON content. Using service accounts is not recommended, but if it is used it is better to use environment or file variable substitution.
use_base64 (boolean) (optional) (default: false). If true, the service account JSON content will treated as base64 encoded, and will be decoded before being used.
rows_per_record (integer) (optional) (default: 500)
Amount of rows to be packed into a single Apache Arrow record to be sent over the wire during sync.
max_batch_size (integer) (optional) (default: 50000)
Maximum batch size for each request when reading Firestore data. You might need to be configure it depending on row data size and system resources.
order_by (string) (optional) (default: empty)
The field(s) to order the results by. If not specified, the results will be ordered by document id. This requires you to add an index on the field(s), which can only be done on the Firebase UI right now. See more here
order_direction (string) (optional) (default: empty)
The direction to order the results by. Only used when order_by is specified. Accepts asc or desc, and if not specified the results will be ordered in ascending order.

Licenses #

The following tools / packages are used in this plugin:

Name	License
cloud.google.com/go/auth	Apache-2.0
cloud.google.com/go/auth/oauth2adapt	Apache-2.0
cloud.google.com/go/compute/metadata	Apache-2.0
cloud.google.com/go/firestore	Apache-2.0
cloud.google.com/go/internal	Apache-2.0
cloud.google.com/go/longrunning/autogen/longrunningpb	Apache-2.0
github.com/adrg/xdg	MIT
github.com/apache/arrow/go/v13	Apache-2.0
github.com/apache/arrow-go/v18	Apache-2.0
github.com/apapsch/go-jsonmerge/v2	MIT
github.com/aws/aws-sdk-go-v2	Apache-2.0
github.com/aws/aws-sdk-go-v2/config	Apache-2.0
github.com/aws/aws-sdk-go-v2/credentials	Apache-2.0
github.com/aws/aws-sdk-go-v2/feature/ec2/imds	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/configsources	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/ini	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/sync/singleflight	BSD-3-Clause
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/licensemanager	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/marketplacemetering	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/sso	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/ssooidc	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/sts	Apache-2.0
github.com/aws/smithy-go	Apache-2.0
github.com/aws/smithy-go/internal/sync/singleflight	BSD-3-Clause
github.com/bahlo/generic-list-go	BSD-3-Clause
github.com/buger/jsonparser	MIT
github.com/cenkalti/backoff/v4	MIT
github.com/cloudquery/cloudquery-api-go	MPL-2.0
github.com/cloudquery/plugin-pb-go	MPL-2.0
github.com/cloudquery/plugin-sdk/v2/internal/glob	MIT
github.com/cloudquery/plugin-sdk/v2/schema	MIT
github.com/cloudquery/plugin-sdk/v2/types	MPL-2.0
github.com/cloudquery/plugin-sdk/v4	MPL-2.0
github.com/cloudquery/plugin-sdk/v4/glob	MIT
github.com/cloudquery/plugin-sdk/v4/scalar	MIT
github.com/davecgh/go-spew/spew	ISC
github.com/felixge/httpsnoop	MIT
github.com/ghodss/yaml	MIT
github.com/go-logr/logr	Apache-2.0
github.com/go-logr/stdr	Apache-2.0
github.com/goccy/go-json	MIT
github.com/golang/groupcache/lru	Apache-2.0
github.com/google/flatbuffers/go	Apache-2.0
github.com/google/s2a-go	Apache-2.0
github.com/google/uuid	BSD-3-Clause
github.com/googleapis/enterprise-certificate-proxy/client	Apache-2.0
github.com/googleapis/gax-go/v2	BSD-3-Clause
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors	Apache-2.0
github.com/grpc-ecosystem/grpc-gateway/v2	BSD-3-Clause
github.com/hashicorp/go-cleanhttp	MPL-2.0
github.com/hashicorp/go-retryablehttp	MPL-2.0
github.com/invopop/jsonschema	MIT
github.com/klauspost/compress	Apache-2.0
github.com/klauspost/compress/internal/snapref	BSD-3-Clause
github.com/klauspost/compress/zstd/internal/xxhash	MIT
github.com/mailru/easyjson	MIT
github.com/mattn/go-colorable	MIT
github.com/mattn/go-isatty	MIT
github.com/oapi-codegen/runtime	Apache-2.0
github.com/pierrec/lz4/v4	BSD-3-Clause
github.com/pmezard/go-difflib/difflib	BSD-3-Clause
github.com/rs/zerolog	MIT
github.com/santhosh-tekuri/jsonschema/v6	Apache-2.0
github.com/spf13/cobra	Apache-2.0
github.com/spf13/pflag	BSD-3-Clause
github.com/stretchr/testify	MIT
github.com/thoas/go-funk	MIT
github.com/wk8/go-ordered-map/v2	Apache-2.0
github.com/zeebo/xxh3	BSD-2-Clause
go.opencensus.io	Apache-2.0
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc	Apache-2.0
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp	Apache-2.0
go.opentelemetry.io/otel	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp	Apache-2.0
go.opentelemetry.io/otel/log	Apache-2.0
go.opentelemetry.io/otel/metric	Apache-2.0
go.opentelemetry.io/otel/sdk	Apache-2.0
go.opentelemetry.io/otel/sdk/log	Apache-2.0
go.opentelemetry.io/otel/sdk/metric	Apache-2.0
go.opentelemetry.io/otel/trace	Apache-2.0
go.opentelemetry.io/proto/otlp	Apache-2.0
golang.org/x/crypto	BSD-3-Clause
golang.org/x/exp	BSD-3-Clause
golang.org/x/net	BSD-3-Clause
golang.org/x/oauth2	BSD-3-Clause
golang.org/x/sync	BSD-3-Clause
golang.org/x/sys	BSD-3-Clause
golang.org/x/text	BSD-3-Clause
golang.org/x/time/rate	BSD-3-Clause
golang.org/x/xerrors	BSD-3-Clause
google.golang.org/api	BSD-3-Clause
google.golang.org/api/internal/third_party/uritemplates	BSD-3-Clause
google.golang.org/genproto/googleapis/api	Apache-2.0
google.golang.org/genproto/googleapis/rpc	Apache-2.0
google.golang.org/genproto/googleapis/type/latlng	Apache-2.0
google.golang.org/grpc	Apache-2.0
google.golang.org/protobuf	BSD-3-Clause
gopkg.in/yaml.v2	Apache-2.0
gopkg.in/yaml.v3	MIT

Loading plugin documentation

Overview #

Firestore Source Plugin

The CloudQuery Firestore plugin reads information from your Firestore database and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more).

Authentication #

The Firestore plugin authenticates using your Application Default Credentials. Available options are all the same options described here in detail:

Local Environment:

gcloud auth application-default login (recommended when running locally)

Google Cloud cloud-based development environment:

When you run on Cloud Shell or Cloud Code credentials are already available.

Google Cloud containerized environment:

When running on GKE use workload identity.

Google Cloud services that support attaching a service account:

Services such as Compute Engine, App Engine and functions supporting attaching a user-managed service account which will CloudQuery will be able to utilize.

On-premises or another cloud provider

The suggested way is to use Workload identity federation
If not available you can always use service account keys and export the location of the key via GOOGLE_APPLICATION_CREDENTIALS. Highly not recommended as long-lived keys are a security risk

Configuration #

kind: source
spec:
  # Source spec section
  name: firestore
  path: cloudquery/firestore
  registry: cloudquery
  version: "v4.5.2"
  tables: ["*"]
  destinations: ["postgresql"]
  # Learn more about the configuration options at https://cql.ink/firestore_source
  spec:
    # Firestore specific configuration goes here

For more information on downloading, installing and running the CloudQuery CLI, see the Quickstart guide.

Firestore Source Plugin Configuration Reference

Example #

This example connects a single Firestore project to a PostgreSQL destination. The (top level) source spec section is described in the Source Spec Reference.

kind: source
spec:
  # Source spec section
  name: "firestore"
  path: "cloudquery/firestore"
  version: "v4.5.2"
  destinations: ["postgresql"]
  tables: ["*"]
  spec:
    # Firestore Spec section described below
    project_id: "my-project"

Firestore Spec #

This is the (nested) spec used by Firestore Source Plugin

project_id ([]string) (optional) (default: auto detect. Will detect the project id from the credentials)
Specify project id to use when syncing Firestore resources. If not specified, the project id will be auto detected from the credentials. See more about auto detection here
service_account_json (string) (optional) (default: empty).
Firestore service account JSON content. Using service accounts is not recommended, but if it is used it is better to use environment or file variable substitution.
use_base64 (boolean) (optional) (default: false). If true, the service account JSON content will treated as base64 encoded, and will be decoded before being used.
rows_per_record (integer) (optional) (default: 500)
Amount of rows to be packed into a single Apache Arrow record to be sent over the wire during sync.
max_batch_size (integer) (optional) (default: 50000)
Maximum batch size for each request when reading Firestore data. You might need to be configure it depending on row data size and system resources.
order_by (string) (optional) (default: empty)
The field(s) to order the results by. If not specified, the results will be ordered by document id. This requires you to add an index on the field(s), which can only be done on the Firebase UI right now. See more here
order_direction (string) (optional) (default: empty)
The direction to order the results by. Only used when order_by is specified. Accepts asc or desc, and if not specified the results will be ordered in ascending order.

Name

License

cloud.google.com/go/auth

Apache-2.0

cloud.google.com/go/auth/oauth2adapt

Apache-2.0

cloud.google.com/go/compute/metadata

Apache-2.0

cloud.google.com/go/firestore

Apache-2.0

cloud.google.com/go/internal

Apache-2.0

cloud.google.com/go/longrunning/autogen/longrunningpb

Apache-2.0

github.com/adrg/xdg

MIT

github.com/apache/arrow/go/v13

Apache-2.0

github.com/apache/arrow-go/v18

Apache-2.0

github.com/apapsch/go-jsonmerge/v2

MIT

github.com/aws/aws-sdk-go-v2

Apache-2.0

github.com/aws/aws-sdk-go-v2/config

Apache-2.0

github.com/aws/aws-sdk-go-v2/credentials

Apache-2.0

github.com/aws/aws-sdk-go-v2/feature/ec2/imds

Apache-2.0

github.com/aws/aws-sdk-go-v2/internal/configsources

Apache-2.0

github.com/aws/aws-sdk-go-v2/internal/endpoints/v2

Apache-2.0

github.com/aws/aws-sdk-go-v2/internal/ini

Apache-2.0

github.com/aws/aws-sdk-go-v2/internal/sync/singleflight

BSD-3-Clause

github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding

Apache-2.0

github.com/aws/aws-sdk-go-v2/service/internal/presigned-url

Apache-2.0

github.com/aws/aws-sdk-go-v2/service/licensemanager

Apache-2.0

github.com/aws/aws-sdk-go-v2/service/marketplacemetering

Apache-2.0

github.com/aws/aws-sdk-go-v2/service/sso

Apache-2.0

github.com/aws/aws-sdk-go-v2/service/ssooidc

Apache-2.0

github.com/aws/aws-sdk-go-v2/service/sts

Apache-2.0

github.com/aws/smithy-go

Apache-2.0

github.com/aws/smithy-go/internal/sync/singleflight

BSD-3-Clause

github.com/bahlo/generic-list-go

BSD-3-Clause

github.com/buger/jsonparser

MIT

github.com/cenkalti/backoff/v4

MIT

github.com/cloudquery/cloudquery-api-go

MPL-2.0

github.com/cloudquery/plugin-pb-go

MPL-2.0

github.com/cloudquery/plugin-sdk/v2/internal/glob

MIT

github.com/cloudquery/plugin-sdk/v2/schema

MIT

github.com/cloudquery/plugin-sdk/v2/types

MPL-2.0

github.com/cloudquery/plugin-sdk/v4

MPL-2.0

github.com/cloudquery/plugin-sdk/v4/glob

MIT

github.com/cloudquery/plugin-sdk/v4/scalar

MIT

github.com/davecgh/go-spew/spew

ISC

github.com/felixge/httpsnoop

MIT

github.com/ghodss/yaml

MIT

github.com/go-logr/logr

Apache-2.0

github.com/go-logr/stdr

Apache-2.0

github.com/goccy/go-json

MIT

github.com/golang/groupcache/lru

Apache-2.0

github.com/google/flatbuffers/go

Apache-2.0

github.com/google/s2a-go

Apache-2.0

github.com/google/uuid

BSD-3-Clause

github.com/googleapis/enterprise-certificate-proxy/client

Apache-2.0

github.com/googleapis/gax-go/v2

BSD-3-Clause

github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors

Apache-2.0

github.com/grpc-ecosystem/grpc-gateway/v2

BSD-3-Clause

github.com/hashicorp/go-cleanhttp

MPL-2.0

github.com/hashicorp/go-retryablehttp

MPL-2.0

github.com/invopop/jsonschema

MIT

github.com/klauspost/compress

Apache-2.0

github.com/klauspost/compress/internal/snapref

BSD-3-Clause

github.com/klauspost/compress/zstd/internal/xxhash

MIT

github.com/mailru/easyjson

MIT

github.com/mattn/go-colorable

MIT

github.com/mattn/go-isatty

MIT

github.com/oapi-codegen/runtime

Apache-2.0

github.com/pierrec/lz4/v4

BSD-3-Clause

github.com/pmezard/go-difflib/difflib

BSD-3-Clause

github.com/rs/zerolog

MIT

github.com/santhosh-tekuri/jsonschema/v6

Apache-2.0

github.com/spf13/cobra

Apache-2.0

github.com/spf13/pflag

BSD-3-Clause

github.com/stretchr/testify

MIT

github.com/thoas/go-funk

MIT

github.com/wk8/go-ordered-map/v2

Apache-2.0

github.com/zeebo/xxh3

BSD-2-Clause

go.opencensus.io

Apache-2.0

go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc

Apache-2.0

go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

Apache-2.0

go.opentelemetry.io/otel

Apache-2.0

go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp

Apache-2.0

go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp

Apache-2.0

go.opentelemetry.io/otel/exporters/otlp/otlptrace

Apache-2.0

go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp

Apache-2.0

go.opentelemetry.io/otel/log

Apache-2.0

go.opentelemetry.io/otel/metric

Apache-2.0

go.opentelemetry.io/otel/sdk

Apache-2.0

go.opentelemetry.io/otel/sdk/log

Apache-2.0

go.opentelemetry.io/otel/sdk/metric

Apache-2.0

go.opentelemetry.io/otel/trace

Apache-2.0

go.opentelemetry.io/proto/otlp

Apache-2.0

golang.org/x/crypto

BSD-3-Clause

golang.org/x/exp

BSD-3-Clause

golang.org/x/net

BSD-3-Clause

golang.org/x/oauth2

BSD-3-Clause

golang.org/x/sync

BSD-3-Clause

golang.org/x/sys

BSD-3-Clause

golang.org/x/text

BSD-3-Clause

golang.org/x/time/rate

BSD-3-Clause

golang.org/x/xerrors

BSD-3-Clause

google.golang.org/api

BSD-3-Clause

google.golang.org/api/internal/third_party/uritemplates

BSD-3-Clause

google.golang.org/genproto/googleapis/api

Apache-2.0

google.golang.org/genproto/googleapis/rpc

Apache-2.0

google.golang.org/genproto/googleapis/type/latlng

Apache-2.0

google.golang.org/grpc

Apache-2.0

google.golang.org/protobuf

BSD-3-Clause

gopkg.in/yaml.v2

Apache-2.0

gopkg.in/yaml.v3

MIT

CloudOps

Firestore source integration documentation

Overview #

Firestore Source Plugin

Authentication #

Configuration #

Firestore Source Plugin Configuration Reference

Example #

Firestore Spec #

Overview #

Firestore Source Plugin

Authentication #

Configuration #

Firestore Source Plugin Configuration Reference

Example #

Firestore Spec #