Back to source list
firestore
Official
Premium

Firestore

The CloudQuery Firestore plugin reads information from your Firestore database and loads it into any supported CloudQuery destinationf

Publisher

cloudquery

Latest version

v4.4.0

Type

Source

Platforms
Date Published

Price per 1M rows

Starting from $17

monthly free quota

1M rows

Set up process #


brew install cloudquery/tap/cloudquery

1. Download CLI and login

See installation options

2. Create source and destination configs

Plugin configuration

cloudquery sync firestore.yml postgresql.yml

3. Run the sync

CloudQuery sync

Overview #

Firestore Source Plugin

The CloudQuery Firestore plugin reads information from your Firestore database and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more).

Authentication #

The Firestore plugin authenticates using your Application Default Credentials. Available options are all the same options described here in detail:
Local Environment:
  • gcloud auth application-default login (recommended when running locally)
Google Cloud cloud-based development environment:
  • When you run on Cloud Shell or Cloud Code credentials are already available.
Google Cloud containerized environment:
  • Services such as Compute Engine, App Engine and functions supporting attaching a user-managed service account which will CloudQuery will be able to utilize.
On-premises or another cloud provider
  • The suggested way is to use Workload identity federation
  • If not available you can always use service account keys and export the location of the key via GOOGLE_APPLICATION_CREDENTIALS. Highly not recommended as long-lived keys are a security risk

Configuration #

kind: source
spec:
  # Source spec section
  name: firestore
  path: cloudquery/firestore
  registry: cloudquery
  version: "v4.4.0"
  tables: ["*"]
  destinations: ["postgresql"]
  # Learn more about the configuration options at https://cql.ink/firestore_source
  spec:
    # Firestore specific configuration goes here
For more information on downloading, installing and running the CloudQuery CLI, see the Quickstart guide.

Firestore Source Plugin Configuration Reference

Example #

This example connects a single Firestore project to a PostgreSQL destination. The (top level) source spec section is described in the Source Spec Reference.
kind: source
spec:
  # Source spec section
  name: "firestore"
  path: "cloudquery/firestore"
  version: "v4.4.0"
  destinations: ["postgresql"]
  tables: ["*"]
  spec:
    # Firestore Spec section described below
    project_id: "my-project"

Firestore Spec #

This is the (nested) spec used by Firestore Source Plugin
  • project_id ([]string) (optional) (default: auto detect. Will detect the project id from the credentials)
    Specify project id to use when syncing Firestore resources. If not specified, the project id will be auto detected from the credentials. See more about auto detection here
  • service_account_json (string) (optional) (default: empty).
    Firestore service account JSON content. Using service accounts is not recommended, but if it is used it is better to use environment or file variable substitution.
  • use_base64 (boolean) (optional) (default: false). If true, the service account JSON content will treated as base64 encoded, and will be decoded before being used.
  • rows_per_record (integer) (optional) (default: 500)
    Amount of rows to be packed into a single Apache Arrow record to be sent over the wire during sync.
  • max_batch_size (integer) (optional) (default: 50000)
    Maximum batch size for each request when reading Firestore data. You might need to be configure it depending on row data size and system resources.
  • order_by (string) (optional) (default: empty)
    The field(s) to order the results by. If not specified, the results will be ordered by document id. This requires you to add an index on the field(s), which can only be done on the Firebase UI right now. See more here
  • order_direction (string) (optional) (default: empty)
    The direction to order the results by. Only used when order_by is specified. Accepts asc or desc, and if not specified the results will be ordered in ascending order.


Licenses #

The following tools / packages are used in this plugin:
NameLicense
cloud.google.com/go/authApache-2.0
cloud.google.com/go/auth/oauth2adaptApache-2.0
cloud.google.com/go/compute/metadataApache-2.0
cloud.google.com/go/firestoreApache-2.0
cloud.google.com/go/internalApache-2.0
cloud.google.com/go/longrunning/autogen/longrunningpbApache-2.0
github.com/adrg/xdgMIT
github.com/apache/arrow/go/v13Apache-2.0
github.com/apache/arrow/go/v17Apache-2.0
github.com/apapsch/go-jsonmerge/v2MIT
github.com/aws/aws-sdk-go-v2Apache-2.0
github.com/aws/aws-sdk-go-v2/configApache-2.0
github.com/aws/aws-sdk-go-v2/credentialsApache-2.0
github.com/aws/aws-sdk-go-v2/feature/ec2/imdsApache-2.0
github.com/aws/aws-sdk-go-v2/internal/configsourcesApache-2.0
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/iniApache-2.0
github.com/aws/aws-sdk-go-v2/internal/sync/singleflightBSD-3-Clause
github.com/aws/aws-sdk-go-v2/service/internal/accept-encodingApache-2.0
github.com/aws/aws-sdk-go-v2/service/internal/presigned-urlApache-2.0
github.com/aws/aws-sdk-go-v2/service/licensemanagerApache-2.0
github.com/aws/aws-sdk-go-v2/service/marketplacemeteringApache-2.0
github.com/aws/aws-sdk-go-v2/service/ssoApache-2.0
github.com/aws/aws-sdk-go-v2/service/ssooidcApache-2.0
github.com/aws/aws-sdk-go-v2/service/stsApache-2.0
github.com/aws/smithy-goApache-2.0
github.com/aws/smithy-go/internal/sync/singleflightBSD-3-Clause
github.com/bahlo/generic-list-goBSD-3-Clause
github.com/buger/jsonparserMIT
github.com/cenkalti/backoff/v4MIT
github.com/cloudquery/cloudquery-api-goMPL-2.0
github.com/cloudquery/plugin-pb-goMPL-2.0
github.com/cloudquery/plugin-sdk/v2/internal/globMIT
github.com/cloudquery/plugin-sdk/v2/schemaMIT
github.com/cloudquery/plugin-sdk/v2/typesMPL-2.0
github.com/cloudquery/plugin-sdk/v4MPL-2.0
github.com/cloudquery/plugin-sdk/v4/globMIT
github.com/cloudquery/plugin-sdk/v4/scalarMIT
github.com/davecgh/go-spew/spewISC
github.com/felixge/httpsnoopMIT
github.com/ghodss/yamlMIT
github.com/go-logr/logrApache-2.0
github.com/go-logr/stdrApache-2.0
github.com/goccy/go-jsonMIT
github.com/golang/groupcache/lruApache-2.0
github.com/google/flatbuffers/goApache-2.0
github.com/google/s2a-goApache-2.0
github.com/google/uuidBSD-3-Clause
github.com/googleapis/enterprise-certificate-proxy/clientApache-2.0
github.com/googleapis/gax-go/v2BSD-3-Clause
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptorsApache-2.0
github.com/grpc-ecosystem/grpc-gateway/v2BSD-3-Clause
github.com/hashicorp/go-cleanhttpMPL-2.0
github.com/hashicorp/go-retryablehttpMPL-2.0
github.com/invopop/jsonschemaMIT
github.com/klauspost/compressApache-2.0
github.com/klauspost/compress/internal/snaprefBSD-3-Clause
github.com/klauspost/compress/zstd/internal/xxhashMIT
github.com/mailru/easyjsonMIT
github.com/mattn/go-colorableMIT
github.com/mattn/go-isattyMIT
github.com/oapi-codegen/runtimeApache-2.0
github.com/pierrec/lz4/v4BSD-3-Clause
github.com/pmezard/go-difflib/difflibBSD-3-Clause
github.com/rs/zerologMIT
github.com/santhosh-tekuri/jsonschema/v6Apache-2.0
github.com/spf13/cobraApache-2.0
github.com/spf13/pflagBSD-3-Clause
github.com/stretchr/testifyMIT
github.com/thoas/go-funkMIT
github.com/wk8/go-ordered-map/v2Apache-2.0
github.com/zeebo/xxh3BSD-2-Clause
go.opencensus.ioApache-2.0
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpcApache-2.0
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttpApache-2.0
go.opentelemetry.io/otelApache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttpApache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttpApache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptraceApache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttpApache-2.0
go.opentelemetry.io/otel/logApache-2.0
go.opentelemetry.io/otel/metricApache-2.0
go.opentelemetry.io/otel/sdkApache-2.0
go.opentelemetry.io/otel/sdk/logApache-2.0
go.opentelemetry.io/otel/sdk/metricApache-2.0
go.opentelemetry.io/otel/traceApache-2.0
go.opentelemetry.io/proto/otlpApache-2.0
golang.org/x/cryptoBSD-3-Clause
golang.org/x/expBSD-3-Clause
golang.org/x/netBSD-3-Clause
golang.org/x/oauth2BSD-3-Clause
golang.org/x/syncBSD-3-Clause
golang.org/x/sysBSD-3-Clause
golang.org/x/textBSD-3-Clause
golang.org/x/time/rateBSD-3-Clause
golang.org/x/xerrorsBSD-3-Clause
google.golang.org/apiBSD-3-Clause
google.golang.org/api/internal/third_party/uritemplatesBSD-3-Clause
google.golang.org/genproto/googleapis/apiApache-2.0
google.golang.org/genproto/googleapis/rpcApache-2.0
google.golang.org/genproto/googleapis/type/latlngApache-2.0
google.golang.org/grpcApache-2.0
google.golang.org/protobufBSD-3-Clause
gopkg.in/yaml.v2Apache-2.0
gopkg.in/yaml.v3MIT


Join our mailing list

Subscribe to our newsletter to make sure you don't miss any updates.

Legal

© 2024 CloudQuery, Inc. All rights reserved.

We use tracking cookies to understand how you use the product and help us improve it. Please accept cookies to help us improve. You can always opt out later via the link in the footer.