Amazon Kinesis Firehose Destination Plugin

Authentication #

Example #

kind: destination
spec:
  name: "firehose"
  path: "cloudquery/firehose"
  registry: "cloudquery"
  version: "v2.7.15"
  write_mode: "append" # this plugin only supports 'append' mode
  spec:
    # Required parameters e.g. arn:aws:firehose:us-east-1:111122223333:deliverystream/TestRedshiftStream
    stream_arn: "${FIREHOSE_STREAM_ARN}"
    # Optional parameters
    # max_retries: 5
    # max_record_size_bytes: 1024000 # optional
    # max_batch_records: 500 # optional
    # max_batch_size_bytes: 4194000 # optional

• The maximum size of a record sent to Kinesis Data Firehose, before base64-encoding, is 1,000 KiB.
• The PutRecordBatch operation can take up to 500 records per batch or 4 MiB per batch, whichever is smaller.

Firehose Spec #

• stream_arn (string) (required)

  Kinesis Firehose delivery stream ARN where data will be sent to.

  Format: arn:${Partition}:firehose:${Region}:${Account}:deliverystream/${DeliveryStreamName}.
• max_retries (integer) (optional) (default: 5)

  Amount of retries to perform when writing a batch.
• max_record_size_bytes (integer) (optional) (default: 1024000 (1000 KiB))

  Number of bytes (as Arrow buffer size) to write before starting a new record.
• max_batch_records (integer) (optional) (default: 500 (1000 KiB))

  Number of records allowed in a single batch.
• max_batch_size_bytes (integer) (optional) (default: 4194000 (~4000 KiB))

  Number of bytes allowed in a single batch.