feat: Added tap-cratedb and target-cratedb#2235
Open
edgarrmondragon wants to merge 10 commits intomainfrom
Open
feat: Added tap-cratedb and target-cratedb#2235edgarrmondragon wants to merge 10 commits intomainfrom
tap-cratedb and target-cratedb#2235edgarrmondragon wants to merge 10 commits intomainfrom
Conversation
Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
✅ Deploy Preview for meltano-hub ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Collaborator
Author
|
cc @amotl |
|
Testing plugin
Auto-generated README.md
|
| Setting | Required | Default | Description |
|---|---|---|---|
| host | False | None | Hostname for postgres instance. Note if sqlalchemy_url is set this will be ignored. |
| port | False | 5432 | The port on which postgres is awaiting connection. Note if sqlalchemy_url is set this will be ignored. |
| user | False | None | User name used to authenticate. Note if sqlalchemy_url is set this will be ignored. |
| password | False | None | Password used to authenticate. Note if sqlalchemy_url is set this will be ignored. |
| database | False | None | Database name. Note if sqlalchemy_url is set this will be ignored. |
| use_copy | False | False | Use the COPY command to insert data. This is usually faster than INSERT statements. This option is only available for the postgresql+psycopg dialect+driver. |
| sqlalchemy_url | False | None | SQLAlchemy connection string. This will override using host, user, password, port, dialect, and all ssl settings. It is strongly recommended the driver is compatible with psycopg3. Note that you must escape password special characters properly. See https://docs.sqlalchemy.org/en/20/core/engines.html#escaping-special-characters-such-as-signs-in-passwords |
| dialect+driver | False | postgresql+psycopg | DEPRECATED. Dialect+driver see https://docs.sqlalchemy.org/en/20/core/engines.html. Generally just leave this alone. |
| default_target_schema | False | melty | Postgres schema to send data to, example: tap-clickup |
| activate_version | False | True | If set to false, the tap will ignore activate version messages. If set to true, add_record_metadata must be set to true as well. |
| hard_delete | False | False | When activate version is sent from a tap this specifies if we should delete the records that don't match, or mark them with a date in the _sdc_deleted_at column. This config option is ignored if activate_version is set to false. |
| add_record_metadata | False | True | Note that this must be enabled for activate_version to work!This adds _sdc_extracted_at, _sdc_batched_at, and more to every table. See https://sdk.meltano.com/en/latest/implementation/record_metadata.html for more information. |
| interpret_content_encoding | False | False | If set to true, the target will interpret the content encoding of the schema to determine how to store the data. Using this option may result in a more efficient storage of the data but may also result in an error if the data is not encoded as expected. |
| sanitize_null_text_characters | False | False | If set to true, the target will sanitize null characters in char/text/varchar fields, as they are not supported by Postgres. See postgres documentation for more information about chr(0) not being supported. |
| ssl_enable | False | False | Whether or not to use ssl to verify the server's identity. Use ssl_certificate_authority and ssl_mode for further customization. To use a client certificate to authenticate yourself to the server, use ssl_client_certificate_enable instead. Note if sqlalchemy_url is set this will be ignored. |
| ssl_client_certificate_enable | False | False | Whether or not to provide client-side certificates as a method of authentication to the server. Use ssl_client_certificate and ssl_client_private_key for further customization. To use SSL to verify the server's identity, use ssl_enable instead. Note if sqlalchemy_url is set this will be ignored. |
| ssl_mode | False | verify-full | SSL Protection method, see postgres documentation for more information. Must be one of disable, allow, prefer, require, verify-ca, or verify-full. Note if sqlalchemy_url is set this will be ignored. |
| ssl_certificate_authority | False | ~/.postgresql/root.crl | The certificate authority that should be used to verify the server's identity. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored. |
| ssl_client_certificate | False | ~/.postgresql/postgresql.crt | The certificate that should be used to verify your identity to the server. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored. |
| ssl_client_private_key | False | ~/.postgresql/postgresql.key | The private key for the certificate you provided. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored. |
| ssl_storage_directory | False | .secrets | The folder in which to store SSL certificates provided as raw values. When a certificate/key is provided as a raw value instead of as a filepath, it must be written to a file before it can be used. This configuration option determines where that file is created. |
| ssh_tunnel | False | None | SSH Tunnel Configuration, this is a json object |
| ssh_tunnel.enable | False | False | Enable an ssh tunnel (also known as bastion host), see the other ssh_tunnel.* properties for more details |
| ssh_tunnel.host | False | None | Host of the bastion host, this is the host we'll connect to via ssh |
| ssh_tunnel.username | False | None | Username to connect to bastion host |
| ssh_tunnel.port | False | 22 | Port to connect to bastion host |
| ssh_tunnel.private_key | False | None | Private Key for authentication to the bastion host |
| ssh_tunnel.private_key_password | False | None | Private Key Password, leave None if no password is set |
| load_method | False | append-only | The method to use when loading data into the destination. append-only will always write all input records whether that records already exists or not. upsert will update existing records and insert new records. overwrite will delete all existing records and insert all input records. |
| batch_size_rows | False | None | Maximum number of rows in each batch. |
| process_activate_version_messages | False | True | Whether to process ACTIVATE_VERSION messages. |
| validate_records | False | True | Whether to validate the schema of the incoming streams. |
| stream_maps | False | None | Config object for stream maps capability. For more information check out Stream Maps. |
| stream_maps.else | False | None | Currently, only setting this to __NULL__ is supported. This will remove all other streams. |
| stream_map_config | False | None | User-defined config values to be used within map expressions. |
| faker_config | False | None | Config for the Faker instance variable fake used within map expressions. Only applicable if the plugin specifies faker as an additional dependency (through the singer-sdk faker extra or directly). |
| faker_config.seed | False | None | Value to seed the Faker generator for deterministic output: https://faker.readthedocs.io/en/master/#seeding-the-generator |
| faker_config.locale | False | None | One or more LCID locale strings to produce localized output for: https://faker.readthedocs.io/en/master/#localization |
| flattening_enabled | False | None | 'True' to enable schema flattening and automatically expand nested properties. |
| flattening_max_depth | False | None | The max depth to flatten schemas. |
| flattening_max_key_length | False | None | The maximum length of a flattened key. |
A full list of supported settings and capabilities is available by running: target-cratedb --about
Version info
target-cratedb v0.6.3, Meltano SDK v0.53.6
Usage info
melty-bot % meltano invoke target-cratedb --help
Usage: target-cratedb [OPTIONS]
Execute the Singer target.
Options:
--version Display the package version.
--about Display package metadata and settings.
--format [json|markdown] Specify output style for --about
--config TEXT Configuration file location or 'ENV' to use
environment variables.
--input FILENAME A path to read messages from instead of from
standard in.
--help Show this message and exit.
Detected capabilities
- ❌ 'discover'
- ❌ 'catalog'
- ❌ 'properties'
- ❌ 'state'
- ✅ 'about'
JSON Metadata
{
"name": "target-cratedb",
"description": "Target for CrateDB.",
"version": "0.6.3",
"sdk_version": "0.53.6",
"supported_python_versions": [
"3.10",
"3.11",
"3.12",
"3.13",
"3.14"
],
"capabilities": [
"about",
"stream-maps",
"schema-flattening",
"structured-logging",
"validate-records",
"activate-version",
"target-schema",
"hard-delete"
],
"settings": {
"type": "object",
"properties": {
"host": {
"type": [
"string",
"null"
],
"description": "Hostname for postgres instance. Note if sqlalchemy_url is set this will be ignored."
},
"port": {
"type": [
"integer",
"null"
],
"default": 5432,
"description": "The port on which postgres is awaiting connection. Note if sqlalchemy_url is set this will be ignored."
},
"user": {
"type": [
"string",
"null"
],
"description": "User name used to authenticate. Note if sqlalchemy_url is set this will be ignored."
},
"password": {
"type": [
"string",
"null"
],
"description": "Password used to authenticate. Note if sqlalchemy_url is set this will be ignored."
},
"database": {
"type": [
"string",
"null"
],
"description": "Database name. Note if sqlalchemy_url is set this will be ignored."
},
"use_copy": {
"type": [
"boolean",
"null"
],
"title": "Use COPY",
"default": false,
"description": "Use the COPY command to insert data. This is usually faster than INSERT statements. This option is only available for the postgresql+psycopg dialect+driver."
},
"sqlalchemy_url": {
"type": [
"string",
"null"
],
"description": "SQLAlchemy connection string. This will override using host, user, password, port, dialect, and all ssl settings. It is strongly recommended the driver is compatible with psycopg3. Note that you must escape password special characters properly. See https://docs.sqlalchemy.org/en/20/core/engines.html#escaping-special-characters-such-as-signs-in-passwords"
},
"dialect+driver": {
"type": [
"string",
"null"
],
"default": "postgresql+psycopg",
"description": "DEPRECATED. Dialect+driver see https://docs.sqlalchemy.org/en/20/core/engines.html. Generally just leave this alone."
},
"default_target_schema": {
"type": [
"string",
"null"
],
"default": "melty",
"description": "Postgres schema to send data to, example: tap-clickup"
},
"activate_version": {
"type": [
"boolean",
"null"
],
"default": true,
"description": "If set to false, the tap will ignore activate version messages. If set to true, add_record_metadata must be set to true as well."
},
"hard_delete": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "When activate version is sent from a tap this specifies if we should delete the records that don't match, or mark them with a date in the `_sdc_deleted_at` column. This config option is ignored if `activate_version` is set to false."
},
"add_record_metadata": {
"type": [
"boolean",
"null"
],
"default": true,
"description": "Note that this must be enabled for activate_version to work!This adds _sdc_extracted_at, _sdc_batched_at, and more to every table. See https://sdk.meltano.com/en/latest/implementation/record_metadata.html for more information."
},
"interpret_content_encoding": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "If set to true, the target will interpret the content encoding of the schema to determine how to store the data. Using this option may result in a more efficient storage of the data but may also result in an error if the data is not encoded as expected."
},
"sanitize_null_text_characters": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "If set to true, the target will sanitize null characters in char/text/varchar fields, as they are not supported by Postgres. See [postgres documentation](https://www.postgresql.org/docs/current/functions-string.html) for more information about chr(0) not being supported."
},
"ssl_enable": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Whether or not to use ssl to verify the server's identity. Use ssl_certificate_authority and ssl_mode for further customization. To use a client certificate to authenticate yourself to the server, use ssl_client_certificate_enable instead. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_client_certificate_enable": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Whether or not to provide client-side certificates as a method of authentication to the server. Use ssl_client_certificate and ssl_client_private_key for further customization. To use SSL to verify the server's identity, use ssl_enable instead. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_mode": {
"type": [
"string",
"null"
],
"default": "verify-full",
"description": "SSL Protection method, see [postgres documentation](https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-PROTECTION) for more information. Must be one of disable, allow, prefer, require, verify-ca, or verify-full. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_certificate_authority": {
"type": [
"string",
"null"
],
"default": "~/.postgresql/root.crl",
"description": "The certificate authority that should be used to verify the server's identity. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_client_certificate": {
"type": [
"string",
"null"
],
"default": "~/.postgresql/postgresql.crt",
"description": "The certificate that should be used to verify your identity to the server. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_client_private_key": {
"type": [
"string",
"null"
],
"default": "~/.postgresql/postgresql.key",
"description": "The private key for the certificate you provided. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_storage_directory": {
"type": [
"string",
"null"
],
"default": ".secrets",
"description": "The folder in which to store SSL certificates provided as raw values. When a certificate/key is provided as a raw value instead of as a filepath, it must be written to a file before it can be used. This configuration option determines where that file is created."
},
"ssh_tunnel": {
"type": [
"object",
"null"
],
"properties": {
"enable": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Enable an ssh tunnel (also known as bastion host), see the other ssh_tunnel.* properties for more details"
},
"host": {
"type": [
"string",
"null"
],
"description": "Host of the bastion host, this is the host we'll connect to via ssh"
},
"username": {
"type": [
"string",
"null"
],
"description": "Username to connect to bastion host"
},
"port": {
"type": [
"integer",
"null"
],
"default": 22,
"description": "Port to connect to bastion host"
},
"private_key": {
"type": [
"string",
"null"
],
"description": "Private Key for authentication to the bastion host",
"secret": true,
"writeOnly": true
},
"private_key_password": {
"type": [
"string",
"null"
],
"description": "Private Key Password, leave None if no password is set",
"secret": true,
"writeOnly": true
}
},
"additionalProperties": true,
"description": "SSH Tunnel Configuration, this is a json object"
},
"load_method": {
"type": [
"string",
"null"
],
"default": "append-only",
"description": "The method to use when loading data into the destination. `append-only` will always write all input records whether that records already exists or not. `upsert` will update existing records and insert new records. `overwrite` will delete all existing records and insert all input records.",
"enum": [
"append-only",
"upsert",
"overwrite"
]
},
"batch_size_rows": {
"type": [
"integer",
"null"
],
"title": "Batch Size Rows",
"description": "Maximum number of rows in each batch."
},
"process_activate_version_messages": {
"type": [
"boolean",
"null"
],
"title": "Process `ACTIVATE_VERSION` messages",
"default": true,
"description": "Whether to process `ACTIVATE_VERSION` messages."
},
"validate_records": {
"type": [
"boolean",
"null"
],
"title": "Validate Records",
"default": true,
"description": "Whether to validate the schema of the incoming streams."
},
"stream_maps": {
"type": [
"object",
"null"
],
"properties": {
"__else__": {
"anyOf": [
{
"const": "__NULL__"
},
{
"type": "null"
},
"null"
],
"title": "Other streams",
"description": "Currently, only setting this to `__NULL__` is supported. This will remove all other streams."
}
},
"additionalProperties": {
"anyOf": [
{
"const": "__NULL__"
},
{
"type": "null"
},
{
"type": "object",
"properties": {
"__alias__": {
"type": [
"string",
"null"
],
"title": "Stream Alias",
"description": "Alias to use for the stream."
},
"__else__": {
"anyOf": [
{
"const": "__NULL__"
},
{
"type": "null"
},
"null"
],
"title": "Other properties",
"description": "Currently, only setting this to `__NULL__` is supported. This will remove all other properties from the stream."
},
"__filter__": {
"type": [
"string",
"null"
],
"title": "Filter",
"description": "Filter out records from a stream. A string expression which must evaluate to `true` to include the record, or `false` to exclude it. Filter expressions should be wrapped in `bool()` to ensure proper type conversion."
},
"__key_properties__": {
"type": [
"array",
"null"
],
"items": {
"type": [
"string"
]
},
"title": "Key Properties",
"description": "Primary key properties for the stream."
},
"__source__": {
"type": [
"string",
"null"
],
"description": "Create a new stream from this source stream."
}
},
"additionalProperties": {
"anyOf": [
{
"const": "__NULL__"
},
{
"type": "null"
},
{
"type": [
"string"
]
}
]
}
}
]
},
"title": "Stream Maps",
"description": "Config object for stream maps capability. For more information check out [Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html)."
},
"stream_map_config": {
"type": [
"object",
"null"
],
"properties": {},
"additionalProperties": true,
"title": "User Stream Map Configuration",
"description": "User-defined config values to be used within map expressions."
},
"faker_config": {
"type": [
"object",
"null"
],
"properties": {
"seed": {
"oneOf": [
{
"type": [
"number"
]
},
{
"type": [
"string"
]
},
{
"type": [
"boolean"
]
},
{
"type": "null"
}
],
"title": "Faker Seed",
"description": "Value to seed the Faker generator for deterministic output: https://faker.readthedocs.io/en/master/#seeding-the-generator"
},
"locale": {
"oneOf": [
{
"type": [
"string"
]
},
{
"type": "array",
"items": {
"type": [
"string"
]
}
},
{
"type": "null"
}
],
"title": "Faker Locale",
"description": "One or more LCID locale strings to produce localized output for: https://faker.readthedocs.io/en/master/#localization"
}
},
"additionalProperties": true,
"title": "Faker Configuration",
"description": "Config for the [`Faker`](https://faker.readthedocs.io/en/master/) instance variable `fake` used within map expressions. Only applicable if the plugin specifies `faker` as an additional dependency (through the `singer-sdk` `faker` extra or directly)."
},
"flattening_enabled": {
"type": [
"boolean",
"null"
],
"title": "Enable Schema Flattening",
"description": "'True' to enable schema flattening and automatically expand nested properties."
},
"flattening_max_depth": {
"type": [
"integer",
"null"
],
"title": "Max Flattening Depth",
"description": "The max depth to flatten schemas."
},
"flattening_max_key_length": {
"type": [
"integer",
"null"
],
"title": "Max Key Length",
"description": "The maximum length of a flattened key."
}
},
"$schema": "https://json-schema.org/draft/2020-12/schema"
}
}
|
Testing plugin
Auto-generated README.md
|
| Setting | Required | Default | Description |
|---|---|---|---|
| replication_slot_name | False | tappostgres | Name of the replication slot to use for logical replication. Must be unique for parallel extractions. Only applicable when replication_method is LOG_BASED.- Contain only letters, numbers, and underscores. - Be less than or equal to 63 characters. - Not start with 'pg_'. |
| replication_max_run_seconds | False | 600 | The maximum number of seconds to run the replication loop for. Only applicable when replication_method is LOG_BASED. |
| replication_idle_exit_seconds | False | 60 | The number of seconds to wait for data messages before exiting the replication loop. Only applicable when replication_method is LOG_BASED. |
| host | False | None | Hostname for postgres instance. Note if sqlalchemy_url is set this will be ignored. |
| port | False | 5432 | The port on which postgres is awaiting connection. Note if sqlalchemy_url is set this will be ignored. |
| user | False | None | User name used to authenticate. Note if sqlalchemy_url is set this will be ignored. |
| password | False | None | Password used to authenticate. Note if sqlalchemy_url is set this will be ignored. |
| database | False | None | Database name. Note if sqlalchemy_url is set this will be ignored. |
| max_record_count | False | None | Optional. The maximum number of records to return in a single stream. |
| sqlalchemy_url | False | None | Example postgresql://[username]:[password]@localhost:5432/[db_name] |
| filter_schemas | False | None | If an array of schema names is provided, the tap will only process the specified Postgres schemas and ignore others. If left blank, the tap automatically determines ALL available Postgres schemas. |
| dates_as_string | False | False | Defaults to false, if true, date, and timestamp fields will be Strings. If you see ValueError: Year is out of range, try setting this to True. |
| json_as_object | False | False | Defaults to false, if true, json and jsonb fields will be Objects. |
| stream_options | False | None | |
| ssh_tunnel | False | None | SSH Tunnel Configuration, this is a json object |
| ssh_tunnel.enable | False | False | Enable an ssh tunnel (also known as bastion server), see the other ssh_tunnel.* properties for more details |
| ssh_tunnel.host | False | None | Host of the bastion server, this is the host we'll connect to via ssh |
| ssh_tunnel.username | False | None | Username to connect to bastion server |
| ssh_tunnel.port | False | 22 | Port to connect to bastion server |
| ssh_tunnel.private_key | False | None | Private Key for authentication to the bastion server |
| ssh_tunnel.private_key_password | False | None | Private Key Password, leave None if no password is set |
| ssl_enable | False | False | Whether or not to use ssl to verify the server's identity. Use ssl_certificate_authority and ssl_mode for further customization. To use a client certificate to authenticate yourself to the server, use ssl_client_certificate_enable instead. Note if sqlalchemy_url is set this will be ignored. |
| ssl_client_certificate_enable | False | False | Whether or not to provide client-side certificates as a method of authentication to the server. Use ssl_client_certificate and ssl_client_private_key for further customization. To use SSL to verify the server's identity, use ssl_enable instead. Note if sqlalchemy_url is set this will be ignored. |
| ssl_mode | False | verify-full | SSL Protection method, see postgres documentation for more information. Must be one of disable, allow, prefer, require, verify-ca, or verify-full. Note if sqlalchemy_url is set this will be ignored. |
| ssl_certificate_authority | False | ~/.postgresql/root.crl | The certificate authority that should be used to verify the server's identity. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored. |
| ssl_client_certificate | False | ~/.postgresql/postgresql.crt | The certificate that should be used to verify your identity to the server. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored. |
| ssl_client_private_key | False | ~/.postgresql/postgresql.key | The private key for the certificate you provided. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored. |
| ssl_storage_directory | False | .secrets | The folder in which to store SSL certificates provided as raw values. When a certificate/key is provided as a raw value instead of as a filepath, it must be written to a file before it can be used. This configuration option determines where that file is created. |
| default_replication_method | False | FULL_TABLE | Replication method to use if there is not a catalog entry to override this choice. One of FULL_TABLE, INCREMENTAL, or LOG_BASED. |
| use_singer_decimal | False | None | Whether to use use strings with x-singer.decimal format for decimals in the discovered schema. This is useful to avoid precision loss when working with large numbers. |
| stream_maps | False | None | Config object for stream maps capability. For more information check out Stream Maps. |
| stream_maps.else | False | None | Currently, only setting this to __NULL__ is supported. This will remove all other streams. |
| stream_map_config | False | None | User-defined config values to be used within map expressions. |
| faker_config | False | None | Config for the Faker instance variable fake used within map expressions. Only applicable if the plugin specifies faker as an additional dependency (through the singer-sdk faker extra or directly). |
| faker_config.seed | False | None | Value to seed the Faker generator for deterministic output: https://faker.readthedocs.io/en/master/#seeding-the-generator |
| faker_config.locale | False | None | One or more LCID locale strings to produce localized output for: https://faker.readthedocs.io/en/master/#localization |
| flattening_enabled | False | None | 'True' to enable schema flattening and automatically expand nested properties. |
| flattening_max_depth | False | None | The max depth to flatten schemas. |
| flattening_max_key_length | False | None | The maximum length of a flattened key. |
| batch_config | False | None | Configuration for BATCH message capabilities. |
| batch_config.encoding | False | None | Specifies the format and compression of the batch files. |
| batch_config.encoding.format | False | None | Format to use for batch files. |
| batch_config.encoding.compression | False | None | Compression format to use for batch files. |
| batch_config.storage | False | None | Defines the storage layer to use when writing batch files |
| batch_config.storage.root | False | None | Root path to use when writing batch files. |
| batch_config.storage.prefix | False | None | Prefix to use when writing batch files. |
A full list of supported settings and capabilities is available by running: tap-cratedb --about
Version info
tap-cratedb v0.0.1, Meltano SDK v0.53.6
Usage info
melty-bot % meltano invoke tap-cratedb --help
Usage: tap-cratedb [OPTIONS]
Execute the Singer tap.
Options:
--version Display the package version.
--about Display package metadata and settings.
--format [json|markdown] Specify output style for --about
--config TEXT Configuration file location or 'ENV' to use
environment variables.
--discover Run the tap in discovery mode.
--test TEXT Use --test to sync a single record for each
stream. Use --test=schema to test schema output
without syncing records.
--catalog FILENAME Use a Singer catalog file with the tap.
--state FILENAME Use a bookmarks file for incremental replication.
--help Show this message and exit.
Detected capabilities
- ✅ 'discover'
- ✅ 'catalog'
- ✅ 'state'
- ✅ 'about'
JSON Metadata
{
"name": "tap-cratedb",
"description": null,
"version": "0.0.1",
"sdk_version": "0.53.6",
"supported_python_versions": [
"3.9",
"3.10",
"3.11",
"3.12",
"3.13",
"3.14"
],
"capabilities": [
"catalog",
"state",
"discover",
"activate-version",
"about",
"stream-maps",
"schema-flattening",
"batch",
"structured-logging"
],
"settings": {
"type": "object",
"properties": {
"replication_slot_name": {
"type": [
"string",
"null"
],
"pattern": "^(?!pg_)[A-Za-z0-9_]{1,63}$",
"default": "tappostgres",
"description": "Name of the replication slot to use for logical replication. Must be unique for parallel extractions. Only applicable when replication_method is LOG_BASED.- Contain only letters, numbers, and underscores. - Be less than or equal to 63 characters. - Not start with 'pg_'."
},
"replication_max_run_seconds": {
"type": [
"integer",
"null"
],
"default": 600,
"description": "The maximum number of seconds to run the replication loop for. Only applicable when replication_method is LOG_BASED."
},
"replication_idle_exit_seconds": {
"type": [
"integer",
"null"
],
"default": 60,
"description": "The number of seconds to wait for data messages before exiting the replication loop. Only applicable when replication_method is LOG_BASED."
},
"host": {
"type": [
"string",
"null"
],
"description": "Hostname for postgres instance. Note if sqlalchemy_url is set this will be ignored."
},
"port": {
"type": [
"integer",
"null"
],
"default": 5432,
"description": "The port on which postgres is awaiting connection. Note if sqlalchemy_url is set this will be ignored."
},
"user": {
"type": [
"string",
"null"
],
"description": "User name used to authenticate. Note if sqlalchemy_url is set this will be ignored."
},
"password": {
"type": [
"string",
"null"
],
"description": "Password used to authenticate. Note if sqlalchemy_url is set this will be ignored.",
"secret": true,
"writeOnly": true
},
"database": {
"type": [
"string",
"null"
],
"description": "Database name. Note if sqlalchemy_url is set this will be ignored."
},
"max_record_count": {
"type": [
"integer",
"null"
],
"description": "Optional. The maximum number of records to return in a single stream."
},
"sqlalchemy_url": {
"type": [
"string",
"null"
],
"description": "Example postgresql://[username]:[password]@localhost:5432/[db_name]",
"secret": true,
"writeOnly": true
},
"filter_schemas": {
"type": [
"array",
"null"
],
"items": {
"type": [
"string"
]
},
"description": "If an array of schema names is provided, the tap will only process the specified Postgres schemas and ignore others. If left blank, the tap automatically determines ALL available Postgres schemas."
},
"dates_as_string": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Defaults to false, if true, date, and timestamp fields will be Strings. If you see ValueError: Year is out of range, try setting this to True."
},
"json_as_object": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Defaults to false, if true, json and jsonb fields will be Objects."
},
"stream_options": {
"type": [
"object",
"null"
],
"properties": {},
"additionalProperties": {
"type": "object",
"properties": {
"custom_where_clauses": {
"type": [
"array",
"null"
],
"items": {
"type": [
"string"
]
},
"default": [],
"description": "If an array of custom WHERE clauses is provided, the tap will only process the records that match the WHERE clauses. The WHERE clauses are combined using the AND operator."
}
},
"additionalProperties": true
}
},
"ssh_tunnel": {
"type": [
"object",
"null"
],
"properties": {
"enable": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Enable an ssh tunnel (also known as bastion server), see the other ssh_tunnel.* properties for more details"
},
"host": {
"type": [
"string",
"null"
],
"description": "Host of the bastion server, this is the host we'll connect to via ssh"
},
"username": {
"type": [
"string",
"null"
],
"description": "Username to connect to bastion server"
},
"port": {
"type": [
"integer",
"null"
],
"default": 22,
"description": "Port to connect to bastion server"
},
"private_key": {
"type": [
"string",
"null"
],
"description": "Private Key for authentication to the bastion server",
"secret": true,
"writeOnly": true
},
"private_key_password": {
"type": [
"string",
"null"
],
"description": "Private Key Password, leave None if no password is set",
"secret": true,
"writeOnly": true
}
},
"additionalProperties": true,
"description": "SSH Tunnel Configuration, this is a json object"
},
"ssl_enable": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Whether or not to use ssl to verify the server's identity. Use ssl_certificate_authority and ssl_mode for further customization. To use a client certificate to authenticate yourself to the server, use ssl_client_certificate_enable instead. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_client_certificate_enable": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Whether or not to provide client-side certificates as a method of authentication to the server. Use ssl_client_certificate and ssl_client_private_key for further customization. To use SSL to verify the server's identity, use ssl_enable instead. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_mode": {
"type": [
"string",
"null"
],
"default": "verify-full",
"description": "SSL Protection method, see [postgres documentation](https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-PROTECTION) for more information. Must be one of disable, allow, prefer, require, verify-ca, or verify-full. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_certificate_authority": {
"type": [
"string",
"null"
],
"default": "~/.postgresql/root.crl",
"description": "The certificate authority that should be used to verify the server's identity. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_client_certificate": {
"type": [
"string",
"null"
],
"default": "~/.postgresql/postgresql.crt",
"description": "The certificate that should be used to verify your identity to the server. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_client_private_key": {
"type": [
"string",
"null"
],
"default": "~/.postgresql/postgresql.key",
"description": "The private key for the certificate you provided. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_storage_directory": {
"type": [
"string",
"null"
],
"default": ".secrets",
"description": "The folder in which to store SSL certificates provided as raw values. When a certificate/key is provided as a raw value instead of as a filepath, it must be written to a file before it can be used. This configuration option determines where that file is created."
},
"default_replication_method": {
"type": [
"string",
"null"
],
"default": "FULL_TABLE",
"description": "Replication method to use if there is not a catalog entry to override this choice. One of `FULL_TABLE`, `INCREMENTAL`, or `LOG_BASED`.",
"enum": [
"FULL_TABLE",
"INCREMENTAL",
"LOG_BASED"
]
},
"use_singer_decimal": {
"type": [
"boolean",
"null"
],
"title": "Use Singer Decimal",
"description": "Whether to use use strings with `x-singer.decimal` format for decimals in the discovered schema. This is useful to avoid precision loss when working with large numbers."
},
"stream_maps": {
"type": [
"object",
"null"
],
"properties": {
"__else__": {
"anyOf": [
{
"const": "__NULL__"
},
{
"type": "null"
},
"null"
],
"title": "Other streams",
"description": "Currently, only setting this to `__NULL__` is supported. This will remove all other streams."
}
},
"additionalProperties": {
"anyOf": [
{
"const": "__NULL__"
},
{
"type": "null"
},
{
"type": "object",
"properties": {
"__alias__": {
"type": [
"string",
"null"
],
"title": "Stream Alias",
"description": "Alias to use for the stream."
},
"__else__": {
"anyOf": [
{
"const": "__NULL__"
},
{
"type": "null"
},
"null"
],
"title": "Other properties",
"description": "Currently, only setting this to `__NULL__` is supported. This will remove all other properties from the stream."
},
"__filter__": {
"type": [
"string",
"null"
],
"title": "Filter",
"description": "Filter out records from a stream. A string expression which must evaluate to `true` to include the record, or `false` to exclude it. Filter expressions should be wrapped in `bool()` to ensure proper type conversion."
},
"__key_properties__": {
"type": [
"array",
"null"
],
"items": {
"type": [
"string"
]
},
"title": "Key Properties",
"description": "Primary key properties for the stream."
},
"__source__": {
"type": [
"string",
"null"
],
"description": "Create a new stream from this source stream."
}
},
"additionalProperties": {
"anyOf": [
{
"const": "__NULL__"
},
{
"type": "null"
},
{
"type": [
"string"
]
}
]
}
}
]
},
"title": "Stream Maps",
"description": "Config object for stream maps capability. For more information check out [Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html)."
},
"stream_map_config": {
"type": [
"object",
"null"
],
"properties": {},
"additionalProperties": true,
"title": "User Stream Map Configuration",
"description": "User-defined config values to be used within map expressions."
},
"faker_config": {
"type": [
"object",
"null"
],
"properties": {
"seed": {
"oneOf": [
{
"type": [
"number"
]
},
{
"type": [
"string"
]
},
{
"type": [
"boolean"
]
},
{
"type": "null"
}
],
"title": "Faker Seed",
"description": "Value to seed the Faker generator for deterministic output: https://faker.readthedocs.io/en/master/#seeding-the-generator"
},
"locale": {
"oneOf": [
{
"type": [
"string"
]
},
{
"type": "array",
"items": {
"type": [
"string"
]
}
},
{
"type": "null"
}
],
"title": "Faker Locale",
"description": "One or more LCID locale strings to produce localized output for: https://faker.readthedocs.io/en/master/#localization"
}
},
"additionalProperties": true,
"title": "Faker Configuration",
"description": "Config for the [`Faker`](https://faker.readthedocs.io/en/master/) instance variable `fake` used within map expressions. Only applicable if the plugin specifies `faker` as an additional dependency (through the `singer-sdk` `faker` extra or directly)."
},
"flattening_enabled": {
"type": [
"boolean",
"null"
],
"title": "Enable Schema Flattening",
"description": "'True' to enable schema flattening and automatically expand nested properties."
},
"flattening_max_depth": {
"type": [
"integer",
"null"
],
"title": "Max Flattening Depth",
"description": "The max depth to flatten schemas."
},
"flattening_max_key_length": {
"type": [
"integer",
"null"
],
"title": "Max Key Length",
"description": "The maximum length of a flattened key."
},
"batch_config": {
"type": [
"object",
"null"
],
"properties": {
"encoding": {
"type": [
"object",
"null"
],
"properties": {
"format": {
"type": [
"string",
"null"
],
"title": "Batch Encoding Format",
"description": "Format to use for batch files.",
"enum": [
"jsonl",
"parquet"
]
},
"compression": {
"type": [
"string",
"null"
],
"title": "Batch Compression Format",
"description": "Compression format to use for batch files.",
"enum": [
"gzip",
"none"
]
}
},
"additionalProperties": true,
"title": "Batch Encoding Configuration",
"description": "Specifies the format and compression of the batch files."
},
"storage": {
"type": [
"object",
"null"
],
"properties": {
"root": {
"type": [
"string",
"null"
],
"title": "Batch Storage Root",
"description": "Root path to use when writing batch files."
},
"prefix": {
"type": [
"string",
"null"
],
"title": "Batch Storage Prefix",
"description": "Prefix to use when writing batch files."
}
},
"additionalProperties": true,
"title": "Batch Storage Configuration",
"description": "Defines the storage layer to use when writing batch files"
}
},
"additionalProperties": true,
"title": "Batch Configuration",
"description": "Configuration for BATCH message capabilities."
}
},
"$schema": "https://json-schema.org/draft/2020-12/schema"
}
}
amotl
reviewed
Mar 31, 2026
amotl
left a comment
amotl
left a comment
There was a problem hiding this comment.
Thank you so much for making a start by submitting this patch. I found a few spots where CrateDB is different than PostgreSQL and flagged them for review, with a slight bias to remove them altogether.
However, the list might not be exhaustive: In general, about database options, only sqlalchemy_url is respected, with optional URL parameters schema and ssl.
Comment on lines
+112
to
+117
| - description: Defaults to false, if true, json and jsonb fields will be | ||
| Objects. | ||
| kind: boolean | ||
| label: Json As Object | ||
| name: json_as_object | ||
| value: false |
There was a problem hiding this comment.
I am not sure about this. Do you think we should validate such settings first, whether true or false is applicable here? I guess the setting is inherited from tap-postgres?
On the other hand, we can bring the adapters into the hub first, then wait for reports from users, or alternatively conduct user testing ourselves afterwards. 🍀
Comment on lines
+190
to
+248
| - description: The certificate authority that should be used to verify the | ||
| server's identity. Can be provided either as the certificate itself (in | ||
| .env) or as a filepath to the certificate. Note if sqlalchemy_url is set | ||
| this will be ignored. | ||
| kind: string | ||
| label: SSL Certificate Authority | ||
| name: ssl_certificate_authority | ||
| value: ~/.postgresql/root.crl | ||
| - description: The certificate that should be used to verify your identity to | ||
| the server. Can be provided either as the certificate itself (in .env) or as | ||
| a filepath to the certificate. Note if sqlalchemy_url is set this will be | ||
| ignored. | ||
| kind: string | ||
| label: SSL Client Certificate | ||
| name: ssl_client_certificate | ||
| value: ~/.postgresql/postgresql.crt | ||
| - description: Whether or not to provide client-side certificates as a method of | ||
| authentication to the server. Use ssl_client_certificate and | ||
| ssl_client_private_key for further customization. To use SSL to verify the | ||
| server's identity, use ssl_enable instead. Note if sqlalchemy_url is set | ||
| this will be ignored. | ||
| kind: boolean | ||
| label: SSL Client Certificate Enable | ||
| name: ssl_client_certificate_enable | ||
| value: false | ||
| - description: The private key for the certificate you provided. Can be provided | ||
| either as the certificate itself (in .env) or as a filepath to the | ||
| certificate. Note if sqlalchemy_url is set this will be ignored. | ||
| kind: password | ||
| label: SSL Client Private Key | ||
| name: ssl_client_private_key | ||
| sensitive: true | ||
| value: ~/.postgresql/postgresql.key | ||
| - description: Whether or not to use ssl to verify the server's identity. Use | ||
| ssl_certificate_authority and ssl_mode for further customization. To use a | ||
| client certificate to authenticate yourself to the server, use | ||
| ssl_client_certificate_enable instead. Note if sqlalchemy_url is set this | ||
| will be ignored. | ||
| kind: boolean | ||
| label: SSL Enable | ||
| name: ssl_enable | ||
| value: false | ||
| - description: SSL Protection method, see [postgres | ||
| documentation](https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-PROTECTION) | ||
| for more information. Must be one of disable, allow, prefer, require, | ||
| verify-ca, or verify-full. Note if sqlalchemy_url is set this will be | ||
| ignored. | ||
| kind: string | ||
| label: SSL Mode | ||
| name: ssl_mode | ||
| value: verify-full | ||
| - description: The folder in which to store SSL certificates provided as raw | ||
| values. When a certificate/key is provided as a raw value instead of as a | ||
| filepath, it must be written to a file before it can be used. This | ||
| configuration option determines where that file is created. | ||
| kind: string | ||
| label: SSL Storage Directory | ||
| name: ssl_storage_directory | ||
| value: .secrets |
There was a problem hiding this comment.
The CrateDB adapter dispatches to sqlalchemy-cratedb under the hood. This means all those parameters are not evaluated at all. The adapter exclusively considers the sqlalchemy_url configuration option.
Collaborator
Author
There was a problem hiding this comment.
I have no strong opinion, happy to remove these
amotl
reviewed
Apr 1, 2026
amotl
left a comment
amotl
left a comment
There was a problem hiding this comment.
We have one more question about the variant attribute. 🙏
Co-authored-by: Andreas Motl <andreas.motl@elmyra.de>
Co-authored-by: Andreas Motl <andreas.motl@elmyra.de>
edgarrmondragon
commented
Apr 1, 2026
Co-authored-by: Andreas Motl <andreas.motl@elmyra.de> Co-authored-by: Edgar Ramírez Mondragón <16805946+edgarrmondragon@users.noreply.github.com>
Co-authored-by: Andreas Motl <andreas.motl@elmyra.de>
|
Testing plugin
Auto-generated README.md
|
| Setting | Required | Default | Description |
|---|---|---|---|
| host | False | None | Hostname for postgres instance. Note if sqlalchemy_url is set this will be ignored. |
| port | False | 5432 | The port on which postgres is awaiting connection. Note if sqlalchemy_url is set this will be ignored. |
| user | False | None | User name used to authenticate. Note if sqlalchemy_url is set this will be ignored. |
| password | False | None | Password used to authenticate. Note if sqlalchemy_url is set this will be ignored. |
| database | False | None | Database name. Note if sqlalchemy_url is set this will be ignored. |
| use_copy | False | False | Use the COPY command to insert data. This is usually faster than INSERT statements. This option is only available for the postgresql+psycopg dialect+driver. |
| sqlalchemy_url | False | None | SQLAlchemy connection string. This will override using host, user, password, port, dialect, and all ssl settings. It is strongly recommended the driver is compatible with psycopg3. Note that you must escape password special characters properly. See https://docs.sqlalchemy.org/en/20/core/engines.html#escaping-special-characters-such-as-signs-in-passwords |
| dialect+driver | False | postgresql+psycopg | DEPRECATED. Dialect+driver see https://docs.sqlalchemy.org/en/20/core/engines.html. Generally just leave this alone. |
| default_target_schema | False | melty | Postgres schema to send data to, example: tap-clickup |
| activate_version | False | True | If set to false, the tap will ignore activate version messages. If set to true, add_record_metadata must be set to true as well. |
| hard_delete | False | False | When activate version is sent from a tap this specifies if we should delete the records that don't match, or mark them with a date in the _sdc_deleted_at column. This config option is ignored if activate_version is set to false. |
| add_record_metadata | False | True | Note that this must be enabled for activate_version to work!This adds _sdc_extracted_at, _sdc_batched_at, and more to every table. See https://sdk.meltano.com/en/latest/implementation/record_metadata.html for more information. |
| interpret_content_encoding | False | False | If set to true, the target will interpret the content encoding of the schema to determine how to store the data. Using this option may result in a more efficient storage of the data but may also result in an error if the data is not encoded as expected. |
| sanitize_null_text_characters | False | False | If set to true, the target will sanitize null characters in char/text/varchar fields, as they are not supported by Postgres. See postgres documentation for more information about chr(0) not being supported. |
| ssl_enable | False | False | Whether or not to use ssl to verify the server's identity. Use ssl_certificate_authority and ssl_mode for further customization. To use a client certificate to authenticate yourself to the server, use ssl_client_certificate_enable instead. Note if sqlalchemy_url is set this will be ignored. |
| ssl_client_certificate_enable | False | False | Whether or not to provide client-side certificates as a method of authentication to the server. Use ssl_client_certificate and ssl_client_private_key for further customization. To use SSL to verify the server's identity, use ssl_enable instead. Note if sqlalchemy_url is set this will be ignored. |
| ssl_mode | False | verify-full | SSL Protection method, see postgres documentation for more information. Must be one of disable, allow, prefer, require, verify-ca, or verify-full. Note if sqlalchemy_url is set this will be ignored. |
| ssl_certificate_authority | False | ~/.postgresql/root.crl | The certificate authority that should be used to verify the server's identity. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored. |
| ssl_client_certificate | False | ~/.postgresql/postgresql.crt | The certificate that should be used to verify your identity to the server. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored. |
| ssl_client_private_key | False | ~/.postgresql/postgresql.key | The private key for the certificate you provided. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored. |
| ssl_storage_directory | False | .secrets | The folder in which to store SSL certificates provided as raw values. When a certificate/key is provided as a raw value instead of as a filepath, it must be written to a file before it can be used. This configuration option determines where that file is created. |
| ssh_tunnel | False | None | SSH Tunnel Configuration, this is a json object |
| ssh_tunnel.enable | False | False | Enable an ssh tunnel (also known as bastion host), see the other ssh_tunnel.* properties for more details |
| ssh_tunnel.host | False | None | Host of the bastion host, this is the host we'll connect to via ssh |
| ssh_tunnel.username | False | None | Username to connect to bastion host |
| ssh_tunnel.port | False | 22 | Port to connect to bastion host |
| ssh_tunnel.private_key | False | None | Private Key for authentication to the bastion host |
| ssh_tunnel.private_key_password | False | None | Private Key Password, leave None if no password is set |
| load_method | False | append-only | The method to use when loading data into the destination. append-only will always write all input records whether that records already exists or not. upsert will update existing records and insert new records. overwrite will delete all existing records and insert all input records. |
| batch_size_rows | False | None | Maximum number of rows in each batch. |
| process_activate_version_messages | False | True | Whether to process ACTIVATE_VERSION messages. |
| validate_records | False | True | Whether to validate the schema of the incoming streams. |
| stream_maps | False | None | Config object for stream maps capability. For more information check out Stream Maps. |
| stream_maps.else | False | None | Currently, only setting this to __NULL__ is supported. This will remove all other streams. |
| stream_map_config | False | None | User-defined config values to be used within map expressions. |
| faker_config | False | None | Config for the Faker instance variable fake used within map expressions. Only applicable if the plugin specifies faker as an additional dependency (through the singer-sdk faker extra or directly). |
| faker_config.seed | False | None | Value to seed the Faker generator for deterministic output: https://faker.readthedocs.io/en/master/#seeding-the-generator |
| faker_config.locale | False | None | One or more LCID locale strings to produce localized output for: https://faker.readthedocs.io/en/master/#localization |
| flattening_enabled | False | None | 'True' to enable schema flattening and automatically expand nested properties. |
| flattening_max_depth | False | None | The max depth to flatten schemas. |
| flattening_max_key_length | False | None | The maximum length of a flattened key. |
A full list of supported settings and capabilities is available by running: target-cratedb --about
Version info
target-cratedb v0.6.3, Meltano SDK v0.53.6
Usage info
melty-bot % meltano invoke target-cratedb --help
Usage: target-cratedb [OPTIONS]
Execute the Singer target.
Options:
--version Display the package version.
--about Display package metadata and settings.
--format [json|markdown] Specify output style for --about
--config TEXT Configuration file location or 'ENV' to use
environment variables.
--input FILENAME A path to read messages from instead of from
standard in.
--help Show this message and exit.
Detected capabilities
- ❌ 'discover'
- ❌ 'catalog'
- ❌ 'properties'
- ❌ 'state'
- ✅ 'about'
JSON Metadata
{
"name": "target-cratedb",
"description": "Target for CrateDB.",
"version": "0.6.3",
"sdk_version": "0.53.6",
"supported_python_versions": [
"3.10",
"3.11",
"3.12",
"3.13",
"3.14"
],
"capabilities": [
"about",
"stream-maps",
"schema-flattening",
"structured-logging",
"validate-records",
"activate-version",
"target-schema",
"hard-delete"
],
"settings": {
"type": "object",
"properties": {
"host": {
"type": [
"string",
"null"
],
"description": "Hostname for postgres instance. Note if sqlalchemy_url is set this will be ignored."
},
"port": {
"type": [
"integer",
"null"
],
"default": 5432,
"description": "The port on which postgres is awaiting connection. Note if sqlalchemy_url is set this will be ignored."
},
"user": {
"type": [
"string",
"null"
],
"description": "User name used to authenticate. Note if sqlalchemy_url is set this will be ignored."
},
"password": {
"type": [
"string",
"null"
],
"description": "Password used to authenticate. Note if sqlalchemy_url is set this will be ignored."
},
"database": {
"type": [
"string",
"null"
],
"description": "Database name. Note if sqlalchemy_url is set this will be ignored."
},
"use_copy": {
"type": [
"boolean",
"null"
],
"title": "Use COPY",
"default": false,
"description": "Use the COPY command to insert data. This is usually faster than INSERT statements. This option is only available for the postgresql+psycopg dialect+driver."
},
"sqlalchemy_url": {
"type": [
"string",
"null"
],
"description": "SQLAlchemy connection string. This will override using host, user, password, port, dialect, and all ssl settings. It is strongly recommended the driver is compatible with psycopg3. Note that you must escape password special characters properly. See https://docs.sqlalchemy.org/en/20/core/engines.html#escaping-special-characters-such-as-signs-in-passwords"
},
"dialect+driver": {
"type": [
"string",
"null"
],
"default": "postgresql+psycopg",
"description": "DEPRECATED. Dialect+driver see https://docs.sqlalchemy.org/en/20/core/engines.html. Generally just leave this alone."
},
"default_target_schema": {
"type": [
"string",
"null"
],
"default": "melty",
"description": "Postgres schema to send data to, example: tap-clickup"
},
"activate_version": {
"type": [
"boolean",
"null"
],
"default": true,
"description": "If set to false, the tap will ignore activate version messages. If set to true, add_record_metadata must be set to true as well."
},
"hard_delete": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "When activate version is sent from a tap this specifies if we should delete the records that don't match, or mark them with a date in the `_sdc_deleted_at` column. This config option is ignored if `activate_version` is set to false."
},
"add_record_metadata": {
"type": [
"boolean",
"null"
],
"default": true,
"description": "Note that this must be enabled for activate_version to work!This adds _sdc_extracted_at, _sdc_batched_at, and more to every table. See https://sdk.meltano.com/en/latest/implementation/record_metadata.html for more information."
},
"interpret_content_encoding": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "If set to true, the target will interpret the content encoding of the schema to determine how to store the data. Using this option may result in a more efficient storage of the data but may also result in an error if the data is not encoded as expected."
},
"sanitize_null_text_characters": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "If set to true, the target will sanitize null characters in char/text/varchar fields, as they are not supported by Postgres. See [postgres documentation](https://www.postgresql.org/docs/current/functions-string.html) for more information about chr(0) not being supported."
},
"ssl_enable": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Whether or not to use ssl to verify the server's identity. Use ssl_certificate_authority and ssl_mode for further customization. To use a client certificate to authenticate yourself to the server, use ssl_client_certificate_enable instead. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_client_certificate_enable": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Whether or not to provide client-side certificates as a method of authentication to the server. Use ssl_client_certificate and ssl_client_private_key for further customization. To use SSL to verify the server's identity, use ssl_enable instead. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_mode": {
"type": [
"string",
"null"
],
"default": "verify-full",
"description": "SSL Protection method, see [postgres documentation](https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-PROTECTION) for more information. Must be one of disable, allow, prefer, require, verify-ca, or verify-full. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_certificate_authority": {
"type": [
"string",
"null"
],
"default": "~/.postgresql/root.crl",
"description": "The certificate authority that should be used to verify the server's identity. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_client_certificate": {
"type": [
"string",
"null"
],
"default": "~/.postgresql/postgresql.crt",
"description": "The certificate that should be used to verify your identity to the server. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_client_private_key": {
"type": [
"string",
"null"
],
"default": "~/.postgresql/postgresql.key",
"description": "The private key for the certificate you provided. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_storage_directory": {
"type": [
"string",
"null"
],
"default": ".secrets",
"description": "The folder in which to store SSL certificates provided as raw values. When a certificate/key is provided as a raw value instead of as a filepath, it must be written to a file before it can be used. This configuration option determines where that file is created."
},
"ssh_tunnel": {
"type": [
"object",
"null"
],
"properties": {
"enable": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Enable an ssh tunnel (also known as bastion host), see the other ssh_tunnel.* properties for more details"
},
"host": {
"type": [
"string",
"null"
],
"description": "Host of the bastion host, this is the host we'll connect to via ssh"
},
"username": {
"type": [
"string",
"null"
],
"description": "Username to connect to bastion host"
},
"port": {
"type": [
"integer",
"null"
],
"default": 22,
"description": "Port to connect to bastion host"
},
"private_key": {
"type": [
"string",
"null"
],
"description": "Private Key for authentication to the bastion host",
"secret": true,
"writeOnly": true
},
"private_key_password": {
"type": [
"string",
"null"
],
"description": "Private Key Password, leave None if no password is set",
"secret": true,
"writeOnly": true
}
},
"additionalProperties": true,
"description": "SSH Tunnel Configuration, this is a json object"
},
"load_method": {
"type": [
"string",
"null"
],
"default": "append-only",
"description": "The method to use when loading data into the destination. `append-only` will always write all input records whether that records already exists or not. `upsert` will update existing records and insert new records. `overwrite` will delete all existing records and insert all input records.",
"enum": [
"append-only",
"upsert",
"overwrite"
]
},
"batch_size_rows": {
"type": [
"integer",
"null"
],
"title": "Batch Size Rows",
"description": "Maximum number of rows in each batch."
},
"process_activate_version_messages": {
"type": [
"boolean",
"null"
],
"title": "Process `ACTIVATE_VERSION` messages",
"default": true,
"description": "Whether to process `ACTIVATE_VERSION` messages."
},
"validate_records": {
"type": [
"boolean",
"null"
],
"title": "Validate Records",
"default": true,
"description": "Whether to validate the schema of the incoming streams."
},
"stream_maps": {
"type": [
"object",
"null"
],
"properties": {
"__else__": {
"anyOf": [
{
"const": "__NULL__"
},
{
"type": "null"
},
"null"
],
"title": "Other streams",
"description": "Currently, only setting this to `__NULL__` is supported. This will remove all other streams."
}
},
"additionalProperties": {
"anyOf": [
{
"const": "__NULL__"
},
{
"type": "null"
},
{
"type": "object",
"properties": {
"__alias__": {
"type": [
"string",
"null"
],
"title": "Stream Alias",
"description": "Alias to use for the stream."
},
"__else__": {
"anyOf": [
{
"const": "__NULL__"
},
{
"type": "null"
},
"null"
],
"title": "Other properties",
"description": "Currently, only setting this to `__NULL__` is supported. This will remove all other properties from the stream."
},
"__filter__": {
"type": [
"string",
"null"
],
"title": "Filter",
"description": "Filter out records from a stream. A string expression which must evaluate to `true` to include the record, or `false` to exclude it. Filter expressions should be wrapped in `bool()` to ensure proper type conversion."
},
"__key_properties__": {
"type": [
"array",
"null"
],
"items": {
"type": [
"string"
]
},
"title": "Key Properties",
"description": "Primary key properties for the stream."
},
"__source__": {
"type": [
"string",
"null"
],
"description": "Create a new stream from this source stream."
}
},
"additionalProperties": {
"anyOf": [
{
"const": "__NULL__"
},
{
"type": "null"
},
{
"type": [
"string"
]
}
]
}
}
]
},
"title": "Stream Maps",
"description": "Config object for stream maps capability. For more information check out [Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html)."
},
"stream_map_config": {
"type": [
"object",
"null"
],
"properties": {},
"additionalProperties": true,
"title": "User Stream Map Configuration",
"description": "User-defined config values to be used within map expressions."
},
"faker_config": {
"type": [
"object",
"null"
],
"properties": {
"seed": {
"oneOf": [
{
"type": [
"number"
]
},
{
"type": [
"string"
]
},
{
"type": [
"boolean"
]
},
{
"type": "null"
}
],
"title": "Faker Seed",
"description": "Value to seed the Faker generator for deterministic output: https://faker.readthedocs.io/en/master/#seeding-the-generator"
},
"locale": {
"oneOf": [
{
"type": [
"string"
]
},
{
"type": "array",
"items": {
"type": [
"string"
]
}
},
{
"type": "null"
}
],
"title": "Faker Locale",
"description": "One or more LCID locale strings to produce localized output for: https://faker.readthedocs.io/en/master/#localization"
}
},
"additionalProperties": true,
"title": "Faker Configuration",
"description": "Config for the [`Faker`](https://faker.readthedocs.io/en/master/) instance variable `fake` used within map expressions. Only applicable if the plugin specifies `faker` as an additional dependency (through the `singer-sdk` `faker` extra or directly)."
},
"flattening_enabled": {
"type": [
"boolean",
"null"
],
"title": "Enable Schema Flattening",
"description": "'True' to enable schema flattening and automatically expand nested properties."
},
"flattening_max_depth": {
"type": [
"integer",
"null"
],
"title": "Max Flattening Depth",
"description": "The max depth to flatten schemas."
},
"flattening_max_key_length": {
"type": [
"integer",
"null"
],
"title": "Max Key Length",
"description": "The maximum length of a flattened key."
}
},
"$schema": "https://json-schema.org/draft/2020-12/schema"
}
}
|
Testing plugin
Auto-generated README.md
|
| Setting | Required | Default | Description |
|---|---|---|---|
| replication_slot_name | False | tappostgres | Name of the replication slot to use for logical replication. Must be unique for parallel extractions. Only applicable when replication_method is LOG_BASED.- Contain only letters, numbers, and underscores. - Be less than or equal to 63 characters. - Not start with 'pg_'. |
| replication_max_run_seconds | False | 600 | The maximum number of seconds to run the replication loop for. Only applicable when replication_method is LOG_BASED. |
| replication_idle_exit_seconds | False | 60 | The number of seconds to wait for data messages before exiting the replication loop. Only applicable when replication_method is LOG_BASED. |
| host | False | None | Hostname for postgres instance. Note if sqlalchemy_url is set this will be ignored. |
| port | False | 5432 | The port on which postgres is awaiting connection. Note if sqlalchemy_url is set this will be ignored. |
| user | False | None | User name used to authenticate. Note if sqlalchemy_url is set this will be ignored. |
| password | False | None | Password used to authenticate. Note if sqlalchemy_url is set this will be ignored. |
| database | False | None | Database name. Note if sqlalchemy_url is set this will be ignored. |
| max_record_count | False | None | Optional. The maximum number of records to return in a single stream. |
| sqlalchemy_url | False | None | Example postgresql://[username]:[password]@localhost:5432/[db_name] |
| filter_schemas | False | None | If an array of schema names is provided, the tap will only process the specified Postgres schemas and ignore others. If left blank, the tap automatically determines ALL available Postgres schemas. |
| dates_as_string | False | False | Defaults to false, if true, date, and timestamp fields will be Strings. If you see ValueError: Year is out of range, try setting this to True. |
| json_as_object | False | False | Defaults to false, if true, json and jsonb fields will be Objects. |
| stream_options | False | None | |
| ssh_tunnel | False | None | SSH Tunnel Configuration, this is a json object |
| ssh_tunnel.enable | False | False | Enable an ssh tunnel (also known as bastion server), see the other ssh_tunnel.* properties for more details |
| ssh_tunnel.host | False | None | Host of the bastion server, this is the host we'll connect to via ssh |
| ssh_tunnel.username | False | None | Username to connect to bastion server |
| ssh_tunnel.port | False | 22 | Port to connect to bastion server |
| ssh_tunnel.private_key | False | None | Private Key for authentication to the bastion server |
| ssh_tunnel.private_key_password | False | None | Private Key Password, leave None if no password is set |
| ssl_enable | False | False | Whether or not to use ssl to verify the server's identity. Use ssl_certificate_authority and ssl_mode for further customization. To use a client certificate to authenticate yourself to the server, use ssl_client_certificate_enable instead. Note if sqlalchemy_url is set this will be ignored. |
| ssl_client_certificate_enable | False | False | Whether or not to provide client-side certificates as a method of authentication to the server. Use ssl_client_certificate and ssl_client_private_key for further customization. To use SSL to verify the server's identity, use ssl_enable instead. Note if sqlalchemy_url is set this will be ignored. |
| ssl_mode | False | verify-full | SSL Protection method, see postgres documentation for more information. Must be one of disable, allow, prefer, require, verify-ca, or verify-full. Note if sqlalchemy_url is set this will be ignored. |
| ssl_certificate_authority | False | ~/.postgresql/root.crl | The certificate authority that should be used to verify the server's identity. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored. |
| ssl_client_certificate | False | ~/.postgresql/postgresql.crt | The certificate that should be used to verify your identity to the server. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored. |
| ssl_client_private_key | False | ~/.postgresql/postgresql.key | The private key for the certificate you provided. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored. |
| ssl_storage_directory | False | .secrets | The folder in which to store SSL certificates provided as raw values. When a certificate/key is provided as a raw value instead of as a filepath, it must be written to a file before it can be used. This configuration option determines where that file is created. |
| default_replication_method | False | FULL_TABLE | Replication method to use if there is not a catalog entry to override this choice. One of FULL_TABLE, INCREMENTAL, or LOG_BASED. |
| use_singer_decimal | False | None | Whether to use use strings with x-singer.decimal format for decimals in the discovered schema. This is useful to avoid precision loss when working with large numbers. |
| stream_maps | False | None | Config object for stream maps capability. For more information check out Stream Maps. |
| stream_maps.else | False | None | Currently, only setting this to __NULL__ is supported. This will remove all other streams. |
| stream_map_config | False | None | User-defined config values to be used within map expressions. |
| faker_config | False | None | Config for the Faker instance variable fake used within map expressions. Only applicable if the plugin specifies faker as an additional dependency (through the singer-sdk faker extra or directly). |
| faker_config.seed | False | None | Value to seed the Faker generator for deterministic output: https://faker.readthedocs.io/en/master/#seeding-the-generator |
| faker_config.locale | False | None | One or more LCID locale strings to produce localized output for: https://faker.readthedocs.io/en/master/#localization |
| flattening_enabled | False | None | 'True' to enable schema flattening and automatically expand nested properties. |
| flattening_max_depth | False | None | The max depth to flatten schemas. |
| flattening_max_key_length | False | None | The maximum length of a flattened key. |
| batch_config | False | None | Configuration for BATCH message capabilities. |
| batch_config.encoding | False | None | Specifies the format and compression of the batch files. |
| batch_config.encoding.format | False | None | Format to use for batch files. |
| batch_config.encoding.compression | False | None | Compression format to use for batch files. |
| batch_config.storage | False | None | Defines the storage layer to use when writing batch files |
| batch_config.storage.root | False | None | Root path to use when writing batch files. |
| batch_config.storage.prefix | False | None | Prefix to use when writing batch files. |
A full list of supported settings and capabilities is available by running: tap-cratedb --about
Version info
tap-cratedb v0.0.1, Meltano SDK v0.53.6
Usage info
melty-bot % meltano invoke tap-cratedb --help
Usage: tap-cratedb [OPTIONS]
Execute the Singer tap.
Options:
--version Display the package version.
--about Display package metadata and settings.
--format [json|markdown] Specify output style for --about
--config TEXT Configuration file location or 'ENV' to use
environment variables.
--discover Run the tap in discovery mode.
--test TEXT Use --test to sync a single record for each
stream. Use --test=schema to test schema output
without syncing records.
--catalog FILENAME Use a Singer catalog file with the tap.
--state FILENAME Use a bookmarks file for incremental replication.
--help Show this message and exit.
Detected capabilities
- ✅ 'discover'
- ✅ 'catalog'
- ✅ 'state'
- ✅ 'about'
JSON Metadata
{
"name": "tap-cratedb",
"description": null,
"version": "0.0.1",
"sdk_version": "0.53.6",
"supported_python_versions": [
"3.9",
"3.10",
"3.11",
"3.12",
"3.13",
"3.14"
],
"capabilities": [
"catalog",
"state",
"discover",
"activate-version",
"about",
"stream-maps",
"schema-flattening",
"batch",
"structured-logging"
],
"settings": {
"type": "object",
"properties": {
"replication_slot_name": {
"type": [
"string",
"null"
],
"pattern": "^(?!pg_)[A-Za-z0-9_]{1,63}$",
"default": "tappostgres",
"description": "Name of the replication slot to use for logical replication. Must be unique for parallel extractions. Only applicable when replication_method is LOG_BASED.- Contain only letters, numbers, and underscores. - Be less than or equal to 63 characters. - Not start with 'pg_'."
},
"replication_max_run_seconds": {
"type": [
"integer",
"null"
],
"default": 600,
"description": "The maximum number of seconds to run the replication loop for. Only applicable when replication_method is LOG_BASED."
},
"replication_idle_exit_seconds": {
"type": [
"integer",
"null"
],
"default": 60,
"description": "The number of seconds to wait for data messages before exiting the replication loop. Only applicable when replication_method is LOG_BASED."
},
"host": {
"type": [
"string",
"null"
],
"description": "Hostname for postgres instance. Note if sqlalchemy_url is set this will be ignored."
},
"port": {
"type": [
"integer",
"null"
],
"default": 5432,
"description": "The port on which postgres is awaiting connection. Note if sqlalchemy_url is set this will be ignored."
},
"user": {
"type": [
"string",
"null"
],
"description": "User name used to authenticate. Note if sqlalchemy_url is set this will be ignored."
},
"password": {
"type": [
"string",
"null"
],
"description": "Password used to authenticate. Note if sqlalchemy_url is set this will be ignored.",
"secret": true,
"writeOnly": true
},
"database": {
"type": [
"string",
"null"
],
"description": "Database name. Note if sqlalchemy_url is set this will be ignored."
},
"max_record_count": {
"type": [
"integer",
"null"
],
"description": "Optional. The maximum number of records to return in a single stream."
},
"sqlalchemy_url": {
"type": [
"string",
"null"
],
"description": "Example postgresql://[username]:[password]@localhost:5432/[db_name]",
"secret": true,
"writeOnly": true
},
"filter_schemas": {
"type": [
"array",
"null"
],
"items": {
"type": [
"string"
]
},
"description": "If an array of schema names is provided, the tap will only process the specified Postgres schemas and ignore others. If left blank, the tap automatically determines ALL available Postgres schemas."
},
"dates_as_string": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Defaults to false, if true, date, and timestamp fields will be Strings. If you see ValueError: Year is out of range, try setting this to True."
},
"json_as_object": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Defaults to false, if true, json and jsonb fields will be Objects."
},
"stream_options": {
"type": [
"object",
"null"
],
"properties": {},
"additionalProperties": {
"type": "object",
"properties": {
"custom_where_clauses": {
"type": [
"array",
"null"
],
"items": {
"type": [
"string"
]
},
"default": [],
"description": "If an array of custom WHERE clauses is provided, the tap will only process the records that match the WHERE clauses. The WHERE clauses are combined using the AND operator."
}
},
"additionalProperties": true
}
},
"ssh_tunnel": {
"type": [
"object",
"null"
],
"properties": {
"enable": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Enable an ssh tunnel (also known as bastion server), see the other ssh_tunnel.* properties for more details"
},
"host": {
"type": [
"string",
"null"
],
"description": "Host of the bastion server, this is the host we'll connect to via ssh"
},
"username": {
"type": [
"string",
"null"
],
"description": "Username to connect to bastion server"
},
"port": {
"type": [
"integer",
"null"
],
"default": 22,
"description": "Port to connect to bastion server"
},
"private_key": {
"type": [
"string",
"null"
],
"description": "Private Key for authentication to the bastion server",
"secret": true,
"writeOnly": true
},
"private_key_password": {
"type": [
"string",
"null"
],
"description": "Private Key Password, leave None if no password is set",
"secret": true,
"writeOnly": true
}
},
"additionalProperties": true,
"description": "SSH Tunnel Configuration, this is a json object"
},
"ssl_enable": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Whether or not to use ssl to verify the server's identity. Use ssl_certificate_authority and ssl_mode for further customization. To use a client certificate to authenticate yourself to the server, use ssl_client_certificate_enable instead. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_client_certificate_enable": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Whether or not to provide client-side certificates as a method of authentication to the server. Use ssl_client_certificate and ssl_client_private_key for further customization. To use SSL to verify the server's identity, use ssl_enable instead. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_mode": {
"type": [
"string",
"null"
],
"default": "verify-full",
"description": "SSL Protection method, see [postgres documentation](https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-PROTECTION) for more information. Must be one of disable, allow, prefer, require, verify-ca, or verify-full. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_certificate_authority": {
"type": [
"string",
"null"
],
"default": "~/.postgresql/root.crl",
"description": "The certificate authority that should be used to verify the server's identity. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_client_certificate": {
"type": [
"string",
"null"
],
"default": "~/.postgresql/postgresql.crt",
"description": "The certificate that should be used to verify your identity to the server. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_client_private_key": {
"type": [
"string",
"null"
],
"default": "~/.postgresql/postgresql.key",
"description": "The private key for the certificate you provided. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_storage_directory": {
"type": [
"string",
"null"
],
"default": ".secrets",
"description": "The folder in which to store SSL certificates provided as raw values. When a certificate/key is provided as a raw value instead of as a filepath, it must be written to a file before it can be used. This configuration option determines where that file is created."
},
"default_replication_method": {
"type": [
"string",
"null"
],
"default": "FULL_TABLE",
"description": "Replication method to use if there is not a catalog entry to override this choice. One of `FULL_TABLE`, `INCREMENTAL`, or `LOG_BASED`.",
"enum": [
"FULL_TABLE",
"INCREMENTAL",
"LOG_BASED"
]
},
"use_singer_decimal": {
"type": [
"boolean",
"null"
],
"title": "Use Singer Decimal",
"description": "Whether to use use strings with `x-singer.decimal` format for decimals in the discovered schema. This is useful to avoid precision loss when working with large numbers."
},
"stream_maps": {
"type": [
"object",
"null"
],
"properties": {
"__else__": {
"anyOf": [
{
"const": "__NULL__"
},
{
"type": "null"
},
"null"
],
"title": "Other streams",
"description": "Currently, only setting this to `__NULL__` is supported. This will remove all other streams."
}
},
"additionalProperties": {
"anyOf": [
{
"const": "__NULL__"
},
{
"type": "null"
},
{
"type": "object",
"properties": {
"__alias__": {
"type": [
"string",
"null"
],
"title": "Stream Alias",
"description": "Alias to use for the stream."
},
"__else__": {
"anyOf": [
{
"const": "__NULL__"
},
{
"type": "null"
},
"null"
],
"title": "Other properties",
"description": "Currently, only setting this to `__NULL__` is supported. This will remove all other properties from the stream."
},
"__filter__": {
"type": [
"string",
"null"
],
"title": "Filter",
"description": "Filter out records from a stream. A string expression which must evaluate to `true` to include the record, or `false` to exclude it. Filter expressions should be wrapped in `bool()` to ensure proper type conversion."
},
"__key_properties__": {
"type": [
"array",
"null"
],
"items": {
"type": [
"string"
]
},
"title": "Key Properties",
"description": "Primary key properties for the stream."
},
"__source__": {
"type": [
"string",
"null"
],
"description": "Create a new stream from this source stream."
}
},
"additionalProperties": {
"anyOf": [
{
"const": "__NULL__"
},
{
"type": "null"
},
{
"type": [
"string"
]
}
]
}
}
]
},
"title": "Stream Maps",
"description": "Config object for stream maps capability. For more information check out [Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html)."
},
"stream_map_config": {
"type": [
"object",
"null"
],
"properties": {},
"additionalProperties": true,
"title": "User Stream Map Configuration",
"description": "User-defined config values to be used within map expressions."
},
"faker_config": {
"type": [
"object",
"null"
],
"properties": {
"seed": {
"oneOf": [
{
"type": [
"number"
]
},
{
"type": [
"string"
]
},
{
"type": [
"boolean"
]
},
{
"type": "null"
}
],
"title": "Faker Seed",
"description": "Value to seed the Faker generator for deterministic output: https://faker.readthedocs.io/en/master/#seeding-the-generator"
},
"locale": {
"oneOf": [
{
"type": [
"string"
]
},
{
"type": "array",
"items": {
"type": [
"string"
]
}
},
{
"type": "null"
}
],
"title": "Faker Locale",
"description": "One or more LCID locale strings to produce localized output for: https://faker.readthedocs.io/en/master/#localization"
}
},
"additionalProperties": true,
"title": "Faker Configuration",
"description": "Config for the [`Faker`](https://faker.readthedocs.io/en/master/) instance variable `fake` used within map expressions. Only applicable if the plugin specifies `faker` as an additional dependency (through the `singer-sdk` `faker` extra or directly)."
},
"flattening_enabled": {
"type": [
"boolean",
"null"
],
"title": "Enable Schema Flattening",
"description": "'True' to enable schema flattening and automatically expand nested properties."
},
"flattening_max_depth": {
"type": [
"integer",
"null"
],
"title": "Max Flattening Depth",
"description": "The max depth to flatten schemas."
},
"flattening_max_key_length": {
"type": [
"integer",
"null"
],
"title": "Max Key Length",
"description": "The maximum length of a flattened key."
},
"batch_config": {
"type": [
"object",
"null"
],
"properties": {
"encoding": {
"type": [
"object",
"null"
],
"properties": {
"format": {
"type": [
"string",
"null"
],
"title": "Batch Encoding Format",
"description": "Format to use for batch files.",
"enum": [
"jsonl",
"parquet"
]
},
"compression": {
"type": [
"string",
"null"
],
"title": "Batch Compression Format",
"description": "Compression format to use for batch files.",
"enum": [
"gzip",
"none"
]
}
},
"additionalProperties": true,
"title": "Batch Encoding Configuration",
"description": "Specifies the format and compression of the batch files."
},
"storage": {
"type": [
"object",
"null"
],
"properties": {
"root": {
"type": [
"string",
"null"
],
"title": "Batch Storage Root",
"description": "Root path to use when writing batch files."
},
"prefix": {
"type": [
"string",
"null"
],
"title": "Batch Storage Prefix",
"description": "Prefix to use when writing batch files."
}
},
"additionalProperties": true,
"title": "Batch Storage Configuration",
"description": "Defines the storage layer to use when writing batch files"
}
},
"additionalProperties": true,
"title": "Batch Configuration",
"description": "Configuration for BATCH message capabilities."
}
},
"$schema": "https://json-schema.org/draft/2020-12/schema"
}
}Auto-generated README.md
tap-cratedb
None
Built with the Meltano Singer SDK.
Capabilities
catalogstatediscoveractivate-versionaboutstream-mapsschema-flatteningbatchstructured-logging
Supported Python Versions
- 3.9
- 3.10
- 3.11
- 3.12
- 3.13
- 3.14
Settings
| Setting | Required | Default | Description |
|---|---|---|---|
| replication_slot_name | False | tappostgres | Name of the replication slot to use for logical replication. Must be unique for parallel extractions. Only applicable when replication_method is LOG_BASED.- Contain only letters, numbers, and underscores. - Be less than or equal to 63 characters. - Not start with 'pg_'. |
| replication_max_run_seconds | False | 600 | The maximum number of seconds to run the replication loop for. Only applicable when replication_method is LOG_BASED. |
| replication_idle_exit_seconds | False | 60 | The number of seconds to wait for data messages before exiting the replication loop. Only applicable when replication_method is LOG_BASED. |
| host | False | None | Hostname for postgres instance. Note if sqlalchemy_url is set this will be ignored. |
| port | False | 5432 | The port on which postgres is awaiting connection. Note if sqlalchemy_url is set this will be ignored. |
| user | False | None | User name used to authenticate. Note if sqlalchemy_url is set this will be ignored. |
| password | False | None | Password used to authenticate. Note if sqlalchemy_url is set this will be ignored. |
| database | False | None | Database name. Note if sqlalchemy_url is set this will be ignored. |
| max_record_count | False | None | Optional. The maximum number of records to return in a single stream. |
| sqlalchemy_url | False | None | Example postgresql://[username]:[password]@localhost:5432/[db_name] |
| filter_schemas | False | None | If an array of schema names is provided, the tap will only process the specified Postgres schemas and ignore others. If left blank, the tap automatically determines ALL available Postgres schemas. |
| dates_as_string | False | False | Defaults to false, if true, date, and timestamp fields will be Strings. If you see ValueError: Year is out of range, try setting this to True. |
| json_as_object | False | False | Defaults to false, if true, json and jsonb fields will be Objects. |
| stream_options | False | None | |
| ssh_tunnel | False | None | SSH Tunnel Configuration, this is a json object |
| ssh_tunnel.enable | False | False | Enable an ssh tunnel (also known as bastion server), see the other ssh_tunnel.* properties for more details |
| ssh_tunnel.host | False | None | Host of the bastion server, this is the host we'll connect to via ssh |
| ssh_tunnel.username | False | None | Username to connect to bastion server |
| ssh_tunnel.port | False | 22 | Port to connect to bastion server |
| ssh_tunnel.private_key | False | None | Private Key for authentication to the bastion server |
| ssh_tunnel.private_key_password | False | None | Private Key Password, leave None if no password is set |
| ssl_enable | False | False | Whether or not to use ssl to verify the server's identity. Use ssl_certificate_authority and ssl_mode for further customization. To use a client certificate to authenticate yourself to the server, use ssl_client_certificate_enable instead. Note if sqlalchemy_url is set this will be ignored. |
| ssl_client_certificate_enable | False | False | Whether or not to provide client-side certificates as a method of authentication to the server. Use ssl_client_certificate and ssl_client_private_key for further customization. To use SSL to verify the server's identity, use ssl_enable instead. Note if sqlalchemy_url is set this will be ignored. |
| ssl_mode | False | verify-full | SSL Protection method, see postgres documentation for more information. Must be one of disable, allow, prefer, require, verify-ca, or verify-full. Note if sqlalchemy_url is set this will be ignored. |
| ssl_certificate_authority | False | ~/.postgresql/root.crl | The certificate authority that should be used to verify the server's identity. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored. |
| ssl_client_certificate | False | ~/.postgresql/postgresql.crt | The certificate that should be used to verify your identity to the server. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored. |
| ssl_client_private_key | False | ~/.postgresql/postgresql.key | The private key for the certificate you provided. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored. |
| ssl_storage_directory | False | .secrets | The folder in which to store SSL certificates provided as raw values. When a certificate/key is provided as a raw value instead of as a filepath, it must be written to a file before it can be used. This configuration option determines where that file is created. |
| default_replication_method | False | FULL_TABLE | Replication method to use if there is not a catalog entry to override this choice. One of FULL_TABLE, INCREMENTAL, or LOG_BASED. |
| use_singer_decimal | False | None | Whether to use use strings with x-singer.decimal format for decimals in the discovered schema. This is useful to avoid precision loss when working with large numbers. |
| stream_maps | False | None | Config object for stream maps capability. For more information check out Stream Maps. |
| stream_maps.else | False | None | Currently, only setting this to __NULL__ is supported. This will remove all other streams. |
| stream_map_config | False | None | User-defined config values to be used within map expressions. |
| faker_config | False | None | Config for the Faker instance variable fake used within map expressions. Only applicable if the plugin specifies faker as an additional dependency (through the singer-sdk faker extra or directly). |
| faker_config.seed | False | None | Value to seed the Faker generator for deterministic output: https://faker.readthedocs.io/en/master/#seeding-the-generator |
| faker_config.locale | False | None | One or more LCID locale strings to produce localized output for: https://faker.readthedocs.io/en/master/#localization |
| flattening_enabled | False | None | 'True' to enable schema flattening and automatically expand nested properties. |
| flattening_max_depth | False | None | The max depth to flatten schemas. |
| flattening_max_key_length | False | None | The maximum length of a flattened key. |
| batch_config | False | None | Configuration for BATCH message capabilities. |
| batch_config.encoding | False | None | Specifies the format and compression of the batch files. |
| batch_config.encoding.format | False | None | Format to use for batch files. |
| batch_config.encoding.compression | False | None | Compression format to use for batch files. |
| batch_config.storage | False | None | Defines the storage layer to use when writing batch files |
| batch_config.storage.root | False | None | Root path to use when writing batch files. |
| batch_config.storage.prefix | False | None | Prefix to use when writing batch files. |
A full list of supported settings and capabilities is available by running: tap-cratedb --about
Version info
tap-cratedb v0.0.1, Meltano SDK v0.53.6
Usage info
melty-bot % meltano invoke tap-cratedb --help
Usage: tap-cratedb [OPTIONS]
Execute the Singer tap.
Options:
--version Display the package version.
--about Display package metadata and settings.
--format [json|markdown] Specify output style for --about
--config TEXT Configuration file location or 'ENV' to use
environment variables.
--discover Run the tap in discovery mode.
--test TEXT Use --test to sync a single record for each
stream. Use --test=schema to test schema output
without syncing records.
--catalog FILENAME Use a Singer catalog file with the tap.
--state FILENAME Use a bookmarks file for incremental replication.
--help Show this message and exit.
Detected capabilities
- ✅ 'discover'
- ✅ 'catalog'
- ✅ 'state'
- ✅ 'about'
JSON Metadata
{
"name": "tap-cratedb",
"description": null,
"version": "0.0.1",
"sdk_version": "0.53.6",
"supported_python_versions": [
"3.9",
"3.10",
"3.11",
"3.12",
"3.13",
"3.14"
],
"capabilities": [
"catalog",
"state",
"discover",
"activate-version",
"about",
"stream-maps",
"schema-flattening",
"batch",
"structured-logging"
],
"settings": {
"type": "object",
"properties": {
"replication_slot_name": {
"type": [
"string",
"null"
],
"pattern": "^(?!pg_)[A-Za-z0-9_]{1,63}$",
"default": "tappostgres",
"description": "Name of the replication slot to use for logical replication. Must be unique for parallel extractions. Only applicable when replication_method is LOG_BASED.- Contain only letters, numbers, and underscores. - Be less than or equal to 63 characters. - Not start with 'pg_'."
},
"replication_max_run_seconds": {
"type": [
"integer",
"null"
],
"default": 600,
"description": "The maximum number of seconds to run the replication loop for. Only applicable when replication_method is LOG_BASED."
},
"replication_idle_exit_seconds": {
"type": [
"integer",
"null"
],
"default": 60,
"description": "The number of seconds to wait for data messages before exiting the replication loop. Only applicable when replication_method is LOG_BASED."
},
"host": {
"type": [
"string",
"null"
],
"description": "Hostname for postgres instance. Note if sqlalchemy_url is set this will be ignored."
},
"port": {
"type": [
"integer",
"null"
],
"default": 5432,
"description": "The port on which postgres is awaiting connection. Note if sqlalchemy_url is set this will be ignored."
},
"user": {
"type": [
"string",
"null"
],
"description": "User name used to authenticate. Note if sqlalchemy_url is set this will be ignored."
},
"password": {
"type": [
"string",
"null"
],
"description": "Password used to authenticate. Note if sqlalchemy_url is set this will be ignored.",
"secret": true,
"writeOnly": true
},
"database": {
"type": [
"string",
"null"
],
"description": "Database name. Note if sqlalchemy_url is set this will be ignored."
},
"max_record_count": {
"type": [
"integer",
"null"
],
"description": "Optional. The maximum number of records to return in a single stream."
},
"sqlalchemy_url": {
"type": [
"string",
"null"
],
"description": "Example postgresql://[username]:[password]@localhost:5432/[db_name]",
"secret": true,
"writeOnly": true
},
"filter_schemas": {
"type": [
"array",
"null"
],
"items": {
"type": [
"string"
]
},
"description": "If an array of schema names is provided, the tap will only process the specified Postgres schemas and ignore others. If left blank, the tap automatically determines ALL available Postgres schemas."
},
"dates_as_string": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Defaults to false, if true, date, and timestamp fields will be Strings. If you see ValueError: Year is out of range, try setting this to True."
},
"json_as_object": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Defaults to false, if true, json and jsonb fields will be Objects."
},
"stream_options": {
"type": [
"object",
"null"
],
"properties": {},
"additionalProperties": {
"type": "object",
"properties": {
"custom_where_clauses": {
"type": [
"array",
"null"
],
"items": {
"type": [
"string"
]
},
"default": [],
"description": "If an array of custom WHERE clauses is provided, the tap will only process the records that match the WHERE clauses. The WHERE clauses are combined using the AND operator."
}
},
"additionalProperties": true
}
},
"ssh_tunnel": {
"type": [
"object",
"null"
],
"properties": {
"enable": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Enable an ssh tunnel (also known as bastion server), see the other ssh_tunnel.* properties for more details"
},
"host": {
"type": [
"string",
"null"
],
"description": "Host of the bastion server, this is the host we'll connect to via ssh"
},
"username": {
"type": [
"string",
"null"
],
"description": "Username to connect to bastion server"
},
"port": {
"type": [
"integer",
"null"
],
"default": 22,
"description": "Port to connect to bastion server"
},
"private_key": {
"type": [
"string",
"null"
],
"description": "Private Key for authentication to the bastion server",
"secret": true,
"writeOnly": true
},
"private_key_password": {
"type": [
"string",
"null"
],
"description": "Private Key Password, leave None if no password is set",
"secret": true,
"writeOnly": true
}
},
"additionalProperties": true,
"description": "SSH Tunnel Configuration, this is a json object"
},
"ssl_enable": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Whether or not to use ssl to verify the server's identity. Use ssl_certificate_authority and ssl_mode for further customization. To use a client certificate to authenticate yourself to the server, use ssl_client_certificate_enable instead. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_client_certificate_enable": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Whether or not to provide client-side certificates as a method of authentication to the server. Use ssl_client_certificate and ssl_client_private_key for further customization. To use SSL to verify the server's identity, use ssl_enable instead. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_mode": {
"type": [
"string",
"null"
],
"default": "verify-full",
"description": "SSL Protection method, see [postgres documentation](https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-PROTECTION) for more information. Must be one of disable, allow, prefer, require, verify-ca, or verify-full. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_certificate_authority": {
"type": [
"string",
"null"
],
"default": "~/.postgresql/root.crl",
"description": "The certificate authority that should be used to verify the server's identity. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_client_certificate": {
"type": [
"string",
"null"
],
"default": "~/.postgresql/postgresql.crt",
"description": "The certificate that should be used to verify your identity to the server. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_client_private_key": {
"type": [
"string",
"null"
],
"default": "~/.postgresql/postgresql.key",
"description": "The private key for the certificate you provided. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_storage_directory": {
"type": [
"string",
"null"
],
"default": ".secrets",
"description": "The folder in which to store SSL certificates provided as raw values. When a certificate/key is provided as a raw value instead of as a filepath, it must be written to a file before it can be used. This configuration option determines where that file is created."
},
"default_replication_method": {
"type": [
"string",
"null"
],
"default": "FULL_TABLE",
"description": "Replication method to use if there is not a catalog entry to override this choice. One of `FULL_TABLE`, `INCREMENTAL`, or `LOG_BASED`.",
"enum": [
"FULL_TABLE",
"INCREMENTAL",
"LOG_BASED"
]
},
"use_singer_decimal": {
"type": [
"boolean",
"null"
],
"title": "Use Singer Decimal",
"description": "Whether to use use strings with `x-singer.decimal` format for decimals in the discovered schema. This is useful to avoid precision loss when working with large numbers."
},
"stream_maps": {
"type": [
"object",
"null"
],
"properties": {
"__else__": {
"anyOf": [
{
"const": "__NULL__"
},
{
"type": "null"
},
"null"
],
"title": "Other streams",
"description": "Currently, only setting this to `__NULL__` is supported. This will remove all other streams."
}
},
"additionalProperties": {
"anyOf": [
{
"const": "__NULL__"
},
{
"type": "null"
},
{
"type": "object",
"properties": {
"__alias__": {
"type": [
"string",
"null"
],
"title": "Stream Alias",
"description": "Alias to use for the stream."
},
"__else__": {
"anyOf": [
{
"const": "__NULL__"
},
{
"type": "null"
},
"null"
],
"title": "Other properties",
"description": "Currently, only setting this to `__NULL__` is supported. This will remove all other properties from the stream."
},
"__filter__": {
"type": [
"string",
"null"
],
"title": "Filter",
"description": "Filter out records from a stream. A string expression which must evaluate to `true` to include the record, or `false` to exclude it. Filter expressions should be wrapped in `bool()` to ensure proper type conversion."
},
"__key_properties__": {
"type": [
"array",
"null"
],
"items": {
"type": [
"string"
]
},
"title": "Key Properties",
"description": "Primary key properties for the stream."
},
"__source__": {
"type": [
"string",
"null"
],
"description": "Create a new stream from this source stream."
}
},
"additionalProperties": {
"anyOf": [
{
"const": "__NULL__"
},
{
"type": "null"
},
{
"type": [
"string"
]
}
]
}
}
]
},
"title": "Stream Maps",
"description": "Config object for stream maps capability. For more information check out [Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html)."
},
"stream_map_config": {
"type": [
"object",
"null"
],
"properties": {},
"additionalProperties": true,
"title": "User Stream Map Configuration",
"description": "User-defined config values to be used within map expressions."
},
"faker_config": {
"type": [
"object",
"null"
],
"properties": {
"seed": {
"oneOf": [
{
"type": [
"number"
]
},
{
"type": [
"string"
]
},
{
"type": [
"boolean"
]
},
{
"type": "null"
}
],
"title": "Faker Seed",
"description": "Value to seed the Faker generator for deterministic output: https://faker.readthedocs.io/en/master/#seeding-the-generator"
},
"locale": {
"oneOf": [
{
"type": [
"string"
]
},
{
"type": "array",
"items": {
"type": [
"string"
]
}
},
{
"type": "null"
}
],
"title": "Faker Locale",
"description": "One or more LCID locale strings to produce localized output for: https://faker.readthedocs.io/en/master/#localization"
}
},
"additionalProperties": true,
"title": "Faker Configuration",
"description": "Config for the [`Faker`](https://faker.readthedocs.io/en/master/) instance variable `fake` used within map expressions. Only applicable if the plugin specifies `faker` as an additional dependency (through the `singer-sdk` `faker` extra or directly)."
},
"flattening_enabled": {
"type": [
"boolean",
"null"
],
"title": "Enable Schema Flattening",
"description": "'True' to enable schema flattening and automatically expand nested properties."
},
"flattening_max_depth": {
"type": [
"integer",
"null"
],
"title": "Max Flattening Depth",
"description": "The max depth to flatten schemas."
},
"flattening_max_key_length": {
"type": [
"integer",
"null"
],
"title": "Max Key Length",
"description": "The maximum length of a flattened key."
},
"batch_config": {
"type": [
"object",
"null"
],
"properties": {
"encoding": {
"type": [
"object",
"null"
],
"properties": {
"format": {
"type": [
"string",
"null"
],
"title": "Batch Encoding Format",
"description": "Format to use for batch files.",
"enum": [
"jsonl",
"parquet"
]
},
"compression": {
"type": [
"string",
"null"
],
"title": "Batch Compression Format",
"description": "Compression format to use for batch files.",
"enum": [
"gzip",
"none"
]
}
},
"additionalProperties": true,
"title": "Batch Encoding Configuration",
"description": "Specifies the format and compression of the batch files."
},
"storage": {
"type": [
"object",
"null"
],
"properties": {
"root": {
"type": [
"string",
"null"
],
"title": "Batch Storage Root",
"description": "Root path to use when writing batch files."
},
"prefix": {
"type": [
"string",
"null"
],
"title": "Batch Storage Prefix",
"description": "Prefix to use when writing batch files."
}
},
"additionalProperties": true,
"title": "Batch Storage Configuration",
"description": "Defines the storage layer to use when writing batch files"
}
},
"additionalProperties": true,
"title": "Batch Configuration",
"description": "Configuration for BATCH message capabilities."
}
},
"$schema": "https://json-schema.org/draft/2020-12/schema"
}
}
edgarrmondragon
commented
Apr 1, 2026
Co-authored-by: Andreas Motl <andreas.motl@elmyra.de>
amotl
reviewed
Apr 1, 2026
amotl
left a comment
amotl
left a comment
There was a problem hiding this comment.
Just a few more notes. Let me also check if the CrateDB SQLAlchemy driver can do it differently, for example to convey advanced SSL options.
Co-authored-by: Andreas Motl <andreas.motl@elmyra.de>
Co-authored-by: Andreas Motl <andreas.motl@elmyra.de>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Preview