Added option for reference table creation using reference keyword

[jimvekemans]

Issue synopsis

There is a need to create reference tables in dbt-singlestore

Solution

Accept a new reference configuration option for tables which will accept true/false (default false) as it's value, altering the CREATE TABLE statement to CREATE REFERENCE TABLE. This should be done without impacting existing functionality.

Breakdown

testmodel.sql

{{
    config(
        database='staging',
        materialized='table',
        reference=true,
        storage_type='rowstore'
    )
}}
select 1

Before change

Will run as:

create  table `staging`.`testmodel` (SHARD KEY ()) as ...

After considering the reference option in singlestore__create_table_as

create reference table `staging`.`testmodel__dbt_tmp` as ...

Without impacting other types of table

{{
    config(
        database='staging',
        materialized='table',
        storage_type='rowstore'
    )
}}
select 1

Will still normally run as

create rowstore table `staging`.`testmodel` (SHARD KEY ()) as
{{ statement }};

And setting a shard key for the reference table will expectedly result in an error:

{{
    config(
        database='staging',
        materialized='table',
        reference=true,
        shard_key=['1'],
    )
}}
select 1

Will throw the error

1706: Feature 'Sharded reference tables' is not supported by SingleStore.
compiled code at target/run/dbt_academy/models/testmodel.sql

[okramarenko]

hi @jimvekemans — thanks so much for the PRs!

we're updating our contribution guidelines and asking that new functionality include tests -- it helps reviews go faster and keeps the codebase reliable. could you please add a few tests for this change? we've already added the jaffle_shop project to the repo if that's easier to use.

thanks again, and let us know if you'd like any pointers!

[jimvekemans]

hi @jimvekemans — thanks so much for the PRs!

we're updating our contribution guidelines and asking that new functionality include tests -- it helps reviews go faster and keeps the codebase reliable. could you please add a few tests for this change? we've already added the jaffle_shop project to the repo if that's easier to use.

thanks again, and let us know if you'd like any pointers!

Hi @okramarenko I will pick this up next week, and rebase the work on your new version of the adapter!

[jimvekemans]

@okramarenko added tests for validation of the option.

The reference option now is a boolean (like transient tables in snowflake), so {{ config(materialized="table", reference=true, primary_key=["somekey"]) }} now creates a ref table, and {{ config(materialized="table", reference=true, primary_key=["somekey"], storage_type="rowstore") }} now create a rowstore ref table.

This also solves the new open issue #61

[okramarenko]

just run your PR against our test suite — all tests are green :) before we can approve and merge could you please take a look at the comments I left and address them?

thanks again for contributing to the project!

[okramarenko]

there’s a chance we end up with smth like temporary reference or rowstore temporary reference, which triggers a db error (temporary reference tables not supported). that can be a bit confusing for users, so it’s better to raise a clearer dbt error (smth like reference=true isn’t compatible with operations that create temporary tables)

[okramarenko]

you already omit adding the default shard key when reference=true, but if a user defines shard_key in the config alongside reference=true, they’ll get a db error about an unsupported feature. we should add a check like {%- if reference and shard_key | length -%} (say line 56) and raise a dbt error pointing to the config issue (smth like shard_key cannot be used with reference=true because reference tables are not sharded)

[okramarenko]

lets add one more test here with reference = true and shard_key=[...] and check that it fails with a clear compiler error (the comment above)

[okramarenko]

I resolved the comments in #63 to speed up the process a little bit as it would be nice to ship this with 1.10 version of the adapter. Thanks again for your contribution @jimvekemans! :)