Implement function materialization by shachibista · Pull Request #588 · ClickHouse/dbt-clickhouse

shachibista · 2026-01-09T17:37:19Z

Summary

Adds support for User Defined Functions.

Checklist

Delete items not relevant to your PR:

Unit and integration tests covering the common scenarios were added
A human-readable description of the changes was provided to include in CHANGELOG
For significant changes, documentation in https://github.com/ClickHouse/clickhouse-docs was updated with further explanations or tutorials

Note

Medium Risk
Introduces a new relation type and DDL macros for creating ClickHouse functions, plus upgrades to dbt-core>=1.11.0; these may affect relation rendering/schema handling and overall adapter compatibility. Connection logging now emits SQLQuery events, which could change log volume/behavior.

Overview
Adds initial support for dbt User Defined Functions on ClickHouse by introducing a function relation type (treated as global by ignoring schema) and new macros to CREATE OR REPLACE FUNCTION for scalar UDFs.

Upgrades the adapter version to 1.11.0 and bumps development dependency to dbt-core>=1.11.0, adds an integration test (tests/integration/adapter/udf/test_udf.py) covering basic UDF creation, and updates connection execution to emit SQLQuery events (with optional SQL log abridging) before running statements.

^{Written by Cursor Bugbot for commit 37472f6. This will update automatically on new commits. Configure here.}

koletzilla · 2026-01-13T11:56:57Z

Related to #590

koletzilla · 2026-01-13T12:15:18Z

Hi @shachibista! Thanks for the contribution. This is a really interesting functionality and I just started checking the implementation details, so I still don't have a full picture of the work needed. I'm sorry I can't give complete feedback yet, but I wanted to ask a few questions to get your opinion:

Can you walk through all the changes you needed to make to the materialization function macro? It looks like we should be able to use practically the same implementation as the default in dbt. Why did you need to apply these changes?
Were you able to later call this function from a model? I'm getting different errors when I try to use it. Can you add a test to assert that the function can be called from a model?
It looks like we may need to update dbt-adapters to the latest version to be able to use UDFs, but you didn't do it. Is there a reason not to do it?
dbt usually releases tests in the dbt-tests-adapter package that we can call from this adapter like this: https://github.com/dbt-labs/dbt-adapters/tree/main/dbt-tests-adapter/src/dbt/tests/adapter/functions. Have you checked if we can reuse these tests?

shachibista · 2026-01-14T22:35:15Z

It looks like we may need to update dbt-adapters to the latest version to be able to use UDFs, but you didn't do it. Is there a reason not to do it?

Ah yes, I missed that.

Can you walk through all the changes you needed to make to the materialization function macro? It looks like we should be able to use practically the same implementation as the default in dbt. Why did you need to apply these changes?

You are correct, I tried to use the default implementation but did not realise that dbt-adapters needed to be updated too. Should be unnecessary after update.

dbt usually releases tests in the dbt-tests-adapter package that we can call from this adapter like this: https://github.com/dbt-labs/dbt-adapters/tree/main/dbt-tests-adapter/src/dbt/tests/adapter/functions. Have you checked if we can reuse these tests?

Unfortunately we cannot use them because ClickHouse functions expect an expression:

CREATE [OR REPLACE] FUNCTION name [ON CLUSTER cluster] AS (parameter0, ...) -> expression

However the tests defined in the adapters repo expect the function body to be a SELECT statement.

Were you able to later call this function from a model? I'm getting different errors when I try to use it. Can you add a test to assert that the function can be called from a model?

What errors are you getting? I did not try to run it from a model but I currently get the following when trying to run from a model.

 'SourceQuotingBaseConfig' object has no attribute 'credentials'

I am not sure what the reason is.

cursor · 2026-01-14T22:41:41Z

+{% endmacro %}
+
+{% macro clickhouse__scalar_function_create_replace_signature_sql(target_relation) %}
+  CREATE OR REPLACE FUNCTION {{ target_relation.include(database=false, schema=false) }} {{ on_cluster_clause(this) }} AS ({{ clickhouse__formatted_scalar_function_args_sql() }}) ->


Inconsistent relation used for cluster clause

Medium Severity

The clickhouse__scalar_function_create_replace_signature_sql macro receives target_relation as a parameter and uses it for the function name, but then uses the global this variable for on_cluster_clause(this). This is inconsistent with the established pattern throughout the codebase where the same relation is used for both the identifier and the cluster clause. If target_relation and this ever differ, the ON CLUSTER clause may be generated incorrectly or omitted unexpectedly.

koletzilla · 2026-02-04T16:16:46Z

Hi @shachibista ! Sorry for not continuing giving more feedback. This PR is really interesting for us and I like to review it with care, but we have not prioritised this feature right now.

My idea is to work with you to try to merge this PR in near future. I'll give you piece of feedback again as soon as I can.

In the meantime, some feedback:

About your changes in the extra spaces in the changelog: I'm ok if you want to move the small changelog fixes to a different PR. That way we can reduce the size of this one and we can merge the other asap.

About the tests in dbt-tests-adapter:

However the tests defined in the adapters repo expect the function body to be a SELECT statement.

You should be able to override the definition of the files to make them match the CH expectations. That way we should reuse the rest of the tests. Would you try? Maybe we something like this https://github.com/ClickHouse/dbt-clickhouse/blob/main/tests/integration/adapter/unit_testing/test_types.py so the tests functions are reused and we just override the part that are different in CH.

About the errors when using the created UDFs:

What errors are you getting? I did not try to run it from a model but I currently get the following when trying to run from a model.

I recall seeing similar ones. For this feature to be merged we need not just to define the UDFs, but also use it in models later. Please, try to check it again to see if we may be missing something 🙏 .

About dbt-adapter version change: I see somethings are failing related to the dbt-adapter version change. I'm going to try to fix it in a separated PR so you don't have to deal with it in this PR

Also another question: The default UDF materialization in dbt mentions it supports both Python and SQL functions. Did you tested both? Can you also add a test to assert that Python one works?

Thanks!

- Use UDF tests from dbt-adapter - Fire event during SQLQuery (test expectation) - Fix function relation

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

cursor · 2026-03-11T22:06:30Z

            if schema == relation_config.source_name and relation_config.database:
                schema = relation_config.database
+        elif relation_config.resource_type == NODE_TYPE_FUNCTION:
+            schema = None


Functions never get ON CLUSTER clause in clusters

Medium Severity

The elif branch for NODE_TYPE_FUNCTION in create_from sets schema = None but falls through without reading cluster and database_engine from quoting.credentials, since that logic is in the else block. This leaves cluster as "" and can_on_cluster as None, so on_cluster_clause(this) in scalar.sql will never emit the ON CLUSTER clause — even when a cluster is configured. The cluster/database_engine credential reads need to happen for function types too.

Additional Locations (1)

dbt/include/clickhouse/macros/materializations/functions/scalar.sql#L9-L10

shachibista · 2026-03-11T22:08:20Z

About your changes in the extra spaces in the changelog: I'm ok if you want to move the small changelog fixes to a different PR. That way we can reduce the size of this one and we can merge the other asap.

Ah, it was a side-effect of the linter I was using.

You should be able to override the definition of the files to make them match the CH expectations. That way we should reuse the rest of the tests. Would you try? Maybe we something like this https://github.com/ClickHouse/dbt-clickhouse/blob/main/tests/integration/adapter/unit_testing/test_types.py so the tests functions are reused and we just override the part that are different in CH.

I have now changed the tests to re-use the dbt-adapter tests. Although had to add events so that the tests worked as expected, probably a good thing if other tests depend on it and we want to reuse tests in the future.

Also another question: The default UDF materialization in dbt mentions it supports both Python and SQL functions. Did you tested both? Can you also add a test to assert that Python one works?

ClickHouse seems to support python UDFs but they need to be configured on the server. So I deem it outside dbt's responsibility. I have not tested, nor intend to implement it.

cursor bot reviewed Jan 9, 2026

View reviewed changes

Comment thread dbt/include/clickhouse/macros/materializations/function.sql Outdated

cursor bot reviewed Jan 14, 2026

View reviewed changes

shachibista force-pushed the implement-function-materialization branch from 2a75b23 to d90afdc Compare January 17, 2026 21:36

koletzilla mentioned this pull request Feb 6, 2026

Update dbt-adapters version to 1.22 #609

Merged

shachibista added 6 commits March 11, 2026 21:41

feat: implement UDF materialization

0ff4536

tests: Add integration tests for UDF

5963bea

Update CHANGELOG.md

6632fe1

fixup! feat: implement UDF materialization

077e3fa

Linter

a84c68c

Rebase with upstream and PR fixes

271df27

- Use UDF tests from dbt-adapter - Fire event during SQLQuery (test expectation) - Fix function relation

shachibista force-pushed the implement-function-materialization branch from 5b7d70f to 271df27 Compare March 11, 2026 22:04

cursor bot reviewed Mar 11, 2026

View reviewed changes

shachibista added 2 commits March 11, 2026 23:09

fixup! Rebase with upstream and PR fixes

37472f6

Revert formatting changes to CHANGELOG

f71adf7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement function materialization#588

Implement function materialization#588
shachibista wants to merge 8 commits intoClickHouse:mainfrom
shachibista:implement-function-materialization

shachibista commented Jan 9, 2026 •

edited by cursor bot

Loading

Uh oh!

Uh oh!

koletzilla commented Jan 13, 2026

Uh oh!

koletzilla commented Jan 13, 2026

Uh oh!

shachibista commented Jan 14, 2026 •

edited

Loading

Uh oh!

cursor bot Jan 14, 2026

Uh oh!

koletzilla commented Feb 4, 2026

Uh oh!

cursor bot left a comment

Uh oh!

cursor bot Mar 11, 2026

Uh oh!

shachibista commented Mar 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

shachibista commented Jan 9, 2026 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Uh oh!

Uh oh!

koletzilla commented Jan 13, 2026

Uh oh!

koletzilla commented Jan 13, 2026

Uh oh!

shachibista commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor bot Jan 14, 2026

Choose a reason for hiding this comment

Inconsistent relation used for cluster clause

Uh oh!

koletzilla commented Feb 4, 2026

In the meantime, some feedback:

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor bot Mar 11, 2026

Choose a reason for hiding this comment

Functions never get ON CLUSTER clause in clusters

Uh oh!

shachibista commented Mar 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

shachibista commented Jan 9, 2026 •

edited by cursor bot

Loading

shachibista commented Jan 14, 2026 •

edited

Loading