SNOW-3484790: use local list instead of fetch from snowflake#4217
SNOW-3484790: use local list instead of fetch from snowflake#4217sfc-gh-yuwang wants to merge 1 commit intomainfrom
Conversation
|
the updated system function list is fetched with query: |
| for r in self.sql( | ||
| """show functions ->> select "name" from $1 where "is_aggregate" = 'Y'""" |
There was a problem hiding this comment.
My understanding is that queries like this ensure the server is always the source of truth, and we don't have to make client code changes that are aligned with whatever Snowflake version is being run on the server. Hard-coding it to a local change is something of a philosophical change, and may unexpectedly break workloads in some circumstances.
It looks to only ever get run once per workload anyway, so I think the benefits of this would be pretty minimal.
There was a problem hiding this comment.
I had a discussion with Adam and Yijun this morning. One of the issue is that this is always run when there are potentially agg function in the query, which means user would still see this even if they just issued 1 query in SCOS.
This is also a pretty expensive operation that took 3~4 seconds, which is a big portion when user workflow is small.
To solve the problem that this list may diverge from server, I created a followup ticket: https://snowflakecomputing.atlassian.net/browse/SNOW-3489271
Currently my thoughts is adding a step to release pipeline to ensure that each snowpark release's system function list is up-to-date at the point it released. Thus user would only need to use latest snowpark to avoid the mis align issue.
But I think it also make sense to add parameter protection for this. Will add one
There was a problem hiding this comment.
Parameter-protecting makes sense. I'm slightly worried about server divergence because the same client release might get run against different version backends (I'm not sure if we the N-1 version policy still exists, but at the very least testing vs. prod environments, and customers on different release cadences might get different versions)
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4217 +/- ##
==========================================
- Coverage 95.42% 95.04% -0.39%
==========================================
Files 171 171
Lines 43837 43872 +35
Branches 7513 7525 +12
==========================================
- Hits 41833 41698 -135
- Misses 1225 1368 +143
- Partials 779 806 +27 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.
Fixes SNOW-NNNNNNN
Fill out the following pre-review checklist:
Please describe how your code solves the related issue.
Please write a short description of how your code change solves the related issue.