Add reservation DSN param to BigQuery driver#8
Open
junyahondarubrik wants to merge 1 commit intomasterfrom
Open
Add reservation DSN param to BigQuery driver#8junyahondarubrik wants to merge 1 commit intomasterfrom
junyahondarubrik wants to merge 1 commit intomasterfrom
Conversation
Parse reservation query parameter from the BigQuery DSN URI and apply it to queries. This enables job-level reservation routing via the connection string for the 2-reservation-pool resource management approach. - Add reservation field to bigQueryConfig - Parse it from DSN query param in configFromUri() - Apply it in buildQuery() as query.Reservation - Bump cloud.google.com/go/bigquery from v1.12.0 to v1.68.0 (required for QueryConfig.Reservation field) - Add table-driven tests for DSN parsing with reservation param
junyahondarubrik
force-pushed
the
ubuntu-dsn-reservation-priority
branch
from
cfbb560 to
d1e7c7e
Compare
junyahondarubrik
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why
BQ priority-based resource management (SPARK-755723) uses a 2-reservation-pool
approach where foreground and background queries are routed to separate BigQuery
reservations. This requires the ability to set
QueryConfig.ReservationonBigQuery jobs via the DSN connection string.
What
Add support for a
reservationquery parameter in the BigQuery DSN URI. Whenpresent, the reservation value is applied to every query executed through the
connection.
How
reservationfield tobigQueryConfigconfigFromUri()buildQuery()to setquery.Reservationcloud.google.com/go/bigqueryfrom v1.12.0 to v1.68.0 (required forthe
QueryConfig.Reservationfield added in v1.68.0)Example DSN
Test Plan
reservation with other params, invalid scheme, missing host
Related