Skip to content

bq>: does not support partition decorators other than daily type #1834

@potato2003

Description

@potato2003

private static final Pattern TABLE_REFERENCE_PATTERN = Pattern.compile("^(?:(?<project>[^:]+):)?(?:(?<dataset>[^.]+)\\.)?(?<table>[a-zA-Z0-9_]{1,1024}(?:\\$[0-9]{8})?)$");

It looks like only 8 digits are allowed for partition decorators.

https://cloud.google.com/bigquery/docs/partitioned-tables#partition_decorators

Currently that there are many types of partition decorators available, I think the limit on the number of digits should be removed.

+task
  bq>: ${sql_script}
    destination_table: table_a$2025050213
...
java.lang.IllegalArgumentException: Bad table reference: table_a$2025050213
	at io.digdag.standards.operator.gcp.Bq.tableReference(Bq.java:18)
	at io.digdag.standards.operator.gcp.BqOperatorFactory$BqOperator.lambda$jobConfiguration$0(BqOperatorFactory.java:88)
	at com.google.common.base.Present.transform(Present.java:75)
	at io.digdag.standards.operator.gcp.BqOperatorFactory$BqOperator.jobConfiguration(BqOperatorFactory.java:88)
	at io.digdag.standards.operator.gcp.BaseBqJobOperator.run(BaseBqJobOperator.java:27)
	at io.digdag.standards.operator.gcp.BaseBqOperator.run(BaseBqOperator.java:28)
	at io.digdag.standards.operator.gcp.BaseGcpOperator.runTask(BaseGcpOperator.java:25)
	at io.digdag.util.BaseOperator.run(BaseOperator.java:35)
	at io.digdag.core.agent.OperatorManager.callExecutor(OperatorManager.java:399)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invokeMain(DigdagTimedMethodInterceptor.java:58)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invoke(DigdagTimedMethodInterceptor.java:31)
	at io.digdag.core.agent.OperatorManager.runWithWorkspace(OperatorManager.java:308)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invokeMain(DigdagTimedMethodInterceptor.java:58)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invoke(DigdagTimedMethodInterceptor.java:31)
	at io.digdag.core.agent.OperatorManager.lambda$runWithHeartbeat$2(OperatorManager.java:152)
	at io.digdag.core.agent.ExtractArchiveWorkspaceManager.withExtractedArchive(ExtractArchiveWorkspaceManager.java:75)
	at io.digdag.core.agent.OperatorManager.runWithHeartbeat(OperatorManager.java:150)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invokeMain(DigdagTimedMethodInterceptor.java:58)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invoke(DigdagTimedMethodInterceptor.java:31)
	at io.digdag.core.agent.OperatorManager.run(OperatorManager.java:133)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invokeMain(DigdagTimedMethodInterceptor.java:58)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invoke(DigdagTimedMethodInterceptor.java:31)
	at io.digdag.core.agent.MultiThreadAgent.lambda$null$0(MultiThreadAgent.java:132)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions