From 02b9781f1f27c49f65eb02cd1f1a37a75010b8b2 Mon Sep 17 00:00:00 2001 From: Lawrence Qiu Date: Fri, 3 Oct 2025 14:13:23 -0400 Subject: [PATCH 1/8] docs: Point the README.md to the officla Java Docs page Remove all sections that have been migrated to the offical docs page --- README.md | 365 +----------------------------------------------------- 1 file changed, 4 insertions(+), 361 deletions(-) diff --git a/README.md b/README.md index 645540017b21..bfa8b249b937 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,10 @@ * [Java on Google Cloud][cloud-java] +## Documentation + +See the [official guide](https://cloud.google.com/java/docs/setup) to get setup and started with development. + ## Supported APIs Libraries are available on GitHub and Maven Central for developing Java applications that interact with individual Google Cloud services: @@ -236,371 +240,10 @@ If the service is not listed, [google-api-java-client][google-api-java-client-se *When building Java applications, preference should be given to the libraries listed in the table.* - - -## Specifying a Project ID - -Most `google-cloud` libraries require a project ID. There are multiple ways to specify this project ID. - -1. When using `google-cloud` libraries from within Compute/App Engine, there's no need to specify a project ID. It is automatically inferred from the production environment. -2. When using `google-cloud` elsewhere, you can do one of the following: -* Supply the project ID when building the service options. For example, to use Datastore from a project with ID "PROJECT_ID", you can write: - - ```java - Datastore datastore = DatastoreOptions.newBuilder().setProjectId("PROJECT_ID").build().getService(); - ``` -* Specify the environment variable `GOOGLE_CLOUD_PROJECT` to be your desired project ID. -* Set the project ID using the [Google Cloud SDK](https://cloud.google.com/sdk/?hl=en). To use the SDK, [download the SDK](https://cloud.google.com/sdk/?hl=en) if you haven't already, and set the project ID from the command line. For example: - - ``` - gcloud config set project PROJECT_ID - ``` - -`google-cloud` determines the project ID from the following sources in the listed order, stopping once it finds a value: - -1. The project ID supplied when building the service options -2. Project ID specified by the environment variable `GOOGLE_CLOUD_PROJECT` -3. The App Engine / Compute Engine project ID -4. The project ID specified in the JSON credentials file pointed by the `GOOGLE_APPLICATION_CREDENTIALS` environment variable -5. The Google Cloud SDK project ID - -In cases where the library may expect a project ID explicitly, we provide a helper that can provide the inferred project ID: - ```java - import com.google.cloud.ServiceOptions; - ... - String projectId = ServiceOptions.getDefaultProjectId(); - ``` - -## Authentication - -`google-cloud-java` uses -[https://github.com/googleapis/google-auth-library-java](https://github.com/googleapis/google-auth-library-java) -to authenticate requests. `google-auth-library-java` supports a wide range of authentication types; -see the project's [README](https://github.com/google/google-auth-library-java/blob/main/README.md) -and [javadoc](https://cloud.google.com/java/docs/reference/google-auth-library/latest/overview) for more -details. - -### Google Cloud Platform environment - -When using Google Cloud libraries from a Google Cloud Platform environment such as Compute Engine, -Kubernetes Engine, or App Engine, no additional authentication steps are necessary. - -For example: - -```java -Storage storage = StorageOptions.getDefaultInstance().getService(); -``` - -or: - -```java -CloudTasksClient cloudTasksClient = CloudTasksClient.create(); -``` - -### Other environments - -#### Using a service account (recommended) - -1. [Generate a JSON service account key](https://cloud.google.com/storage/docs/authentication?hl=en#service_accounts). - -2. After downloading that key, you must do one of the following: - * Define the environment variable GOOGLE_APPLICATION_CREDENTIALS to be the location of the key. - For example: - ```bash - export GOOGLE_APPLICATION_CREDENTIALS=/path/to/my/key.json - ``` - * Supply the JSON credentials file when building the service options. For example, this Storage - object has the necessary permissions to interact with your Google Cloud Storage data: - ```java - Storage storage = StorageOptions.newBuilder() - .setCredentials(ServiceAccountCredentials.fromStream(new FileInputStream("/path/to/my/key.json"))) - .build() - .getService(); - ``` - -#### Local development/testing - -If running locally for development/testing, you can use the [Google Cloud SDK](https://cloud.google.com/sdk/). -Create Application Default Credentials with `gcloud auth application-default login`, and then -`google-cloud` will automatically detect such credentials. - -#### Existing OAuth2 access token - -If you already have an OAuth2 access token, you can use it to authenticate (notice that in this case, the -access token will not be automatically refreshed): - -```java -Credentials credentials = GoogleCredentials.create(new AccessToken(accessToken, expirationTime)); -Storage storage = StorageOptions.newBuilder() - .setCredentials(credentials) - .build() - .getService(); -``` - -or: - -```java -Credentials credentials = GoogleCredentials.create(new AccessToken(accessToken, expirationTime)); -CloudTasksSettings cloudTasksSettings = CloudTasksSettings.newBuilder() - .setCredentialProvider(FixedCredentialsProvider.create(credentials)) - .build(); -CloudTasksClient cloudTasksClient = CloudTasksClient.create(cloudTasksSettings); -``` - -### Application Default Credentials - -If no credentials are provided, `google-cloud` will attempt to detect them from the environment -using `GoogleCredentials.getApplicationDefault()` which will search for Application Default -Credentials in the following locations (in order): - -1. The credentials file pointed to by the `GOOGLE_APPLICATION_CREDENTIALS` environment variable -2. Credentials provided by the Google Cloud SDK `gcloud auth application-default login` command -3. Google App Engine built-in credentials -4. Google Cloud Shell built-in credentials -5. Google Compute Engine built-in credentials - -### Authenticating with an API Key - -[Authenticating with API Keys](https://cloud.google.com/docs/authentication/api-keys) is supported by a handful of Google Cloud APIs. - -We are actively exploring ways to improve the API Key experience. -Currently, to use an API Key with a Java client library, you need to set the header for the relevant service Client manually. - -For example, to set the API Key with the [Language service](https://cloud.google.com/java/docs/reference/google-cloud-language/latest/overview): - -```java -public LanguageServiceClient createGrpcClientWithApiKey(String apiKey) throws Exception { - // Manually set the api key via the header - Map header = new HashMap() { {put("x-goog-api-key", apiKey);}}; - FixedHeaderProvider headerProvider = FixedHeaderProvider.create(header); - - // Create the client - TransportChannelProvider transportChannelProvider = InstantiatingGrpcChannelProvider.newBuilder().setHeaderProvider(headerProvider).build(); - LanguageServiceSettings settings = LanguageServiceSettings.newBuilder().setTransportChannelProvider(transportChannelProvider).build(); - LanguageServiceClient client = LanguageServiceClient.create(settings); - return client; - } -``` - -An example instantiation with the Language Client using rest: -```java - public LanguageServiceClient createRestClientWithApiKey(String apiKey) throws Exception { - // Manually set the api key header - Map header = new HashMap() { {put("x-goog-api-key", apiKey);}}; - FixedHeaderProvider headerProvider = FixedHeaderProvider.create(header); - - // Create the client - TransportChannelProvider transportChannelProvider = InstantiatingHttpJsonChannelProvider.newBuilder().setHeaderProvider(headerProvider).build(); - LanguageServiceSettings settings = LanguageServiceSettings.newBuilder().setTransportChannelProvider(transportChannelProvider).build(); - LanguageServiceClient client = LanguageServiceClient.create(settings); - return client; - } -``` - ## Troubleshooting To get help, follow the instructions in the [Troubleshooting document](https://github.com/googleapis/google-cloud-java/blob/main/TROUBLESHOOTING.md). -## Configuring a Proxy - -Google Cloud client libraries use HTTPS and gRPC in underlying communication -with the services. -In both protocols, you can configure a proxy using `https.proxyHost` -and (optional) `https.proxyPort` properties. - -### gRPC Custom Proxy Configuration - -For a more custom proxy with gRPC, you will need supply a `ProxyDetector` to -the `ManagedChannelBuilder`: - -```java -import com.google.api.core.ApiFunction; -import com.google.api.gax.rpc.TransportChannelProvider; -import com.google.cloud.tasks.v2.CloudTasksClient; -import com.google.cloud.tasks.v2.CloudTasksSettings; -import com.google.cloud.tasks.v2.stub.CloudTasksStubSettings; -import io.grpc.HttpConnectProxiedSocketAddress; -import io.grpc.ManagedChannelBuilder; -import io.grpc.ProxiedSocketAddress; -import io.grpc.ProxyDetector; - -import javax.annotation.Nullable; -import java.io.IOException; -import java.net.InetSocketAddress; -import java.net.SocketAddress; - -public CloudTasksClient getService() throws IOException { - TransportChannelProvider transportChannelProvider = - CloudTasksStubSettings.defaultGrpcTransportProviderBuilder() - .setChannelConfigurator( - new ApiFunction() { - @Override - public ManagedChannelBuilder apply(ManagedChannelBuilder managedChannelBuilder) { - return managedChannelBuilder.proxyDetector( - new ProxyDetector() { - @Nullable - @Override - public ProxiedSocketAddress proxyFor(SocketAddress socketAddress) - throws IOException { - return HttpConnectProxiedSocketAddress.newBuilder() - .setUsername(PROXY_USERNAME) - .setPassword(PROXY_PASSWORD) - .setProxyAddress(new InetSocketAddress(PROXY_HOST, PROXY_PORT)) - .setTargetAddress((InetSocketAddress) socketAddress) - .build(); - } - }); - } - }) - .build(); - CloudTasksSettings cloudTasksSettings = - CloudTasksSettings.newBuilder() - .setTransportChannelProvider(transportChannelProvider) - .build(); - return CloudTasksClient.create(cloudTasksSettings); -} -``` - -## Long Running Operations - -Long running operations (LROs) are often used for API calls that are expected to -take a long time to complete (i.e. provisioning a GCE instance or a Dataflow pipeline). -The initial API call creates an "operation" on the server and returns an Operation ID -to track its progress. LRO RPCs have the suffix `Async` appended to the call name -(i.e. `clusterControllerClient.createClusterAsync()`) - -Our generated clients provide a nice interface for starting the operation and -then waiting for the operation to complete. This is accomplished by returning an -[`OperationFuture`](https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.longrunning.OperationFuture). -When calling `get()` on the `OperationFuture`, the client library will poll the operation to -check the operation's status. - -For example, take a sample `createCluster` Operation in google-cloud-dataproc v4.20.0: -```java -try (ClusterControllerClient clusterControllerClient = ClusterControllerClient.create()) { - CreateClusterRequest request = - CreateClusterRequest.newBuilder() - .setProjectId("{PROJECT_ID}") - .setRegion("{REGION}") - .setCluster(Cluster.newBuilder().build()) - .setRequestId("{REQUEST_ID}") - .setActionOnFailedPrimaryWorkers(FailureAction.forNumber(0)) - .build(); - OperationFuture future = - clusterControllerClient.createClusterOperationCallable().futureCall(request); - // Do something. - Cluster response = future.get(); -} catch (CancellationException e) { - // Exceeded the default RPC timeout without the Operation completing. - // Library is no longer polling for the Operation status. Consider - // increasing the timeout. -} -``` - -### LRO Timeouts -The polling operations have a default timeout that varies from service to service. -The library will throw a `java.util.concurrent.CancellationException` with the message: -`Task was cancelled.` if the timeout exceeds the operation. A `CancellationException` -does not mean that the backend GCP Operation was cancelled. This exception is thrown from the -client library when it has exceeded the total timeout without receiving a successful status from the operation. -Our client libraries respect the configured values set in the OperationTimedPollAlgorithm for each RPC. - -Note: The client library handles the Operation's polling mechanism for you. By default, there is no need -to manually poll the status yourself. - -### Default LRO Values -Each LRO RPC has a set of pre-configured default values. You can find these values by -searching in each Client's `StubSettings`'s class. The default LRO settings are initialized -inside the `initDefaults()` method in the nested Builder class. - -For example, in google-cloud-aiplatform v3.24.0, the default [OperationTimedPollAlgorithm](https://github.com/googleapis/google-cloud-java/blob/9ae786d1acdc7354adf86b78691570668caa293d/java-aiplatform/google-cloud-aiplatform/src/main/java/com/google/cloud/aiplatform/v1/stub/EndpointServiceStubSettings.java#L755-L765) -has these default values: -```java -OperationTimedPollAlgorithm.create( - RetrySettings.newBuilder() - .setInitialRetryDelay(Duration.ofMillis(5000L)) - .setRetryDelayMultiplier(1.5) - .setMaxRetryDelay(Duration.ofMillis(45000L)) - .setInitialRpcTimeout(Duration.ZERO) - .setRpcTimeoutMultiplier(1.0) - .setMaxRpcTimeout(Duration.ZERO) - .setTotalTimeout(Duration.ofMillis(300000L)) - .build()) -``` -Both retries and LROs share the same RetrySettings class. Note the corresponding link: -- Total Timeout (Max Time allowed for polling): 5 minutes -- Initial Retry Delay (Initial delay before first poll): 5 seconds -- Max Retry Delay (Maximum delay between each poll): 45 seconds -- Retry Delay Multiplier (Multiplier value to increase the poll delay): 1.5 - -The RPC Timeout values have no use in LROs and can be omitted or set to the default values -(`Duration.ZERO` for Timeouts or `1.0` for the multiplier). - -### Configuring LRO Timeouts -To configure the LRO values, create an OperationTimedPollAlgorithm object and update the -RPC's polling algorithm. For example: -```java -ClusterControllerSettings.Builder settingsBuilder = ClusterControllerSettings.newBuilder(); -TimedRetryAlgorithm timedRetryAlgorithm = OperationTimedPollAlgorithm.create( - RetrySettings.newBuilder() - .setInitialRetryDelay(Duration.ofMillis(500L)) - .setRetryDelayMultiplier(1.5) - .setMaxRetryDelay(Duration.ofMillis(5000L)) - .setInitialRpcTimeout(Duration.ZERO) // ignored - .setRpcTimeoutMultiplier(1.0) // ignored - .setMaxRpcTimeout(Duration.ZERO) // ignored - .setTotalTimeout(Duration.ofHours(24L)) // set polling timeout to 24 hours - .build()); -settingsBuilder.createClusterOperationSettings() - .setPollingAlgorithm(timedRetryAlgorithm); -ClusterControllerClient clusterControllerClient = ClusterControllerClient.create(settingsBuilder.build()); -``` - -Note: The configuration above *only* modifies the LRO values for the `createClusterOperation` RPC. -The other RPCs in the Client will still use each RPC's pre-configured LRO values. - -## Managing Dependencies - -If you are using more than one Google Cloud client library, we recommend you use one of -our Bill of Material (BOM) artifacts to help manage dependency versions. For more information, -see [Using the Cloud Client Libraries](https://cloud.google.com/java/docs/bom). - -## Java Versions - -Java 8 or above is required for using the clients in this repository. - -## Supported Platforms - -Clients in this repository use either HTTP or gRPC for the transport layer. All -HTTP-based clients should work in all environments. - -For clients that use gRPC, the supported platforms are constrained by the platforms -that [Forked Tomcat Native](http://netty.io/wiki/forked-tomcat-native.html) supports, -which for architectures means only x86_64, and for operating systems means Mac OS X, -Windows, and Linux. Additionally, gRPC constrains the use of platforms with -threading restrictions. - -Thus, the following are not supported: - -- Android - - Consider [Firebase](https://firebase.google.com), which includes many of these APIs. - - It is possible to use these libraries in many cases, although it is unsupported. - You can find examples, such as [this one](https://github.com/GoogleCloudPlatform/android-docs-samples/tree/master/speech/SpeechRecognitionClient), - in this [example repository](https://github.com/GoogleCloudPlatform/android-docs-samples) but consider the risks carefully before using these libraries in an application. -- Raspberry Pi (since it runs on the ARM architecture) -- Google App Engine Standard Java 7 - -The following environments should work (among others): - -- standalone Windows on x86_64 -- standalone Mac OS X on x86_64 -- standalone Linux on x86_64 -- Google Compute Engine (GCE) -- Google Container Engine (GKE) -- Google App Engine Standard Java 8 (GAE Std J8) -- Google App Engine Flex (GAE Flex) -- Alpine Linux (Java 11+) - ## Testing This library provides tools to help write tests for code that uses google-cloud services. From 1fa79dc45d61ca40b1c933fd71e8764313200f91 Mon Sep 17 00:00:00 2001 From: Lawrence Qiu Date: Fri, 3 Oct 2025 14:21:45 -0400 Subject: [PATCH 2/8] docs: Remove Troubleshooting section --- README.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/README.md b/README.md index bfa8b249b937..4d797bdbdf0e 100644 --- a/README.md +++ b/README.md @@ -240,10 +240,6 @@ If the service is not listed, [google-api-java-client][google-api-java-client-se *When building Java applications, preference should be given to the libraries listed in the table.* -## Troubleshooting - -To get help, follow the instructions in the [Troubleshooting document](https://github.com/googleapis/google-cloud-java/blob/main/TROUBLESHOOTING.md). - ## Testing This library provides tools to help write tests for code that uses google-cloud services. From 1ff1642f702508a7ed9075156dd84eef284c141d Mon Sep 17 00:00:00 2001 From: Lawrence Qiu Date: Thu, 2 Apr 2026 16:51:03 -0400 Subject: [PATCH 3/8] ci: Remove migrated libraries from downstream protobuf check --- ...-java-downstream_protobuf_compatibility_check_nightly.yaml | 4 ---- 1 file changed, 4 deletions(-) diff --git a/.github/workflows/sdk-platform-java-downstream_protobuf_compatibility_check_nightly.yaml b/.github/workflows/sdk-platform-java-downstream_protobuf_compatibility_check_nightly.yaml index 858c02de00f3..1c2bd053a5e3 100644 --- a/.github/workflows/sdk-platform-java-downstream_protobuf_compatibility_check_nightly.yaml +++ b/.github/workflows/sdk-platform-java-downstream_protobuf_compatibility_check_nightly.yaml @@ -40,14 +40,10 @@ jobs: fail-fast: false matrix: repo: - - google-cloud-java - java-bigtable - - java-bigquery - java-firestore - java-pubsub - java-pubsublite - - java-spanner-jdbc - - java-spanner - java-storage - java-storage-nio # Default Protobuf-Java versions to use are specified here. Without this, the nightly workflow won't know From 12839f8bf15226840d7c93d7c86b398cf2ad85ee Mon Sep 17 00:00:00 2001 From: Lawrence Qiu Date: Thu, 2 Apr 2026 16:58:29 -0400 Subject: [PATCH 4/8] chore: Remove path filter --- ...stream_protobuf_compatibility_check_nightly.yaml | 13 ------------- 1 file changed, 13 deletions(-) diff --git a/.github/workflows/sdk-platform-java-downstream_protobuf_compatibility_check_nightly.yaml b/.github/workflows/sdk-platform-java-downstream_protobuf_compatibility_check_nightly.yaml index 1c2bd053a5e3..5de3f076eefc 100644 --- a/.github/workflows/sdk-platform-java-downstream_protobuf_compatibility_check_nightly.yaml +++ b/.github/workflows/sdk-platform-java-downstream_protobuf_compatibility_check_nightly.yaml @@ -16,20 +16,7 @@ name: sdk-platform-java Downstream Protobuf Compatibility Check Nightly env: BUILD_SUBDIR: sdk-platform-java jobs: - filter: - runs-on: ubuntu-latest - outputs: - library: ${{ steps.filter.outputs.library }} - steps: - - uses: actions/checkout@v4 - - uses: dorny/paths-filter@v3 - id: filter - with: - filters: | - library: - - 'sdk-platform-java/**' downstream-protobuf-test: - needs: filter # This job runs if any of the three conditions match: # 1. PR is raised from Release-Please (PR comes from branch: release-please--branches-main) # 2. Job is invoked by the nightly job (scheduled event) From 33819eb4419c9403448bec2261e5671bf9dbd70c Mon Sep 17 00:00:00 2001 From: Lawrence Qiu Date: Thu, 2 Apr 2026 17:00:23 -0400 Subject: [PATCH 5/8] chore: Remove reference to the filter job --- ...rm-java-downstream_protobuf_compatibility_check_nightly.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/sdk-platform-java-downstream_protobuf_compatibility_check_nightly.yaml b/.github/workflows/sdk-platform-java-downstream_protobuf_compatibility_check_nightly.yaml index 5de3f076eefc..45b756a6d6b4 100644 --- a/.github/workflows/sdk-platform-java-downstream_protobuf_compatibility_check_nightly.yaml +++ b/.github/workflows/sdk-platform-java-downstream_protobuf_compatibility_check_nightly.yaml @@ -21,7 +21,7 @@ jobs: # 1. PR is raised from Release-Please (PR comes from branch: release-please--branches-main) # 2. Job is invoked by the nightly job (scheduled event) # 3. Job is manually invoked via Github UI (workflow_dispatch event) - if: needs.filter.outputs.library == 'true' && (github.head_ref == 'release-please--branches--main' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') + if: github.head_ref == 'release-please--branches--main' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' runs-on: ubuntu-22.04 strategy: fail-fast: false From 17a9303ac0d238f7071004bc123a0012427403b6 Mon Sep 17 00:00:00 2001 From: Lawrence Qiu Date: Thu, 2 Apr 2026 17:12:06 -0400 Subject: [PATCH 6/8] chore: Add the correct path for the source and binary compatibility tests --- ...a-downstream_protobuf_compatibility_check_nightly.yaml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/sdk-platform-java-downstream_protobuf_compatibility_check_nightly.yaml b/.github/workflows/sdk-platform-java-downstream_protobuf_compatibility_check_nightly.yaml index 45b756a6d6b4..e22dc4f5238f 100644 --- a/.github/workflows/sdk-platform-java-downstream_protobuf_compatibility_check_nightly.yaml +++ b/.github/workflows/sdk-platform-java-downstream_protobuf_compatibility_check_nightly.yaml @@ -12,9 +12,9 @@ on: schedule: - cron: '0 1 * * *' # Nightly at 1am +# This job intends to test the compatibility of the Protobuf version against repos not in the +# monorepo. As repos are migrated to the monorepo, this job will be obsolete and turned off. name: sdk-platform-java Downstream Protobuf Compatibility Check Nightly -env: - BUILD_SUBDIR: sdk-platform-java jobs: downstream-protobuf-test: # This job runs if any of the three conditions match: @@ -53,7 +53,7 @@ jobs: - name: Print Protobuf-Java testing version run: echo "Testing with Protobuf-Java v${{ matrix.protobuf-version }}" - name: Perform downstream source compatibility testing - run: REPOS_UNDER_TEST="${{ matrix.repo }}" PROTOBUF_RUNTIME_VERSION="${{ matrix.protobuf-version }}" ./.kokoro/nightly/downstream-protobuf-source-compatibility.sh + run: REPOS_UNDER_TEST="${{ matrix.repo }}" PROTOBUF_RUNTIME_VERSION="${{ matrix.protobuf-version }}" ./sdk-platform-java/.kokoro/nightly/downstream-protobuf-source-compatibility.sh - name: Perform downstream binary compatibility testing - run: REPOS_UNDER_TEST="${{ matrix.repo }}" PROTOBUF_RUNTIME_VERSION="${{ matrix.protobuf-version }}" ./.kokoro/nightly/downstream-protobuf-binary-compatibility.sh + run: REPOS_UNDER_TEST="${{ matrix.repo }}" PROTOBUF_RUNTIME_VERSION="${{ matrix.protobuf-version }}" ./sdk-platform-java/.kokoro/nightly/downstream-protobuf-binary-compatibility.sh From ee47a60a25065ff38586d2971d54faa1dab5cca9 Mon Sep 17 00:00:00 2001 From: Lawrence Qiu Date: Thu, 2 Apr 2026 17:16:49 -0400 Subject: [PATCH 7/8] chore: Add missing nightly files --- sdk-platform-java/.kokoro/nightly/common.sh | 37 ++++++ ...ownstream-protobuf-binary-compatibility.sh | 105 ++++++++++++++++++ ...ownstream-protobuf-source-compatibility.sh | 67 +++++++++++ 3 files changed, 209 insertions(+) create mode 100644 sdk-platform-java/.kokoro/nightly/common.sh create mode 100755 sdk-platform-java/.kokoro/nightly/downstream-protobuf-binary-compatibility.sh create mode 100755 sdk-platform-java/.kokoro/nightly/downstream-protobuf-source-compatibility.sh diff --git a/sdk-platform-java/.kokoro/nightly/common.sh b/sdk-platform-java/.kokoro/nightly/common.sh new file mode 100644 index 000000000000..589ee00bfbf7 --- /dev/null +++ b/sdk-platform-java/.kokoro/nightly/common.sh @@ -0,0 +1,37 @@ +#!/bin/bash +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# For google-cloud-java, only test specific handwritten libraries included in the monorepo. This is to +# help speed up the execution as building the entire repo is an expensive operation. Specify the nested +# `google-cloud-*` path (except for grafeas as it doesn't have one) because maven -pl will only build the +# specified folder (i.e. parent folder) and ignore all the related sub-modules inside +google_cloud_java_handwritten_maven_args="java-grafeas,java-vertexai/google-cloud-vertexai,java-resourcemanager/google-cloud-resourcemanager,java-translate/google-cloud-translate" + +# Checks that the protobuf compatibility scripts provide non-empty input +function validate_protobuf_compatibility_script_inputs { + # Comma-delimited list of repos to test + if [ -z "${REPOS_UNDER_TEST}" ]; then + echo "REPOS_UNDER_TEST Env Var must be set. This script expects a" + echo "comma-delimited list: i.e REPOS_UNDER_TEST=\"java-bigtable,java-bigquery\"" + exit 1 + fi + + # A single version of Protobuf-Java runtime to test + if [ -z "${PROTOBUF_RUNTIME_VERSION}" ]; then + echo "PROTOBUF_RUNTIME_VERSION Env Var must be set. This script expects a single " + echo "Protobuf-Java runtime version i.e. PROTOBUF_RUNTIME_VERSION=\"4.28.3\"" + exit 1 + fi +} \ No newline at end of file diff --git a/sdk-platform-java/.kokoro/nightly/downstream-protobuf-binary-compatibility.sh b/sdk-platform-java/.kokoro/nightly/downstream-protobuf-binary-compatibility.sh new file mode 100755 index 000000000000..5877eb614c73 --- /dev/null +++ b/sdk-platform-java/.kokoro/nightly/downstream-protobuf-binary-compatibility.sh @@ -0,0 +1,105 @@ +#!/bin/bash +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -eo pipefail + +scriptDir=$(realpath "$(dirname "${BASH_SOURCE[0]}")") +source "${scriptDir}/common.sh" + +validate_protobuf_compatibility_script_inputs + +# Declare a map of downstream handwritten libraries and the relevant artifacts to test. The map stores a +# K/V pairing of (Key: repo name, Value: comma separate list of Group ID:Artifact ID pairings). Note: The +# value list doesn't hold the version and this needs to be parsed from the repo's versions.txt file +declare -A repo_linkage_checker_arguments +repo_linkage_checker_arguments["google-cloud-java"]="io.grafeas:grafeas,com.google.cloud:google-cloud-vertexai,com.google.cloud:google-cloud-resourcemanager,com.google.cloud:google-cloud-translate,com.google.api.grpc:grpc-google-cloud-vertexai-v1,com.google.api.grpc:grpc-google-cloud-vertexai-v1beta1,com.google.api.grpc:grpc-google-cloud-resourcemanager-v3,com.google.api.grpc:grpc-google-cloud-translate-v3,com.google.api.grpc:grpc-google-cloud-translate-v3beta1" +repo_linkage_checker_arguments["java-bigtable"]="com.google.cloud:google-cloud-bigtable,com.google.api.grpc:grpc-google-cloud-bigtable-admin-v2,com.google.api.grpc:grpc-google-cloud-bigtable-v2" +repo_linkage_checker_arguments["java-bigquery"]="com.google.cloud:google-cloud-bigquery" +repo_linkage_checker_arguments["java-bigquerystorage"]="com.google.cloud:google-cloud-bigquerystorage,com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1beta1,com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1beta2,com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1,com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1alpha" +repo_linkage_checker_arguments["java-datastore"]="com.google.cloud:google-cloud-datastore,com.google.cloud.datastre:datastore-v1-proto-client,com.google.api.grpc:grpc-google-cloud-datastore-admin-v1" +repo_linkage_checker_arguments["java-firestore"]="com.google.cloud:google-cloud-firestore,com.google.cloud:google-cloud-firestore-admin,com.google.api.grpc:grpc-google-cloud-firestore-admin-v1,com.google.api.grpc:grpc-google-cloud-firestore-v1" +repo_linkage_checker_arguments["java-logging"]="com.google.cloud:google-cloud-logging,com.google.api.grpc:grpc-google-cloud-logging-v2" +repo_linkage_checker_arguments["java-logging-logback"]="com.google.cloud:google-cloud-logging-logback" +repo_linkage_checker_arguments["java-pubsub"]="com.google.cloud:google-cloud-pubsub,com.google.api.grpc:grpc-google-cloud-pubsub-v1" +repo_linkage_checker_arguments["java-pubsublite"]="com.google.cloud:google-cloud-pubsublite,com.google.api.grpc:grpc-google-cloud-pubsublite-v1" +repo_linkage_checker_arguments["java-spanner-jdbc"]="com.google.cloud:google-cloud-spanner-jdbc" +repo_linkage_checker_arguments["java-spanner"]="com.google.cloud:google-cloud-spanner,com.google.cloud:google-cloud-spanner-executor,com.google.api.grpc:grpc-google-cloud-spanner-v1,com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1,com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1,com.google.api.grpc:grpc-google-cloud-spanner-executor-v1" +repo_linkage_checker_arguments["java-storage"]="com.google.cloud:google-cloud-storage,com.google.api.grpc:gapic-google-cloud-storage-v2,com.google.api.grpc:grpc-google-cloud-storage-v2,com.google.cloud:google-cloud-storage-control,com.google.api.grpc:grpc-google-cloud-storage-control-v2" +repo_linkage_checker_arguments["java-storage-nio"]="com.google.cloud:google-cloud-nio" + +# This function requires access to the versions.txt to retrieve the versions for the artifacts +# It will try to match the artifact_id in the versions.txt file and attach it to form the GAV +# The GAV list is required by Linkage Checker as program arguments +function build_program_arguments() { + artifact_list="${repo_linkage_checker_arguments[$1]}" + + for artifact in ${artifact_list//,/ }; do # Split on comma + artifact_id=$(echo "${artifact}" | cut -d ':' -f2) + + # The grep query tries to match `{artifact_id}:{released_version}:{current_version}`. + # The artifact_id must be exact otherwise multiple entries may match + version=$(cat "versions.txt" | grep -E "^${artifact_id}:.*:.*$" | cut -d ':' -f3) + repo_gav_coordinate="${artifact}:${version}" + + # The first entry added is not separated with a comma. Avoids generating `,{ARTIFACT_LIST}` + if [ -z "${linkage_checker_arguments}" ]; then + linkage_checker_arguments="${repo_gav_coordinate}" + else + linkage_checker_arguments="${linkage_checker_arguments},${repo_gav_coordinate}" + fi + done +} + +# TODO(https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/2395): Java 17+ support for Linkage Checker +# cloud-opensource-java contains the Linkage Checker tool +git clone https://github.com/GoogleCloudPlatform/cloud-opensource-java.git +pushd cloud-opensource-java +mvn -B -ntp clean compile -T 1C +# Linkage Checker tool resides in the /dependencies subfolder +pushd dependencies + +# REPOS_UNDER_TEST Env Var accepts a comma separated list of googleapis repos to test. For Github CI, +# this will be a single repo as Github will build a matrix of repos with each repo being tested in parallel. +# For local invocation, you can pass a list of repos to test multiple repos together. +for repo in ${REPOS_UNDER_TEST//,/ }; do # Split on comma + # Perform testing on main (with latest changes). Shallow copy as history is not important + git clone "https://github.com/googleapis/${repo}.git" --depth=1 + pushd "${repo}" + + if [ "${repo}" == "google-cloud-java" ]; then + # The `-am` command also builds anything these libraries depend on (i.e. proto-* and grpc-* sub modules) + mvn clean install -B -V -ntp -T 1C -DskipTests -Dclirr.skip -Denforcer.skip -Dmaven.javadoc.skip \ + -pl "${google_cloud_java_handwritten_maven_args}" -am + else + # Install all repo modules to ~/.m2 (there can be multiple relevant artifacts to test i.e. core, admin, control) + mvn clean install -B -V -ntp -T 1C -DskipTests -Dclirr.skip -Denforcer.skip -Dmaven.javadoc.skip + fi + + + linkage_checker_arguments="" + build_program_arguments "${repo}" + + # Linkage Checker /dependencies + popd + + echo "Artifact List: ${linkage_checker_arguments}" + # The `-s` argument filters the linkage check problems that stem from the artifact + program_args="-r --artifacts ${linkage_checker_arguments},com.google.protobuf:protobuf-java:${PROTOBUF_RUNTIME_VERSION},com.google.protobuf:protobuf-java-util:${PROTOBUF_RUNTIME_VERSION} -s ${linkage_checker_arguments}" + echo "Running Linkage Checker on the repo's handwritten modules" + echo "Linkage Checker Program Arguments: ${program_args}" + mvn -B -ntp exec:java -Dexec.args="${program_args}" -P exec-linkage-checker +done +popd +popd diff --git a/sdk-platform-java/.kokoro/nightly/downstream-protobuf-source-compatibility.sh b/sdk-platform-java/.kokoro/nightly/downstream-protobuf-source-compatibility.sh new file mode 100755 index 000000000000..aaa65b82087e --- /dev/null +++ b/sdk-platform-java/.kokoro/nightly/downstream-protobuf-source-compatibility.sh @@ -0,0 +1,67 @@ +#!/bin/bash +# Copyright 2023 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -eo pipefail + +scriptDir=$(realpath "$(dirname "${BASH_SOURCE[0]}")") +source "${scriptDir}/common.sh" + +validate_protobuf_compatibility_script_inputs + +# REPOS_UNDER_TEST Env Var accepts a comma separated list of googleapis repos to test. For Github CI, +# this will be a single repo as Github will build a matrix of repos with each repo being tested in parallel. +# For local invocation, you can pass a list of repos to test multiple repos together. +for repo in ${REPOS_UNDER_TEST//,/ }; do # Split on comma + # Perform source-compatibility testing on main (latest changes) + git clone "https://github.com/googleapis/$repo.git" --depth=1 + pushd "$repo" + + # Compile with Java 11 and run the tests with Java 8 JVM + mvn compile -T 1C + + # JAVA8_HOME is set by the GH Actions CI + if [ -n "${JAVA8_HOME}" ]; then + surefire_opt="-Djvm=${JAVA8_HOME}/bin/java" + else + # Provide a default value for local executions that don't configure JAVA8_HOME + surefire_opt="-Djvm=${JAVA_HOME}/bin/java" + fi + + # Compile the Handwritten Library with the Protobuf-Java version to test source compatibility + # Run unit tests to help check for any behavior differences (dependant on coverage) + if [ "${repo}" == "google-cloud-java" ]; then + # The `-am` command also builds anything these libraries depend on (i.e. proto-* and grpc-* sub modules) + mvn test -B -V -ntp \ + -Dclirr.skip \ + -Denforcer.skip \ + -Dmaven.javadoc.skip \ + -Denforcer.skip \ + -Dprotobuf.version=${PROTOBUF_RUNTIME_VERSION} \ + -pl "${google_cloud_java_handwritten_maven_args}" -am \ + "${surefire_opt}" \ + -T 1C + else + mvn test -B -V -ntp \ + -Dclirr.skip \ + -Denforcer.skip \ + -Dmaven.javadoc.skip \ + -Denforcer.skip \ + -Dprotobuf.version=${PROTOBUF_RUNTIME_VERSION} \ + "${surefire_opt}" \ + -T 1C + fi + + popd +done \ No newline at end of file From 6259c3d3d390223c72337ffb706f1a12ab644926 Mon Sep 17 00:00:00 2001 From: Lawrence Qiu Date: Thu, 2 Apr 2026 17:21:12 -0400 Subject: [PATCH 8/8] chore: Remove the migrated storage repos --- ...rm-java-downstream_protobuf_compatibility_check_nightly.yaml | 2 -- 1 file changed, 2 deletions(-) diff --git a/.github/workflows/sdk-platform-java-downstream_protobuf_compatibility_check_nightly.yaml b/.github/workflows/sdk-platform-java-downstream_protobuf_compatibility_check_nightly.yaml index e22dc4f5238f..9d6e40d734d6 100644 --- a/.github/workflows/sdk-platform-java-downstream_protobuf_compatibility_check_nightly.yaml +++ b/.github/workflows/sdk-platform-java-downstream_protobuf_compatibility_check_nightly.yaml @@ -31,8 +31,6 @@ jobs: - java-firestore - java-pubsub - java-pubsublite - - java-storage - - java-storage-nio # Default Protobuf-Java versions to use are specified here. Without this, the nightly workflow won't know # which values to use and would resolve to ''. protobuf-version: ${{ fromJSON(format('[{0}]', inputs.protobuf_runtime_versions || '"4.33.5"')) }}