Skip to content

Cdap e2e framework update#667

Draft
vedanshugarg04 wants to merge 67 commits into
data-integrations:developfrom
cloudsufi:cdap-e2e-framework-update
Draft

Cdap e2e framework update#667
vedanshugarg04 wants to merge 67 commits into
data-integrations:developfrom
cloudsufi:cdap-e2e-framework-update

Conversation

@vedanshugarg04
Copy link
Copy Markdown

No description provided.

itsankit-google and others added 30 commits January 15, 2025 14:03
…e-e2e-workflow

fix e2e workflow to run on release branches
…952ef738031950e35e8dc195d3cd9e2f8dd4b91

[🍒][CDAP-21093] Add ExternalDocumentationLink in abstract sink
…de940cb40bf9dc23e2579dc40609147270ecadf

[🍒][PLUGIN-1845] Return null for ErrorDetailsProvider by default
…gin.version-2.13.0-snapshot

Bump cdap.plugin.version to 2.13.0-SNAPSHOT
…9a2af04cfa09841772ea7c92eca12c57b701812

[🍒] Revert cdap.plugin.version to 2.13.0-SNAPSHOT
…d7e9cd90ae48bd62b531582c05c7e194c956000

[🍒] Remove DBErrorDetailsProvider
…6702fdee831722ff326a316af5441f08489c2b3

[🍒][PLUGIN-1835] Add OracleErrorDetailsProvider
…3cfd3af30e32290c7a327947cbf56538055b212

[🍒][PLUGIN-1847] Add SqlServerErrorDetailsProvider
…d75ef3a945240a6771940f46edef3096b76e23f

[🍒][PLUGIN-1851] Show correct DB name in error string
…4a420a7cc37e666e3de4212bc1483ba5e46758f

[🍒] e2e tests pipeline preview error fix changes
…e-snapshot

Removing snapshot for 6.11 release
…version-update

[🍒] Updating cdap version to 6.11
…b368503054ab587861f65909f0c9c71a5793912

[🍒][PLUGIN-1873] Error Management AbstractDBAction
Pushpender Saini and others added 28 commits May 5, 2025 22:13
…a56874ce8f8c8c19db8c1d73af71a1d1ad34141/rb-1.12

[🍒][PLUGIN-1888] Skip field validation for compatiblity
…lugins

Add transaction isolation level to PostgreSQL, MYSQL and MSSQL Plugins.
…4650e13274f73e8c173c2895170f512ef4245ee

[🍒][Plugin-1779] Add TRANSACTION_ISOLATION_LEVEL config in MySQL, PostgreSQL & SQL Server plugins
…00f23f51a8e6834e8e0fd680c0c6effed8ba9ed

[🍒][PLUGIN-1815] Commit/Rollback added in Committer.
…napshot/rc-1-12-2

Remove Snapshot for release 1.12.2 database plugins
… like a flag for Backward compatibility issues for Timestamp and number (precisionless)
…y_PLUGIN-1893_6_11

[cherry-pick][6.11][PLUGIN-1893] Adding fields in Oracle source and connector which acts like a flag for Backward compatibility issues for Timestamp and number (precisionless)
…ing_snap_for_hub_rel

removing Snapshot for hub release
…aven-publishing-cherrypick

[🍒] Add required for maven central publishing
…hot-remove

Removing snapshot for CDAP 6.11.1 release
…ypick/6552c2878531670975e0ffc2a1dbc95f9ac4ec47

[🍒] Verify Goal Removal
…tampLTZ-change

[cherrypick] [PLUGIN-1932] Add hidden treatTimestampLTZAsTimestamp flag to map Oracle TIMESTAMP WITH LOCAL TIME ZONE to TIMESTAMP
@google-cla
Copy link
Copy Markdown

google-cla Bot commented May 22, 2026

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces significant updates across multiple database plugins, including the addition of transaction isolation level support, enhanced error reporting with external documentation links, and the implementation of upsert operations for Oracle. It also refactors the core database commons to improve connection management through task-level commit logic. Review feedback highlights critical issues, including a reference to an undefined variable in the Oracle plugin, potential SQL syntax errors when constructing upsert queries, and incorrect argument ordering in error message formatting.

if (!errorMessage.endsWith(".")) {
errorMessage = errorMessage + ".";
}
errorMessage = String.format("%s For more details, see %s", errorMessageWithDetails, errorMessage);
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The externalDocumentationLink is not included in the formatted error message, even though it is checked for presence on line 322. Additionally, the arguments in String.format appear to be incorrect (swapped) compared to the implementation in AbstractDBSource, which will lead to a confusing error summary.

Suggested change
errorMessage = String.format("%s For more details, see %s", errorMessageWithDetails, errorMessage);
errorMessage = String.format("%s For more details, see %s", errorMessage, externalDocumentationLink);

Comment on lines +77 to +94
boolean firstUpdateColumn = true;
for (String fieldName : fieldNames) {
boolean isKeyColumn = false;
for (String listKey : listKeys) {
String listKeyNoQuote = listKey.replace("\"", "");
if (listKeyNoQuote.equals(fieldName)) {
isKeyColumn = true;
break;
}
}
if (!isKeyColumn) {
if (!firstUpdateColumn) {
query.append(", ");
}
query.append("target.").append(fieldName).append(" = source.").append(fieldName);
firstUpdateColumn = false;
}
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If all columns in fieldNames are also present in listKeys, the UPDATE SET clause will be empty, which results in invalid Oracle SQL syntax for the MERGE statement. The WHEN MATCHED clause should only be appended if there are non-key columns to update. Additionally, the nested loop for key lookup is inefficient ($O(N \times M)$); using a Set for lookups is recommended.

Suggested change
boolean firstUpdateColumn = true;
for (String fieldName : fieldNames) {
boolean isKeyColumn = false;
for (String listKey : listKeys) {
String listKeyNoQuote = listKey.replace("\"", "");
if (listKeyNoQuote.equals(fieldName)) {
isKeyColumn = true;
break;
}
}
if (!isKeyColumn) {
if (!firstUpdateColumn) {
query.append(", ");
}
query.append("target.").append(fieldName).append(" = source.").append(fieldName);
firstUpdateColumn = false;
}
}
StringBuilder updateClause = new StringBuilder();
boolean firstUpdateColumn = true;
java.util.Set<String> keySet = new java.util.HashSet<>();
for (String listKey : listKeys) {
keySet.add(listKey.replace("\"", ""));
}
for (String fieldName : fieldNames) {
if (!keySet.contains(fieldName)) {
if (!firstUpdateColumn) {
updateClause.append(", ");
}
updateClause.append("target.").append(fieldName).append(" = source.").append(fieldName);
firstUpdateColumn = false;
}
}
if (!firstUpdateColumn) {
query.append("WHEN MATCHED THEN UPDATE SET ").append(updateClause).append(" ");
}

String fieldName, int fieldIndex) throws SQLException {
int sqlType = columnTypes.get(fieldIndex).getType();
int sqlIndex = fieldIndex + 1;
int sqlType = modifiableColumnTypes.get(fieldIndex).getType();
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The variable modifiableColumnTypes is used but does not appear to be defined in this class or its parent. It should likely be columnTypes, which is the field used in the previous version of the code and in OracleSinkDBRecord. If a modifiable version was intended, ensure it is properly declared and initialized.

Suggested change
int sqlType = modifiableColumnTypes.get(fieldIndex).getType();
int sqlType = columnTypes.get(fieldIndex).getType();

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants