Use lateral join with subquery on user_statistics table in account_view for network stats#12631
Conversation
…tatistics table in account_view for netstats
|
@blueorangutan package |
|
@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## 4.22 #12631 +/- ##
============================================
- Coverage 17.62% 17.62% -0.01%
+ Complexity 15668 15665 -3
============================================
Files 5917 5917
Lines 531255 531255
Branches 64951 64951
============================================
- Hits 93639 93616 -23
- Misses 427077 427100 +23
Partials 10539 10539
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
@DaanHoogland @nvazquez & Others - Should we update MySQL version to 8.0.14 in the compatibility matrix here: https://docs.cloudstack.apache.org/en/4.22.0.0/releasenotes/compat.html ? |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16799 |
|
Thanks @sureshanaparti - actually the support for Mysql 8.4 is planned for 4.22.1 (#12417) as 8.0 is reaching EOL in April this year, so I think we can hold this PR until the 8.4 support is verified (and a doc PR won't be needed for 8.0.14 as the compatibility matrix will be updated to 8.4) |
in that case we need to update the 4.20 branch as well. |
|
@blueorangutan test |
|
@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
|
[SF] Trillian test result (tid-15450)
|
There was a problem hiding this comment.
Pull request overview
This PR optimizes the account_view database view by replacing the separate account_netstats_view with a LATERAL JOIN directly in the main view. The change addresses a performance issue where the MySQL optimizer could not push WHERE predicates through the nested view with GROUP BY, causing full table scans on the user_statistics table. The LATERAL JOIN allows correlated subqueries where predicates can be applied before aggregation, enabling efficient use of the existing UNIQUE KEY on account_id in the user_statistics table.
Changes:
- Replaced
account_netstats_viewreference with LATERAL JOIN subquery inaccount_view - Deleted the
cloud.account_netstats_view.sqlview definition file - Added DROP VIEW statement for
account_netstats_viewin cleanup migration script
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| engine/schema/src/main/resources/META-INF/db/views/cloud.account_view.sql | Replaced left join with account_netstats_view with a LATERAL JOIN containing an inline subquery on user_statistics table |
| engine/schema/src/main/resources/META-INF/db/views/cloud.account_netstats_view.sql | Deleted the view file as it's no longer needed |
| engine/schema/src/main/resources/META-INF/db/schema-42200to42210-cleanup.sql | Added DROP VIEW statement to remove account_netstats_view during upgrade |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| left join lateral ( | ||
| select | ||
| coalesce(sum(`user_statistics`.`net_bytes_received` + `user_statistics`.`current_bytes_received`), 0) AS `bytesReceived`, | ||
| coalesce(sum(`user_statistics`.`net_bytes_sent` + `user_statistics`.`current_bytes_sent`), 0) AS `bytesSent` | ||
| from | ||
| `cloud`.`user_statistics` | ||
| where | ||
| `user_statistics`.`account_id` = `account`.`id` | ||
| ) AS `account_netstats` ON TRUE |
There was a problem hiding this comment.
This PR introduces a LATERAL JOIN which requires MySQL 8.0.14 or later. This is a significant change that creates a new minimum MySQL version requirement for CloudStack. This requirement should be documented in release notes, upgrade documentation, and installation documentation to ensure users are aware before upgrading. Consider adding a migration script check or warning if the MySQL version is below 8.0.14.
| left join lateral ( | |
| select | |
| coalesce(sum(`user_statistics`.`net_bytes_received` + `user_statistics`.`current_bytes_received`), 0) AS `bytesReceived`, | |
| coalesce(sum(`user_statistics`.`net_bytes_sent` + `user_statistics`.`current_bytes_sent`), 0) AS `bytesSent` | |
| from | |
| `cloud`.`user_statistics` | |
| where | |
| `user_statistics`.`account_id` = `account`.`id` | |
| ) AS `account_netstats` ON TRUE | |
| left join ( | |
| select | |
| a.id AS `account_id`, | |
| coalesce(sum(`us`.`net_bytes_received` + `us`.`current_bytes_received`), 0) AS `bytesReceived`, | |
| coalesce(sum(`us`.`net_bytes_sent` + `us`.`current_bytes_sent`), 0) AS `bytesSent` | |
| from | |
| `cloud`.`account` a | |
| left join `cloud`.`user_statistics` us ON us.account_id = a.id | |
| group by a.id | |
| ) AS `account_netstats` ON `account_netstats`.`account_id` = `account`.`id` |
4 participants
Description
This PR uses lateral join (introduced in MySQL 8.0.14) with subquery on user_statistics table in account_view for network stats.
There was a performance issue while querying account_view, which performs full table scans on the user_statistics table, resulting in execution time.
The original account_view used account_netstats_view which aggregates network statistics from the user_statistics table. MySQL's query optimizer could not push WHERE predicates through the nested view with GROUP BY, causing it to materialize the entire derived table before filtering. LATERAL JOIN (supported from MySQL 8.0.14+) allows correlated subqueries where the predicate can be applied before aggregation, avoiding materialization.
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
How Has This Been Tested?
SHOW CREATE VIEW account_view;SHOW CREATE VIEW account_netstats_view;SELECT * FROM account_view WHERE id = 3;EXPLAIN ANALYZE SELECT * FROM account_view WHERE id = 3 \GHow did you try to break this feature and the system with this change?