Changes that add and integrate code coverage support into XDMoD's CI pipeline by ryanrath · Pull Request #1443 · ubccr/xdmod

ryanrath · 2020-09-09T18:10:04Z

NOTE: this PR takes the place of #1350

All of the functional changes are the work of Zi-3 all that I've (@ryanrath) done is clean things up a little.

Description

Added four new scripts responsible for installing & configuring XDebug,
generating the code coverage data from php file execution, and generating the final
reports.
Updated shippable.yml so that XDebug is installed / Code Coverage is only generated during the nightly
test build.
Updated numerous UI tests to work more reliably when XDebug is installed ( i.e. their timeouts were increased )
Updated a couple of UI test selectors to minimize the effects of latency on particular tests.
Added a new webdriverhelper function called clickSelectorAndWaitForMask as this was needed for a test or two.
Updated the timeouts for a few other webdriverhelper functions.

Motivation and Context

Adding code coverage will allow us to get a general sense of where we are lacking test code & possibly where we have unused code thus giving us a few more tools to help streamline / increase the reliability of XDMoD.

Tests performed

All the automated tests for an install / upgrade with & without XDebug installed.

Checklist:

The pull request description is suitable for a Changelog entry
The milestone is set correctly on the pull request
The appropriate labels have been added to the pull request

jpwhite4 · 2020-09-10T01:21:02Z

-                        browser.setValue(input, setting.updated);
-                        const updatedValue = browser.getValue(input);
-                        expect(updatedValue).to.equal(setting.updated);
+                        for (let i = 0; i < 100; i++) {


This will hide test failures if the exception is thrown more than 100 times

If you want to make multiple attempts then you'll need to (a) add a short pause between tries (e.g. 10ms), and (b) make sure to throw an exception if the max attempts is reached.

I've ended up removing these for loops as the ended up being unnecessary with the latest updates.

jpwhite4 · 2020-09-10T01:23:33Z

-                        browser.click(inputTrigger);
+                        setTimeout(function () {
+                            browser.click(inputTrigger);
+                        }, 3000);


three seconds seems like a long time to wait here. Why was it increased so much?

This and the previous waitForVisible could also be combined into a waitAndClick

I've actually ended up removing the setTimeout calls and everything still seems to be passing both w/ and w/ out XDebug.

plessbd

Part of me wants to do a completely separate PR just for all the test updates...But at the same time the reason they were done is because of the timing issues with xdebug enabled.
I am so conflicted...

plessbd

I think this looks good as long as it still passes shippable (cause travis is the broken at the moment)

- Added four new scripts responsible for installing & configuring XDebug, generating the code coverage data from php file execution, and generating the final reports. - Updated `shippable.yml` so that XDebug / Code Coverage is only generated during the nightly test build. - Updated numerous UI tests to work more reliably when XDebug is installed ( i.e. their timeouts were increased ) - Updated a couple of UI test selectors to minimize the effects of latency on particular tests. - Added a new webdriverhelper function called `clickSelectorAndWaitForMask` as this was a commonly feature during developing this PR. - Updated the timeout for a few other webdriverhelper functions.

- It was found while running these tests on the CentOS8 image, that the `setupBeforeClass` function was not being called as expected, this small addition will catch if the required `self::$hashFilePath` variable has been set, and if not then run the `setupBeforeClass` function so that it is populated.

- Since we're now running this script on CentOS7|8 and Rocky8.5 containers we need to take into account the different software versions / configuration options that must be set to successfully collect the code coverage data generated by our tests.

By installing Xdebug and setting up code coverage before running bootstrap we'll generate code coverage information for our install / upgrade code including the ETL code.

ryanrath · 2022-09-02T20:43:35Z

I force triggered the code coverage generation in the last commit to make sure that it works. Before I set it back I just wanted to document the code coverage artifacts from each build:

ryanrath added qa / testing Updates/additions to tests Category:General General labels Sep 9, 2020

ryanrath added this to the 9.5.0 milestone Sep 9, 2020

ryanrath requested review from jtpalmer and plessbd and removed request for jtpalmer September 9, 2020 19:33

jpwhite4 requested changes Sep 10, 2020

View reviewed changes

plessbd reviewed Sep 10, 2020

View reviewed changes

Comment thread tests/ui/test/specs/xdmod/internalDashboard.js Outdated

ryanrath force-pushed the zrcode_bkup branch 5 times, most recently from 180516d to 02428b3 Compare September 15, 2020 19:56

ryanrath requested review from jpwhite4 and removed request for jpwhite4 September 16, 2020 17:51

plessbd reviewed Sep 22, 2020

View reviewed changes

Comment thread html/gui/js/report_builder/ReportsOverview.js

Comment thread tests/ui/test/specs/xdmod/internalDashboard.js Outdated

plessbd previously approved these changes Nov 2, 2020

View reviewed changes

ryanrath dismissed plessbd’s stale review via db77bb7 February 22, 2021 16:14

ryanrath force-pushed the zrcode_bkup branch from 6f42a7b to db77bb7 Compare February 22, 2021 16:14

jpwhite4 modified the milestones: 9.5.0, 10.0.0 Mar 8, 2021

ryanrath force-pushed the zrcode_bkup branch from db77bb7 to c94831e Compare March 26, 2021 14:04

ryanrath requested a review from eiffel777 May 5, 2021 18:21

jtpalmer changed the base branch from xdmod9.5 to xdmod10.0 June 14, 2021 18:54

ryanrath force-pushed the zrcode_bkup branch from d2afac6 to 0cedb0b Compare November 2, 2021 18:00

ryanrath force-pushed the zrcode_bkup branch from c5ba815 to 2ff4108 Compare December 22, 2021 15:46

ryanrath modified the milestones: 10.0.0, 10.5.0 Jan 7, 2022

ryanrath force-pushed the zrcode_bkup branch from 10d2261 to 201668a Compare January 7, 2022 20:40

jtpalmer changed the base branch from xdmod10.0 to xdmod10.5 March 24, 2022 14:50

ryanrath force-pushed the zrcode_bkup branch from b17affe to 0db3b59 Compare April 6, 2022 15:23

Zi-3 and others added 7 commits September 1, 2022 15:28

Updating CircleCI config to latest

71e1474

Updating config to work with new scheduled jobs

2aea510

Updating ci config

94e2c97

Moving the code coverage setup to before the bootstrap

dcaac39

By installing Xdebug and setting up code coverage before running bootstrap we'll generate code coverage information for our install / upgrade code including the ETL code.

ryanrath force-pushed the zrcode_bkup branch from 120462f to dcaac39 Compare September 2, 2022 15:57

ryanrath added 4 commits September 2, 2022 13:17

Force testing code coverage

7e79a95

We don't need to stop / start services now

5c58fbb

Updating the builds that are excluded

d5b678b

Moving the code coverage install step

3b28479

Re-adding the build trigger

1e71cdb

jpwhite4 marked this pull request as draft February 3, 2023 14:49

jpwhite4 modified the milestones: 10.5.0, 11.0.0 May 24, 2023

bluebrey mentioned this pull request Aug 14, 2023

Playwright coverage #1737

Closed

3 tasks

jpwhite4 modified the milestones: 11.0.0, 11.5.0 Jun 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Changes that add and integrate code coverage support into XDMoD's CI pipeline#1443

Changes that add and integrate code coverage support into XDMoD's CI pipeline#1443
ryanrath wants to merge 12 commits intoubccr:xdmod10.5from
Zi-3:zrcode_bkup

ryanrath commented Sep 9, 2020

Uh oh!

jpwhite4 Sep 10, 2020

Uh oh!

jpwhite4 Sep 10, 2020

Uh oh!

ryanrath Sep 16, 2020

Uh oh!

jpwhite4 Sep 10, 2020

Uh oh!

plessbd Sep 10, 2020

Uh oh!

ryanrath Sep 15, 2020

Uh oh!

Uh oh!

Uh oh!

plessbd left a comment

Uh oh!

Uh oh!

Uh oh!

plessbd left a comment

Uh oh!

ryanrath commented Sep 2, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

ryanrath commented Sep 9, 2020

Description

Motivation and Context

Tests performed

Checklist:

Uh oh!

jpwhite4 Sep 10, 2020

Choose a reason for hiding this comment

Uh oh!

jpwhite4 Sep 10, 2020

Choose a reason for hiding this comment

Uh oh!

ryanrath Sep 16, 2020

Choose a reason for hiding this comment

Uh oh!

jpwhite4 Sep 10, 2020

Choose a reason for hiding this comment

Uh oh!

plessbd Sep 10, 2020

Choose a reason for hiding this comment

Uh oh!

ryanrath Sep 15, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

plessbd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

plessbd left a comment

Choose a reason for hiding this comment

Uh oh!

ryanrath commented Sep 2, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants