Reduce Kotlin-compose-wasm iteration count

[kmiller68]

Kotlin-compose-wasm is the longest running test in JS3 by a pretty wide margin. However after the first few iterations almost all code is in the most optimizing compiler in the respective engines.

This change reduces the iterations to 5 and sets adjusts the worst case count accordingly.

[netlify]

✅ Deploy Preview for webkit-jetstream-preview ready!

Name	Link
🔨 Latest commit	ea72931
🔍 Latest deploy log	https://app.netlify.com/projects/webkit-jetstream-preview/deploys/6914ab6f3a376d000770c411
😎 Deploy Preview	https://deploy-preview-225--webkit-jetstream-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[kmiller68]

CC @eqrion

[danleh]

Fine by me. I also briefly checked that the scores on x64 with all three shells are roughly the same. I am however a bit worried about noise/flaky results with worstCaseCount: 1, how about compromising on 10 iterations with 2 worst case?

[danleh]

Are the doxbee changes intended (or accidental)?

[kmiller68]

Whoops, accidental, will revert.

[kmiller68]

I am however a bit worried about noise/flaky results with worstCaseCount: 1, how about compromising on 10 iterations with 2 worst case?

That would make it still quite long (~4 sec) on my M2 MBP (probably a lot worse on phones/older CPUs). Let me run a variance analysis on the worst case count for this benchmark and get back to you. It seems somewhat unlikely it's dramatically worse than Firsts though, in general, especially since the iteration time is so long.

[kmiller68]

Some numbers:

====================================================================================================
JetStream3 Benchmark Power Analysis
Significance Level (α): 0.05
Statistical Power: 0.8
Detection multiple: 1.02
====================================================================================================

Chrome:
Kotlin-compose-wasm
----------------------------------------------------------------------------------------------------

  Category: RawTime - Worst
    Sample size (n): 40
    Mean: 392.1590
    Std Dev: 8.8443
    CV: 2.26%
    Effect Size (Cohen's d): 0.887
    Sample size needed: 20.966

  Category: Worst
    Sample size (n): 20
    Mean: 12.7520
    Std Dev: 0.1661
    CV: 1.30%
    Effect Size (Cohen's d): 1.535
    Sample size needed: 7.747
    
Firefox:
Kotlin-compose-wasm
----------------------------------------------------------------------------------------------------

  Category: RawTime - Worst
    Sample size (n): 40
    Mean: 1361.0875
    Std Dev: 68.3518
    CV: 5.02%
    Effect Size (Cohen's d): 0.398
    Sample size needed: 99.940

  Category: Worst
    Sample size (n): 20
    Mean: 3.6805
    Std Dev: 0.1648
    CV: 4.48%
    Effect Size (Cohen's d): 0.447
    Sample size needed: 79.664

Safari: 
Kotlin-compose-wasm
----------------------------------------------------------------------------------------------------

  Category: RawTime - Worst
    Sample size (n): 40
    Mean: 491.7625
    Std Dev: 15.5822
    CV: 3.17%
    Effect Size (Cohen's d): 0.631
    Sample size needed: 40.386

  Category: Worst
    Sample size (n): 20
    Mean: 10.1751
    Std Dev: 0.2878
    CV: 2.83%
    Effect Size (Cohen's d): 0.707
    Sample size needed: 32.380
    

[kmiller68]

We can discuss offline but those numbers don't seem too bad relative to other Worsts in the benchmark. Many of the line items are on the 1000s of samples needed.

[kmiller68]

Per discussion at the sync, this seems like an acceptable noise rate for Worst. Will wait until next week to merge unless @cbr comments (or signs off) before then.