[cleaner] Better distribution of files among child cleaner processes

[pmoravec]

Static split of input files often meant uneven distribution of load among the child processes. That caused just one process being busy at the end for many minutes, which ruins concurrency benefits.

Using multiprocessing.[Process|Queue] allow to:

1. order files per sizes, starting from biggest (with the assumption of
   longest cleanup time)
2. dynamic fetching of new file by a child process, only when it gets
   idle

Relevant: #4227
Closes: #4228

---

Please place an 'X' inside each '[]' to confirm you adhere to our Contributor Guidelines

• Is the commit message split over multiple lines and hard-wrapped at 72 characters?
• Is the subject and message clear and concise?
• Does the subject start with [plugin_name] if submitting a plugin patch or a [section_name] if part of the core sosreport code?
• Does the commit contain a Signed-off-by: First Lastname email@example.com?
• Are any related Issues or existing PRs properly referenced via a Closes (Issue) or Resolved (PR) line?
• Are all passwords or private data gathered by this PR obfuscated?

[pmoravec]

Performance comparison (on one small testing sosreport, with default 4 workers/jobs):

• original runtime: 45s
• improved runtime: 32s, hence improved by 1/3

For bigger sosreports, I assume smaller, yet evident improvement (I plan to share some stats once my performance tests complete).

[packit-as-a-service]

Congratulations! One of the builds has completed. 🍾

You can install the built RPMs by following these steps:

• sudo dnf install -y 'dnf*-command(copr)'
• dnf copr enable packit/sosreport-sos-4228
• And now you can install the packages.

Please note that the RPMs should be used only in a testing environment.

[pmoravec]

Some tests are failing in a weird way, moving PR to draft to investigate that.

[pmoravec]

Any idea why "Report Stage One - debian-13" repeatedly fails on ModuleNotFoundError: No module named 'pkg_resources'?

[pmoravec]

Kindly asking for a review anybody from the list of people I marked as reviewers.

This PR prevents cases when whole cleaner waits a long time for one child process that is busy with its remaining work, while the other child processes are done already.

Two improvements made:

• dynamic assigning of next file to obfuscate to idle child processes, instead of originl static assignment at the startup
• ordering of files from biggest to slowest, to make the "waiting for last child" time yet smaller

This requires using multiprocessing instead of ProcessPoolExecutor, but imho that is no big deal.

[jcastill]

+1 to add debug info, it could be very useful. But perhaps, at least in this case, it could be added directly inside should_skip_file(), where it looks like we don't log the skip either

[pmoravec]

Hmm.. that method is an auxiliary one and those usually dont have logs.. but as it is called just from here and sometimes it can be beneficial to know the reason for the skip ("not a file or symlink, skipping" != "filename in skip list, skipping"), it makes sense.

[jcastill]

Honestly, either way, inside the function or in the 'if's that you have here, I don't have a strong preference. But for sure these messages would be very useful

[pmoravec]

Kindly asking for a PR review here.

Feedback about exceptions is incorporated, this alone improves performance by several percents and namely it makes the cleaner runtime more deterministic.

[pmoravec]

Kindly asking @TurboTurtle and/or @arif-ali for a review.