Use pyproject.toml instead of setup.cfg, pytest.ini & requirements.txt and other Python related config files

[stv0g]

Modern Python is slowly adopting pyproject.toml as a single place to define a Python package and related tool settings like black for formatting, flake8 for linting and pytest for testing.

Lets try to move as much of the Python-related project configuration to this file and unclutter the root of the repo.

This PR does:

• Merge requirements*.txt, pytest.ini and setup.cfg files into pyproject.toml
• Update shell scripts and Dockerfiles to use pyproject.toml
• Introduce a pre-commit configuration file
• Add a GitHub actions workflow for running pre-commit also in the CI
• Apply/fix pre-commit checks to all files in the repo:
  • Harmonize file endings (single empty line)
  • Remove trailing whitespaces
  • Apply formatting of all Python code (also in Jupyter notebooks)
• Remove obsolete CI jobs from .gitlab-ci.yml

[stv0g]

I am interested in your opinions @leonardocarreras and @m-mirz.
What do you think about consolidate all the Python related project files in pyproject.toml

[m-mirz]

Good idea 👍

[leonardocarreras]

I think it is a great idea 🙌🏻

[stv0g]

This is ready for review :)

I've also added pre-commit as a tool to perform several checks against changes which are staged for commit. Such as removing trailing whitespaces, checking JSON, YAML and XML syntax and Python / C++ code formatting.

While doing so, I realized that much of the Python code was ugly formatted.
We are now using the de-facto standard Python formatter Black.
It even can format Python code with in Jupyter notebooks :)

[stv0g]

@m-mirz I guess the SonarCloud warnings are false-positives. Its at least not code which I've touched.

[stv0g]

Hi @martinmoraga @leonardocarreras @dinkelbachjan,

Markus has reviewed and approved this change. However, we would like to find more approvals before going ahead and merging it.

Do you have a second to look at this change?

[leonardocarreras]

Here looks like we are doing a different thing, not only installing dependencies but also the package in the currently active folder (dpsim). Is this the intended behaviour?

[m-mirz]

Actually, this looks fine to me @stv0g @leonardocarreras

[leonardocarreras]

Any particular reason for the difference here about only pip3 install . vs .[test,docs] in the packaging/Shell/install-fedora-deps-dev.sh

[m-mirz]

@stv0g Doesn't this install the dpsim package as well and not just deps?

[leonardocarreras]

Any particular reason for the difference here about only pip3 install . vs .[test,docs] in the packaging/Shell/install-fedora-deps-dev.sh

[m-mirz]

@stv0g Here I would also expect only the installation of deps and not the dpsim package.

[leonardocarreras]

Hi @stv0g! Thanks for the extra formatting, I think the code will look a lot more standardized now... From the side of the code it looks good to me in general, I would like some more opinions still on the how we do...

[leonardocarreras]

Side topic: Apparently, the more changes are here, the more Sonar complains and demands higher quality from reformatted existing code...

[sonarqubecloud]

Quality Gate failed

Failed conditions
5 Security Hotspots
E Security Review Rating on New Code (required ≥ A)
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarCloud

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

[m-mirz]

@stv0g @leonardocarreras I think this can be merged after the question regarding pip install have been resolved.

[stv0g]

I converted this PR to a draft, since I am splitting it up into smaller changesets and new PRs.

Once, everthing is merged, I will close this one.