feat: Move test formatter from .github/workflows/format_diff.py to python-gardenlinux-lib by Leon-hk · Pull Request #294 · gardenlinux/python-gardenlinux-lib

Leon-hk · 2026-01-07T10:12:59Z

What this PR does / why we need it:

For the still unmerged reproducibility checker, I've written a format_diff.py script which generates a readme report for the workflow run
This script manually parsed the flavors directory and created feature trees. But this can also be way easier and more reliable done using the python-gardenlinux-lib
This PR ports the readme generator to the python-gardenlinux-lib in order to clean up the workflows directory, to use features of the library and to be able to test the output with simple unit tests

…thon-gardenlinux-lib

codecov · 2026-01-08T11:14:06Z

Codecov Report

❌ Patch coverage is 95.45455% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.06%. Comparing base (a776256) to head (a15ba3e).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
...gardenlinux/features/reproducibility/comparator.py	89.06%	7 Missing ⚠️
...c/gardenlinux/features/reproducibility/__main__.py	89.36%	5 Missing ⚠️
...ardenlinux/features/reproducibility/diff_parser.py	98.73%	1 Missing ⚠️
...nux/features/reproducibility/markdown_formatter.py	99.13%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #294      +/-   ##
==========================================
+ Coverage   91.51%   92.06%   +0.54%     
==========================================
  Files          42       46       +4     
  Lines        2158     2457     +299     
==========================================
+ Hits         1975     2262     +287     
- Misses        183      195      +12

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

NotTheEvilOne · 2026-01-17T09:54:40Z

src/gardenlinux/features/difference_formatter.py

+    commit: str
+
+
+class Formatter(object):


File name and class name should be the same. Furthermore the class name should indicate what it formats or into which format. I guess MarkdownFormatter might be most suitable as we do not intend to support multiple output formats anyway.

To not loose what it formats a package called gardenlinux.features.reproducibility might be suitable

NotTheEvilOne · 2026-01-17T09:55:10Z

src/gardenlinux/features/reproducibility/markdown_formatter.py

This file mixes output and parsing logic. Separating the functionality might allow reusability.

NotTheEvilOne · 2026-01-17T10:00:44Z

src/gardenlinux/features/difference_formatter.py

+        diff_dir = pathlib.Path(gardenlinux_root).joinpath(diff_dir)
+
+        self._all = set()
+        self._flavors = os.listdir(diff_dir)


This variable indicates a different content then what it contains. Based on GardenLinux naming convention a flavor is a specific set of features as a string. E.g. gcp-gardener_prod_trustedboot resulting in a GardenLinux canonical name if appended with the CPU architecture, version and commit.

This variable contains files produced during one build if I understand it correctly.

NotTheEvilOne · 2026-01-17T10:01:42Z

src/gardenlinux/features/reproducibility/markdown_formatter.py

+
+    remove_arch = re.compile("(-arm64|-amd64)$")
+
+    def __init__(


The constructor does way too much. Usually a constructor should only set-up an instance. If needed it might call class methods to increase readability.

NotTheEvilOne · 2026-01-17T10:04:02Z

src/gardenlinux/features/difference_formatter.py

+
+        self._parser = Parser(gardenlinux_root, feature_dir_name, logger)
+        if gardenlinux_root is None:
+            gardenlinux_root = self._parser._GARDENLINUX_ROOT


Accessing private members are discouraged. If needed Parser can be extended to provide the directory information used at an instance of it.

NotTheEvilOne · 2026-01-17T10:11:16Z

src/gardenlinux/features/difference_formatter.py

+        :since:  1.0.0
+        """
+
+        if len(items) <= 10:


Magic numbers - please use a constant for the limit :)

NotTheEvilOne · 2026-01-17T10:12:35Z

src/gardenlinux/features/difference_formatter.py

+                        found.update(fnd)
+                        s += "  " + st.replace("\n", "\n  ") + "\n"
+        # Remove last linebreak as the last line can contain spaces
+        return "\n".join(s.split("\n")[:-1]), found


Wouldn't rstrip() be more appropriate here?

NotTheEvilOne · 2026-01-17T10:14:19Z

src/gardenlinux/features/difference_formatter.py

+            first = True
+            tree = None
+            for flavor in flavors:
+                if not flavor.startswith("bare-"):


Hidden magic is always a request for hard to debug issues later on. Please add # @TODO comments where applicable.

NotTheEvilOne · 2026-01-17T10:15:52Z

src/gardenlinux/features/difference_formatter.py

+        """
+
+        with open(
+            self._parser._feature_base_dir.joinpath(f"{node}/info.yaml"), "r"


You already accessed the private root directory variable from Parser. Best practice (if not otherwise possible) would be to store it in this instance in __init__() for later reuse.

NotTheEvilOne · 2026-01-17T10:18:18Z

src/gardenlinux/features/difference_formatter_main.py

I would expect gl-diff to at least additionally produce the diff files formatted here as well. Based on if that would be feasible the file name should either indicate it's handling flavor build differential content or it's only parsing and providing output for it.

Leon-hk · 2026-01-29T10:40:25Z

Thanks for your detailed feedback. I have heavily restricted the formatter and implemented all your comments. It's still a massive PR, as it also contains the difference generator now. So take your time!

I was not able to find a python package or better way to unpack .oci containers, so I've implemented unpacking logic myself. It works so far.

NotTheEvilOne

In general this is a nice feature. I'm not sure if gl-diff is the most suitable command line tool name for it. As you compare OCI images or tar files based on features you may want to name it something similar to gl-feature-fs-diff?

NotTheEvilOne · 2026-02-25T08:34:34Z

src/gardenlinux/features/reproducibility/__main__.py

+        print(result, end="")
+
+    if files != []:
+        exit(64)


What means "magic number" 64? Please use a constant :)

NotTheEvilOne · 2026-02-25T08:35:58Z

src/gardenlinux/features/reproducibility/comparator.py

+
+    _default_whitelist: list[str] = []
+
+    _nightly_whitelist = [


Please move this list to an external JSON file. You may want to use importlib.resources.read_text() :)

NotTheEvilOne · 2026-02-25T08:40:10Z

src/gardenlinux/features/reproducibility/diff_parser.py

+        self._parser = Parser(gardenlinux_root, feature_dir_name, logger)
+        self._feature_dir_name = Path(self._gardenlinux_root).joinpath(feature_dir_name)
+
+        self.all: set[str] = set()


Could you please rename or document these variables? self.all is unfortunately not really self explanatory.

NotTheEvilOne · 2026-02-25T08:45:28Z

src/gardenlinux/features/reproducibility/diff_parser.py

+        self,
+    ) -> Dict[frozenset[str], tuple[frozenset[str], nx.DiGraph]]:
+        """
+        Intersects all features of the affected flavors and removes all features from unaffected flavors to identify features causing the issue


Please ensure you can't use Parser.filter() with additional_filter_func to get the information you want.

NotTheEvilOne · 2026-02-25T08:46:04Z

src/gardenlinux/features/reproducibility/diff_parser.py

+        self.missing_flavors: set[str] = set()
+        self.unexpected_falvors: set[str] = set()
+
+    def read_feature_info(self, feature: str) -> Any:


I think this method is unused. Additionally you can access this information utilizing Parser.graph.

NotTheEvilOne · 2026-02-25T08:47:52Z

src/gardenlinux/features/reproducibility/diff_parser.py

+            for flavor in flavors:
+                # Ignore bare flavors, as they may not be affected due to removing the file and could therefore disrupt the analysis
+                if not flavor.startswith("bare-"):
+                    t = self._parser.filter(self._remove_arch.sub("", flavor))


nit: t means filtered tree?

NotTheEvilOne · 2026-02-25T08:49:38Z

src/gardenlinux/features/reproducibility/markdown_formatter.py

+        :since:  1.0.0
+        """
+
+        info = self._diff_parser.read_feature_info(node)


This function can be replaced by Parser.filter_as_dict(). A prefix of a single number seems odd :(

NotTheEvilOne · 2026-02-25T08:50:27Z

src/gardenlinux/features/reproducibility/markdown_formatter.py

+                Nightly(*n.split(",")) for n in f.read().rstrip().split("\n")
+            )
+            if nightly_a.run_number != "":
+                result += f"\n\nComparison of nightly **[#{nightly_a.run_number}](https://github.com/gardenlinux/gardenlinux/actions/runs/{nightly_a.id})** \


This and the following URLs should be placed in a constant for future change. Maybe you can use a template for the generic structure?

NotTheEvilOne · 2026-02-25T08:53:43Z

src/gardenlinux/features/reproducibility/markdown_formatter.py

+            "✅"
+            if len(self._diff_parser.expected_falvors)
+            == len(self._diff_parser.successful)
+            else ("⚠️" if successrate >= 50.0 else "❌")


The threshold should be a constant :)

Leon-hk added 3 commits January 7, 2026 11:04

feat: Move test formatter from .github/workflows/format_diff.py to py…

96a78f7

…thon-gardenlinux-lib

fix: add ordering constraints for the markdown files

e1d34d8

fix: sorting_function

7130e57

Leon-hk added 4 commits January 13, 2026 08:58

Fix Lint

98c7e24

Fix more linting

6cce0a9

Merge branch 'main' into feat/port-difference-formatter

91b4be0

fix ruff format

9abe888

NotTheEvilOne self-requested a review January 17, 2026 09:48

NotTheEvilOne requested changes Jan 17, 2026

View reviewed changes

Leon-hk mentioned this pull request Jan 21, 2026

Improve reproducibility checker gardenlinux/gardenlinux#4212

Open

6 tasks

Implement feedback, implement difference generator, restructure code

b5e0dc0

Leon-hk added 9 commits January 29, 2026 12:20

fix return value structure

dcda524

fix test and linting

9ed5afb

fix test

43d488e

fix ruff lint

fc73c21

retry lint

47955e2

fix security issue

20f5470

Add tests for .oci files

30623b9

Merge branch 'main' into feat/port-difference-formatter

28c59b2

fix test cases and security warning

752d9f8

Leon-hk requested a review from NotTheEvilOne February 23, 2026 10:44

Leon-hk added 3 commits February 24, 2026 11:32

change exit code to 64

246320d

fix nightly_stats extension

1e952e3

add output option for generator to simplify usage in workflow

a15ba3e

NotTheEvilOne requested changes Feb 25, 2026

View reviewed changes

Comments

Conversation

Leon-hk commented Jan 7, 2026

Uh oh!

codecov bot commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Leon-hk commented Jan 29, 2026

Uh oh!

NotTheEvilOne left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov bot commented Jan 8, 2026 •

edited

Loading