[MNT] Dockerized tests for CI runs using localhost by satvshr · Pull Request #1629 · openml/openml-python

satvshr · 2026-01-29T11:04:10Z

Metadata

Reference Issue: fixes [MNT] Intermediate test plan #1614, stacks on [ENH] Allow using a local test server #1630
New Tests Added: No
Documentation Updated: No

Details

What does this PR implement/fix? Explain your changes.
This PR implements the setting up of the v1 and v2 test servers in CI using docker via localhost.

Locally, MinIO already has more parquet files than on the test server.

Note that the previously strategy didn't work anymore if the server returned a parquet file, which is the case for the new local setup.

This means it is not reliant on the evaluation engine processing the dataset. Interestingly, the database state purposely seems to keep the last task's dataset in preparation explicitly (by having processing marked as done but having to dataset_status entry).

codecov-commenter · 2026-01-29T21:26:14Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 52.82%. Comparing base (7feb2a3) to head (18960d5).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1629   +/-   ##
=======================================
  Coverage   52.82%   52.82%           
=======================================
  Files          37       37           
  Lines        4371     4371           
=======================================
  Hits         2309     2309           
  Misses       2062     2062

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Co-authored-by: Armaghan Shakir <raoarmaghanshakir040@gmail.com>

geetu040 · 2026-02-18T06:53:24Z

.github/workflows/test.yml

+        # sed -i 's|/minio/|/data/|g' config/database/update.sh
+
+        # echo "=== Patched Update Script ==="
+        # cat config/database/update.sh | grep "nginx"


why extra work here? locally just running the services is enough

Kindly ignore these, the pr isnt ready for review yet as tests are still failing and I was trying to debug tests.

geetu040 · 2026-02-18T06:53:31Z

openml/config.py

+if sys.platform.startswith("win"):
+    TEST_SERVER_URL = "http://localhost"
+else:
+    TEST_SERVER_URL = "http://localhost:8000"
+


we should actually use an env variable here, please see https://github.com/openml/openml-python/pull/1629/changes#r2797509441
should be controlled by that env variable, which if not set, should default to use https://test.openml.org/

This is not how I plan to resolve this either, just a temporary fix to the windows issue.

geetu040 · 2026-02-18T08:37:07Z

The tests are taking too long because connection_n_retries is set to 5, you can set it to 1 for this PR, to avoid delays in CI.

satvshr · 2026-02-18T08:41:30Z

The tests are taking too long because connection_n_retries is set to 5, you can set it to 1 for this PR, to avoid delays in CI.

Will do that to prevent hold ups for other CIs in the repo, for my branch it is noticeable if a run is going to fail if it has been stuck on a single test for more than a minute.

geetu040 · 2026-02-18T09:37:16Z

Will do that to prevent hold ups for other CIs in the repo, for my branch it is noticeable if a run is going to fail if it has been stuck on a single test for more than a minute.

yeah but each job in this PR still takes full 150 minutes

geetu040 · 2026-02-23T06:46:10Z

openml/config.py

    "avoid_duplicate_runs": False,
    "retry_policy": "human",
-    "connection_n_retries": 5,
+    "connection_n_retries": 1,


I don't think this would work, since we change this again in conftest.py.
To be completely sure that this works, you can temporarily set n_retries = 1 in _api_calls.py::_send_request

.github/workflows/test.yml

geetu040 · 2026-02-23T09:13:53Z

tests/test_flows/test_flow.py

            f"collected from {__file__.split('/')[-1]}: {flow.flow_id}",
        )

+    @pytest.mark.skip(reason="Pending resolution of #1657")


skip these only if OPENML_USE_LOCAL_SERVICES is set to True

geetu040 · 2026-02-23T09:14:31Z

tests/test_flows/test_flow.py

some tests are skipped though they are not mentioned in #1657, why is that?

I was using the failures from here

geetu040

I don't think any failing test is coming from this PR. It would be better to conditionally skip them and link #1657. If there is new failure message which is not already mentioned there, then please comment down the failure with the failing tests so it could be tracked there. Also if some tests are failing because of pandas, create a separate issue for that, skip and link to these then.

geetu040 · 2026-02-24T07:53:50Z

.github/workflows/test.yml

+        files: coverage.xml
+        token: ${{ secrets.CODECOV_TOKEN }}
+        fail_ci_if_error: true
+        verbose: true


why is this part moved above?

Codecov was giving errors, I do not recall why the errors were occurring.

geetu040 · 2026-02-24T07:53:53Z

tests/conftest.py

+    if os.getenv("OPENML_USE_LOCAL_SERVICES") == "true":
+        openml.config.TEST_SERVER_URL = "http://localhost:8000"


I would ask for you suggestion. Do you think it should be here or rather moved to config.py, since you had it there before this.
I think, having this in config.py would be better, it would also be in favor of global config architecture in #1577 and migration in #1576

First thing that comes to mind is the fact that something pertaining to the testing env should be kept in conftest, though I have no hard opinion on this. If it is more convenient for you I can move it to config too.

Let's keep it here then

geetu040

Looks good, this has not been addressed yet #1629 (comment)

geetu040 · 2026-02-24T18:05:44Z

openml/config.py

    "avoid_duplicate_runs": False,
    "retry_policy": "human",
-    "connection_n_retries": 5,
+    "connection_n_retries": 1,


you can undo this now

Was just waiting to see if tests pass before reverting all changes, apologies!

geetu040 · 2026-02-24T18:05:55Z

openml/_api_calls.py

    md5_checksum: str | None = None,
 ) -> requests.Response:
-    n_retries = max(1, config.connection_n_retries)
+    n_retries = 1


you can undo this now

PGijsbers and others added 11 commits January 20, 2026 12:35

Use the correct path to the cache directory for the task

83e1531

Push configuration of test server URL exclusively to config.py

f90036d

Update the test to use a dataset which does not have a parquet file

3a257ab

Locally, MinIO already has more parquet files than on the test server.

Replace hard-coded cache directory by configured one

3b79017

Update test to use dataset file that is already in cache

f524d75

Note that the previously strategy didn't work anymore if the server returned a parquet file, which is the case for the new local setup.

Windows test

7ef12c2

relax assumptions on local file structure

a5601e3

Do not use static cache directory

d862be2

bug fixing

7c14c68

merge main

78b2038

geetu040 assigned satvshr Jan 29, 2026

satvshr added 4 commits January 30, 2026 02:06

remove db refresh every test

16ceeaa

bug fixing

015acf4

bug fixing

937fc77

bug fixing

30972f8

PGijsbers and others added 9 commits January 30, 2026 10:30

Add symlink to regular test cache directory

775dcf7

Skip test for 1.8 since expected results differ too much

319cb35

Simplify path to static cache directory

a680ebe

Update symbolic link to be relative

b161b3b

Fix typo

0b989d1

trying ot fix multiple threads issue

892ea6c

removed test file

ae3befb

removed unnecessary code (?)

5f396a0

Trigger Build

8a319cd

satvshr marked this pull request as ready for review January 31, 2026 16:13

satvshr marked this pull request as draft January 31, 2026 16:14

satvshr added 2 commits February 1, 2026 17:18

Clean up code

4ba4239

comment fixing

0292404

satvshr and others added 7 commits February 13, 2026 21:07

merge 1630

4c6bd2f

revert test.yml changes

ba0e480

Update openml/config.py

519d5cb

Co-authored-by: Armaghan Shakir <raoarmaghanshakir040@gmail.com>

Merge branch 'main' into update-tests-for-local

30fd44d

Merge branch 'main' into update-tests-for-local

c908993

Keep port as part of cache directory path

06b9741

merge update-tests-for-local

015cca3

geetu040 suggested changes Feb 18, 2026

View reviewed changes

geetu040 mentioned this pull request Feb 18, 2026

[MNT] Failures when running tests against locally replicated test-server #1657

Open

Sandipmandal25 mentioned this pull request Feb 19, 2026

[ENH] Make TEST_SERVER_URL configurable via environment variable #1663

Open

satvshr added 3 commits February 23, 2026 01:25

merge main

8de78af

req changes

4482a2c

changing retries

278b546

geetu040 suggested changes Feb 23, 2026

View reviewed changes

satvshr added 5 commits February 23, 2026 13:04

fixes

cf30367

testing replacement

b41a9b2

bug fix

f737cb1

added skip tests

b40d702

final touches

5f079ba

geetu040 suggested changes Feb 23, 2026

View reviewed changes

geetu040 suggested changes Feb 24, 2026

View reviewed changes

Added skip tests

f75b2de

satvshr requested a review from geetu040 February 24, 2026 15:14

geetu040 suggested changes Feb 24, 2026

View reviewed changes

satvshr added 3 commits February 25, 2026 01:30

final fixes

3648972

new line bug fix

c782fda

os imports

18960d5

		if os.getenv("OPENML_USE_LOCAL_SERVICES") == "true":
		openml.config.TEST_SERVER_URL = "http://localhost:8000"

Uh oh!

Comments

Conversation

satvshr commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Metadata

Details

Uh oh!

codecov-commenter commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

geetu040 commented Feb 18, 2026

Uh oh!

satvshr commented Feb 18, 2026

Uh oh!

geetu040 commented Feb 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

geetu040 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

geetu040 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

satvshr commented Jan 29, 2026 •

edited

Loading

codecov-commenter commented Jan 29, 2026 •

edited

Loading