Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request implements a structured pipeline for performing train-test splits on medical imaging datasets. It provides the necessary logic to filter, stratify, and export data splits, while ensuring these results are tracked and versioned via MLflow. The changes also include the configuration management required to apply these splits to specific data providers and a mechanism to execute these tasks in a containerized environment. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThis PR introduces data split preprocessing capabilities for the PANDA dataset with Radboud provider support. Changes include updates to dataset source configurations, new preprocessing configuration files with stratified train/test split parameters, a data split script implementation with MLflow artifact management, and a remote job submission script. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a data splitting pipeline for the PANDA dataset, including new Hydra configurations, a Python script for stratified train-test splits with MLflow logging, and a Kubernetes job submission script. Key feedback includes replacing the ... placeholder in the submission script to prevent runtime errors, parameterizing hardcoded absolute paths in the dataset configuration for better portability, and aligning the Hydra configuration with the Python logic to make data restrictions properly optional.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (2)
preprocessing/data_split.py (2)
20-23: OmegaConfNonecheck may not behave as expected.With OmegaConf, checking
config.restriction is not Nonemay not work as intended since therestrictionkey is always defined in the config template (with required???fields). If you want to support optional restrictions in the future, consider checking for a sentinel value or adjusting the config structure.For the current implementation where
restrictionis always provided by experiment configs, this works correctly—just noting for future flexibility.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@preprocessing/data_split.py` around lines 20 - 23, Replace the direct None check on config.restriction with an OmegaConf-safe check: test that config.restriction is neither missing nor explicitly set to null using OmegaConf.is_missing and OmegaConf.is_none before reading provider_column/provider_value and filtering slides (i.e., change the condition around config.restriction, which guards access to provider_column/provider_value and the slides = slides[...] line, to use not (OmegaConf.is_missing(config.restriction) or OmegaConf.is_none(config.restriction))).
31-32: Consider using.copy()to avoidSettingWithCopyWarning.
train_test_splitmay return views of the original DataFrame. Assigning to thesetcolumn on these views can trigger pandasSettingWithCopyWarningand may not modify the data as intended in some pandas versions.Proposed fix
train, test = train_test_split( slides, test_size=config.test_size, random_state=42, stratify=slides[config.stratify_column], ) - train["set"] = "train" - test["set"] = "test" + train = train.copy() + test = test.copy() + train["set"] = "train" + test["set"] = "test" split = pd.concat([train, test], ignore_index=True)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@preprocessing/data_split.py` around lines 31 - 32, The assignment train["set"] = "train" and test["set"] = "test" can operate on views returned by train_test_split and trigger pandas SettingWithCopyWarning; to fix, ensure you work on explicit copies by calling train = train.copy() and test = test.copy() (or otherwise use .loc to assign into an ensured copy) before setting the "set" column so the modifications to the DataFrame variables train and test are applied reliably.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@preprocessing/data_split.py`:
- Around line 20-23: Replace the direct None check on config.restriction with an
OmegaConf-safe check: test that config.restriction is neither missing nor
explicitly set to null using OmegaConf.is_missing and OmegaConf.is_none before
reading provider_column/provider_value and filtering slides (i.e., change the
condition around config.restriction, which guards access to
provider_column/provider_value and the slides = slides[...] line, to use not
(OmegaConf.is_missing(config.restriction) or
OmegaConf.is_none(config.restriction))).
- Around line 31-32: The assignment train["set"] = "train" and test["set"] =
"test" can operate on views returned by train_test_split and trigger pandas
SettingWithCopyWarning; to fix, ensure you work on explicit copies by calling
train = train.copy() and test = test.copy() (or otherwise use .loc to assign
into an ensured copy) before setting the "set" column so the modifications to
the DataFrame variables train and test are applied reliably.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 562c68da-a276-4d98-99ad-e66ddef87711
📒 Files selected for processing (7)
configs/base.yamlconfigs/data/sources/panda.yamlconfigs/data/sources/prostate_cancer.yamlconfigs/experiment/preprocessing/data_split/radboud.yamlconfigs/preprocessing/data_split.yamlpreprocessing/data_split.pyscripts/preprocessing/run_data_split.py
3 participants
Depends on #8
Summary by CodeRabbit
New Features
Chores