feat(panda): standardize nuclei seg data#9
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
✅ Files skipped from review due to trivial changes (1)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughWalkthroughIntroduces a nuclei standardization processing pipeline for the PANDA dataset with configuration, a Ray-backed preprocessing script that standardizes nuclei segmentation data into polygons and centroids, and a Kubernetes job submission script. Updates dataset configuration paths and MLflow artifact reference. Changes
Sequence DiagramsequenceDiagram
participant Config as Hydra Config
participant Main as Main Process
participant MLflow as MLflow
participant Ray as Ray Cluster
participant Disk as Disk Storage
participant Worker as Ray Worker
Main->>Config: Load nuclei_standardization config
Main->>MLflow: Download metadata from config.metadata_uri
MLflow-->>Main: Metadata with slide_id, segmentation_id
Main->>Main: Filter rows where has_segmentation=true
Main->>Ray: Dispatch work via process_items(max_concurrent=10)
Note over Ray: Concurrent processing of slide/segmentation pairs
Ray->>Worker: standardize_nuclei(item, output_dir, nuclei_dir)
Worker->>Disk: Read nuclei.parquet (points, radial_distances)
Worker->>Worker: Convert radial distances to polygon vertices<br/>(polar coordinate sampling)
Worker->>Worker: Compute centroid from polygon vertices
Worker->>Worker: Generate deterministic ID (SHA-256 hash)
Worker->>Disk: Write standardized nuclei.parquet<br/>(id, polygon, centroid)
Ray-->>Main: All items processed
Main->>MLflow: Log run metrics
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request focuses on enhancing data management and preprocessing capabilities, particularly for nuclei segmentation data. It introduces a standardized approach for handling the PANDA dataset by defining its specific data configurations and implementing a robust script to standardize nuclei segmentation files. This standardization involves converting radial distance representations into Cartesian polygons and assigning globally unique identifiers to each nucleus, which is crucial for consistent downstream analysis. Furthermore, the PR refactors the base data path configuration to improve flexibility and maintainability across different datasets. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new feature to standardize nuclei segmentation data, specifically for the PANDA dataset. It includes updates to base and dataset-specific configurations, along with a new Python script for processing nuclei data and a job submission script. The changes correctly integrate new data paths and configurations within the existing Hydra and MLflow setup. The core logic for converting radial distances to Cartesian polygons and generating unique nucleus IDs is well-implemented. The overall structure and approach align well with the repository's research focus and existing conventions.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (4)
configs/data/sources/panda.yaml (2)
11-11: Consider using path interpolation for consistency.This path is hardcoded while others use
${project_path}or${data_path}interpolations. If this intentionally references a separate project location, consider documenting it with a comment.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@configs/data/sources/panda.yaml` at line 11, The slides_properties entry uses a hardcoded absolute path instead of the project's interpolation variables; update slides_properties to use the same interpolation pattern (e.g., ${project_path} or ${data_path}) as other entries so it becomes consistent with the rest of the config, or if it intentionally points to a different project, add a brief inline comment next to slides_properties explaining why the absolute path is required; locate the slides_properties key in the YAML to make the change.
14-14: Consider using path interpolation for consistency.Similar to
slides_properties, this hardcoded path differs from the pattern used elsewhere. If this is the expected source location from another project, a brief comment would clarify the intent.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@configs/data/sources/panda.yaml` at line 14, The entry nuclei_seg_radial uses a hardcoded absolute path; change it to use the same path interpolation pattern as slides_properties (e.g., reference project/dataset variables or an interpolated base path) so it remains consistent with other sources, or if this exact external path is intentional add a one-line comment explaining it's an expected external location; update the nuclei_seg_radial value accordingly and ensure it follows the project's interpolation tokens/variable names used by slides_properties.scripts/preprocessing/run_nuclei_standardization.py (1)
11-16: Consider pinning to a specific branch or commit.The script clones the default branch without specifying a version. For reproducibility, consider adding a branch/tag/commit reference, especially since this PR targets
masterbut originates frompanda/data-standardization.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@scripts/preprocessing/run_nuclei_standardization.py` around lines 11 - 16, The git clone in the script array currently clones the repository without a pinned ref, which harms reproducibility; update the "git clone ..." command in scripts/preprocessing/run_nuclei_standardization.py (the script variable/array) to include a specific branch, tag, or commit (e.g., append --branch <branch> or clone then checkout a commit) so the pipeline always uses the intended revision from the panda/data-standardization work (ensure the chosen ref corresponds to the PR's source branch or a stable tag).preprocessing/nuclei_standardization.py (1)
22-23: Consider increasing memory allocation for larger partitions.1 GiB memory per worker may be tight if partitions contain many nuclei with large radial distance arrays. Monitor for OOM issues during execution.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@preprocessing/nuclei_standardization.py` around lines 22 - 23, The Ray task decorator on standardize_nuclei currently sets memory=(1 * 1024**3) which may be insufficient for large partitions; update the `@ray.remote` annotation for standardize_nuclei to allocate more memory (e.g., 2-4 GiB) or make the memory value configurable via a constant/env var so it can be tuned at runtime, and ensure any tests/launch scripts that spawn standardize_nuclei workers are updated to use the new configurable value; reference the `@ray.remote` decorator and the standardize_nuclei function when making the change.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@preprocessing/nuclei_standardization.py`:
- Around line 64-72: The slide directory is being constructed from the wrong
field (using row.id) causing a mismatch with the recorded slide_id; update the
slide_dir construction so it uses row.slide_id instead of row.id (the block that
sets slide_dir = nuclei_dir / f"slide_id={...}" and then globs partition_files)
so the directory naming matches the slide_id stored when appending to
items_to_process (keys: "nuclei_partition", "slide_id") and ensure no other
references to row.id remain in this loop.
---
Nitpick comments:
In `@configs/data/sources/panda.yaml`:
- Line 11: The slides_properties entry uses a hardcoded absolute path instead of
the project's interpolation variables; update slides_properties to use the same
interpolation pattern (e.g., ${project_path} or ${data_path}) as other entries
so it becomes consistent with the rest of the config, or if it intentionally
points to a different project, add a brief inline comment next to
slides_properties explaining why the absolute path is required; locate the
slides_properties key in the YAML to make the change.
- Line 14: The entry nuclei_seg_radial uses a hardcoded absolute path; change it
to use the same path interpolation pattern as slides_properties (e.g., reference
project/dataset variables or an interpolated base path) so it remains consistent
with other sources, or if this exact external path is intentional add a one-line
comment explaining it's an expected external location; update the
nuclei_seg_radial value accordingly and ensure it follows the project's
interpolation tokens/variable names used by slides_properties.
In `@preprocessing/nuclei_standardization.py`:
- Around line 22-23: The Ray task decorator on standardize_nuclei currently sets
memory=(1 * 1024**3) which may be insufficient for large partitions; update the
`@ray.remote` annotation for standardize_nuclei to allocate more memory (e.g., 2-4
GiB) or make the memory value configurable via a constant/env var so it can be
tuned at runtime, and ensure any tests/launch scripts that spawn
standardize_nuclei workers are updated to use the new configurable value;
reference the `@ray.remote` decorator and the standardize_nuclei function when
making the change.
In `@scripts/preprocessing/run_nuclei_standardization.py`:
- Around line 11-16: The git clone in the script array currently clones the
repository without a pinned ref, which harms reproducibility; update the "git
clone ..." command in scripts/preprocessing/run_nuclei_standardization.py (the
script variable/array) to include a specific branch, tag, or commit (e.g.,
append --branch <branch> or clone then checkout a commit) so the pipeline always
uses the intended revision from the panda/data-standardization work (ensure the
chosen ref corresponds to the PR's source branch or a stable tag).
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: adcb34b0-e9e2-4aad-8cc5-3b3de10ee337
📒 Files selected for processing (6)
configs/base.yamlconfigs/data/sources/panda.yamlconfigs/data/sources/prostate_cancer.yamlconfigs/preprocessing/nuclei_standardization.yamlpreprocessing/nuclei_standardization.pyscripts/preprocessing/run_nuclei_standardization.py
3 participants
Depends on PR #8
Summary by CodeRabbit
New Features
Chores