Skip to content

🆕 Simplified SAM Segmentation in TIAViz#968

Merged
shaneahmed merged 120 commits intodevelopfrom
sam-viz-fix
Mar 10, 2026
Merged

🆕 Simplified SAM Segmentation in TIAViz#968
shaneahmed merged 120 commits intodevelopfrom
sam-viz-fix

Conversation

@measty
Copy link
Copy Markdown
Collaborator

@measty measty commented Nov 7, 2025 •
edited
Loading

This PR takes the sam-viz PR and fixes and substantially simplifies it.

I think the original PR was going for a more general prompt segmentor engine that would in theory cope with prompts dispersed over a whole WSI, but I think that approach has issues in practice (and it would also need some significant work to fix from its current state, I think). This PR focusses on segmenting prompts made within a specific view window, which I think is more practical and allows for a much simplified approach.

To use, draw a few points (and/or a box) on some object(s) in the field of view in TIAViz using this widget:
image
Then select SAM in the model select box and click run. The segmetations should be displayed after a short time.

Examples:
image
image
image

mbasheer04 and others added 30 commits January 17, 2025 14:18
@mbasheer04
Merge pull request #1 from TissueImageAnalytics/develop
ca37400 
Merge develop
@mbasheer04
Merge branch 'TissueImageAnalytics:develop' into develop
dab2693
@mbasheer04
Squashed commit of the following:
bc6afda 
commit d315a5f415542890f351ddb07c771d62c278c2f2
Merge: dab2693 1a27178
Author: Musraf Basheer <Musraf.Basheer@warwick.ac.uk>
Date:   Fri Jan 24 15:07:28 2025 +0000

    Merge branch 'sam-architecture' into develop

commit dab2693
Merge: ca37400 8ff4f5e
Author: mbasheer04 <78800844+mbasheer04@users.noreply.github.com>
Date:   Fri Jan 24 15:04:28 2025 +0000

    Merge branch 'TissueImageAnalytics:develop' into develop

commit 1a27178
Merge: 3b51acb 8ff4f5e
Author: mbasheer04 <78800844+mbasheer04@users.noreply.github.com>
Date:   Fri Jan 24 14:57:22 2025 +0000

    Merge branch 'TissueImageAnalytics:develop' into sam-architecture

commit 3b51acb
Author: Musraf Basheer <Musraf.Basheer@warwick.ac.uk>
Date:   Fri Jan 24 14:53:06 2025 +0000

    Improved Engine

commit 8ff4f5e
Author: Mark Eastwood <20169086+measty@users.noreply.github.com>
Date:   Thu Jan 23 10:03:43 2025 +0000

    📌 Pin `zarr<3.0.0` (#905)

    - `zarr>3.0.0` is not compatible yet

commit 46d30f2
Merge: 1e136c5 ca37400
Author: Musraf Basheer <Musraf.Basheer@warwick.ac.uk>
Date:   Fri Jan 17 14:22:18 2025 +0000

    Merge branch 'develop' into sam-architecture

commit ca37400
Merge: 32cae0b f385ced
Author: mbasheer04 <78800844+mbasheer04@users.noreply.github.com>
Date:   Fri Jan 17 14:18:47 2025 +0000

    Merge pull request #1 from TissueImageAnalytics/develop

    Merge develop

commit 1e136c5
Merge: 47a2190 bb9399d
Author: Musraf Basheer <Musraf.Basheer@warwick.ac.uk>
Date:   Fri Jan 17 14:12:52 2025 +0000

    Merge branch 'sam-architecture' of https://github.com/mbasheer04/tiatoolbox into sam-architecture

commit 47a2190
Author: Musraf Basheer <Musraf.Basheer@warwick.ac.uk>
Date:   Fri Jan 17 14:08:11 2025 +0000

    Improved engine & added unit tests

commit f385ced
Author: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Date:   Fri Jan 10 12:04:52 2025 +0000

    :technologist: pre-commit autoupdate (#902)

    * 🧑‍💻 pre-commit autoupdate

    updates:
    - [github.com/executablebooks/mdformat: 0.7.19 → 0.7.21](hukkin/mdformat@0.7.19...0.7.21)
    - [github.com/astral-sh/ruff-pre-commit: v0.8.2 → v0.8.6](astral-sh/ruff-pre-commit@v0.8.2...v0.8.6)

    * [pre-commit.ci] auto fixes from pre-commit.com hooks

    for more information, see https://pre-commit.ci

    ---------

    Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

commit 0ddd2b1
Author: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>
Date:   Thu Dec 12 14:45:39 2024 +0000

    🔖 Release 1.6.0 (#898)

    ## TIAToolbox v1.6.0 (2024-12-12)

    ### Major Updates and Feature Improvements

    - **Foundation Models Support via `timm` API** (#856, contributed by @GeorgeBatch)
      - Introduced `TimmBackbone` for running additional PyTorch Image Models.
      - Tested models include `UNI`, `Prov-GigaPath`, and `H-optimus-0`.
      - Added an example notebook demonstrating feature extraction with foundation models.
      - `timm` added as a dependency.
    - **Performance Enhancements with `torch.compile`** (#716)
      - Improved performance on newer GPUs using `torch.compile`.
    - **Multichannel Input Support in `WSIReader`** (#742)
    - **AnnotationStore Filtering for Patch Extraction** (#822)
    - **Python 3.12 Support**
    - **Deprecation of Python 3.8 Support**
    - **CLI Response Time Improvements** (#795)

    ### API Changes

    - **Device Specification Update** (#882)
      - Replaced `has_gpu` with `device` for specifying GPU or CPU usage, aligning with PyTorch's `Model.to()` functionality.
    - **Windows Compatibility Enhancement** (#769)
      - Replaced `POWER` with explicit multiplication.

    ### Bug Fixes and Other Changes

    - **TIFFWSIReader Bound Reading Adjustment** (#777)
      - Fixed `read_bound` to use adjusted bounds.
      - Reduced code complexity in `WSIReader` (#814).
    - **Annotation Rendering Fixes** (#813)
      - Corrected rendering of annotations with holes.
    - **Non-Tiled TIFF Support in `WSIReader`** (#807, contributed by @GeorgeBatch)
    - **HoVer-Net Documentation Update** (#751)
      - Corrected class output information.
    - **Citation File Fix for `cffconvert`** (#869, contributed by @Alon-Alexander)
    - **Bokeh Compatibility Updates**
      - Updated `bokeh_app` for compatibility with `bokeh>=3.5.0`.
      - Switched from `size` to `radius` for `bokeh>3.4.0` compatibility (#796).
    - **JSON Extraction Fixes** (#772)
      - Restructured SQL expression construction for JSON properties with dots in keys.
    - **VahadaneExtractor Warning** (#871)
      - Added warning due to changes in `scikit-learn>0.23.0` dictionary learning (#382).
    - **PatchExtractor Error Message Refinement** (#883)
    - **Immutable Output Fix in `WSIReader`** (#850)

    ### Development-Related Changes

    - **Mypy Checks Added**
      - Applied to `utils`, `tools`, `data`, `annotation`, and `cli/common`.
    - **ReadTheDocs PDF Build Deprecation**
    - **Formatter Update**
      - Replaced `black` with `ruff-format`.
    - **Dependency Removal**
      - Removed `jinja2`.
    - **Test Environment Update**
      - Updated to `Ubuntu 24.04`.
    - **Conda Environment Workflow Update**
      - Implemented `micromamba` setup.
    - **Codecov Reporting Fix** (#811)
      **Full Changelog:** v1.5.1...v1.6.0

commit a38d95f
Author: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>
Date:   Thu Dec 5 17:14:20 2024 +0000

    [skip ci] 🐛 Fix `benchmarks/annotation_store.ipynb` (#894)

    - Fix `benchmarks/annotation_store.ipynb` errors

commit 6b214fe
Author: adamshephard <39619155+adamshephard@users.noreply.github.com>
Date:   Tue Dec 3 14:09:33 2024 +0000

    📝 Add Example Notebook for Foundation Models (#887)

    - Add Example Notebook to explain how to use Foundation Models from `timm` module in TIAToolbox.

    ---------

    Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
    Co-authored-by: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>

commit 442bd3f
Author: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Date:   Tue Dec 3 09:40:40 2024 +0000

    :technologist: pre-commit autoupdate (#891)

    <!--pre-commit.ci start-->
    updates:
    - [github.com/executablebooks/mdformat: 0.7.18 → 0.7.19](hukkin/mdformat@0.7.18...0.7.19)

    ---------

    Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
    Co-authored-by: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>

commit 741463c
Author: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>
Date:   Mon Dec 2 22:45:05 2024 +0000

    :pushpin: Update `ruff` Version to `v0.8.1` (#890)

    - Update `ruff` Version to `v0.8.1`

    ---------

    Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

commit 9a62c10
Author: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>
Date:   Mon Dec 2 17:11:39 2024 +0000

    [skip ci] :memo: Update Jupyter Notebooks for Release `v1.6.0` (#885)

    - Update Jupyter Notebooks for the New Release
    - Fix issues with API changes e.g., device instead of ON_GPU flag.

commit 4a1940d
Author: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>
Date:   Fri Nov 29 16:13:37 2024 +0000

    :technologist: `torch.compile` is not compatible with Windows. (#888)

    - `torch.compile` is not currently compatible with Windows. See pytorch/pytorch#122094

commit 5f1cecb
Author: Jiaqi-Lv <60471431+Jiaqi-Lv@users.noreply.github.com>
Date:   Fri Nov 29 14:37:39 2024 +0000

    🧑‍💻 Refine `PatchExtractor` Error Message (#883)

    - Fix Misleading error message #881

commit 5beb119
Author: Musraf Basheer <Musraf.Basheer@warwick.ac.uk>
Date:   Tue Nov 26 18:47:14 2024 +0000

    Initialised SAM Engine

commit bb9399d
Author: mbasheer04 <78800844+mbasheer04@users.noreply.github.com>
Date:   Fri Nov 22 19:26:45 2024 +0000

    Created using Colab

commit ca13e7f
Author: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>
Date:   Thu Nov 21 22:28:44 2024 +0000

    ♻️ Update Changes from New Engine Design (#882)

    - Add changes from New engine design #578. This will not only simplify the PR but also keep the main repo up to date.
    - Refactor `model_to` to `model_abc`
    - Instead of `on_gpu` use `device` as an input in line with `PyTorch`.
    - `infer_batch` uses `device` as an input instead of `on_gpu`

commit 91ca877
Merge: 1e27436 32cae0b
Author: mbasheer04 <bmusraf1@gmail.com>
Date:   Thu Nov 21 21:03:41 2024 +0000

    Merge branch 'develop' of https://github.com/TissueImageAnalytics/tiatoolbox into sam-architecture

commit 1e27436
Author: mbasheer04 <bmusraf1@gmail.com>
Date:   Thu Nov 21 21:00:34 2024 +0000

    Implementing architecture for full-image segmentation

commit 32cae0b
Author: Abdol <a@fkrtech.com>
Date:   Fri Nov 15 20:29:34 2024 +0000

    ⚡️Add `torch.compile` Functionality (#716)

    - Integrates PyTorch 2.0's [torch.compile](https://pytorch.org/docs/stable/generated/torch.compile.html) functionality to demonstrate performance improvements in torch code. This PR focuses on adding `torch.compile` to `PatchPredictor`.

    **Notes:**
    - According to the [documentation](https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html), noticeable performance can be achieved when using modern NVIDIA GPUs (H100, A100, or V100)

    **TODO:**
    - [x] Resolve compilation errors related to using `torch.compile` in running models
    - [x] Initial config
    - [x] Add to patch predictor
    - [x] Add to registration
    - [x] Add to segmentation
    - [x] Test on custom models
    - [x] Test on `torch.compile` compatible GPUs

    ---------

    Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
    Co-authored-by: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>
    Co-authored-by: Jiaqi-Lv <60471431+Jiaqi-Lv@users.noreply.github.com>

commit e1a57c6
Author: mbasheer04 <bmusraf1@gmail.com>
Date:   Fri Nov 15 17:29:44 2024 +0000

    Created Jupyter Notebook for SAM architecture

commit 9113996
Author: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Date:   Fri Nov 15 17:18:16 2024 +0000

    :technologist: pre-commit autoupdate (#880)

    * 🧑‍💻 pre-commit autoupdate

    updates:
    - [github.com/astral-sh/ruff-pre-commit: v0.7.0 → v0.7.2](astral-sh/ruff-pre-commit@v0.7.0...v0.7.2)

    * 📌 Update `ruff` version.

    ---------

    Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
    Co-authored-by: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>

commit c980eec
Author: George Batchkala <46561186+GeorgeBatch@users.noreply.github.com>
Date:   Fri Nov 15 16:31:42 2024 +0000

    🆕 Integrate Foundation Models Available VIA `timm`: `UNI`, `Prov-GigaPath`, `H-optimus-0` (#856)

    - Integrates pre-trained foundation models from other labs into tiatoolbox.models.architecture.vanilla.py.

    Currently, the `_get_architecture()` function allows the use of models from `torchvision.models`.

    A new function `_get_timm_architecture()` has been incorporated to support foundation models which are available from `timm` with weights on HuggingFace Hub.

    - All the models from `timm` that used require users to sign the licence agreement with the authors.
    - The users can add new models by modifying `_get_timm_architecture()`

    ---------

    Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
    Co-authored-by: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>

commit 7fb9926
Author: Mostafa Jahanifar <74412979+mostafajahanifar@users.noreply.github.com>
Date:   Fri Nov 8 17:25:02 2024 +0000

    🐛 Add Warning for `VahadaneExtractor` Algorithm Instability (#871)

    - Adds a warning to the `VahadaneExtractor` to inform users about the algorithm's instability due to changes in the dictionary learning algorithm in `scikit-learn versions > 0.23.0 (see issue #382)`.
    - The docstrings are updated accordingly to reflect this warning.
    - No other functionality is altered.

    ---------

    Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

commit f4f2fa6
Author: mbasheer04 <bmusraf1@gmail.com>
Date:   Sun Nov 3 23:41:50 2024 +0000

    Initial setup of SAM architecture

commit 12d435e
Author: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>
Date:   Thu Oct 24 16:40:56 2024 +0100

    :pushpin: Pin `numpy<2.0.0` (#876)

    - `numpy>=2.0` breaks the tests.

commit 76f02f8
Author: Mark Eastwood <20169086+measty@users.noreply.github.com>
Date:   Fri Oct 18 14:53:49 2024 +0100

    🆕 Use Annotations as a Filter for Patch Extraction (#822)

    - Adds the ability to get patches filtered by class or label using AnnotationStore.
    - Allows the user to provide a path to an annotation store as the mask.
      - By default the mask will then be built out of all the annotations, but the user can also provide any filter that works in the annotation store, and have only those annotations make the mask.

    For example, if the user have some pathologist annotations with some regions, and they would only want to extract patches which have some overlap with regions labelled 'tumor' or any other class, they would just need to provide the appropriate string ("props['label'] == 'tumor'" for example).

    ---------

    Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

commit 0b857c7
Author: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Date:   Fri Oct 18 12:00:04 2024 +0100

    :technologist: `pre-commit` autoupdate (#868)

    * [pre-commit.ci] pre-commit autoupdate

    updates:
    - [github.com/pre-commit/pre-commit-hooks: v4.6.0 → v5.0.0](pre-commit/pre-commit-hooks@v4.6.0...v5.0.0)
    - [github.com/astral-sh/ruff-pre-commit: v0.6.5 → v0.6.9](astral-sh/ruff-pre-commit@v0.6.5...v0.6.9)

    * 🐛 Fix pre-commit yaml

    * 🧑‍💻 Change to monthly schedule.

    * [pre-commit.ci] pre-commit autoupdate

    updates:
    - [github.com/executablebooks/mdformat: 0.7.17 → 0.7.18](hukkin/mdformat@0.7.17...0.7.18)
    - [github.com/pre-commit/pre-commit-hooks: v4.6.0 → v5.0.0](pre-commit/pre-commit-hooks@v4.6.0...v5.0.0)
    - [github.com/astral-sh/ruff-pre-commit: v0.6.5 → v0.6.9](astral-sh/ruff-pre-commit@v0.6.5...v0.6.9)

    * 🔀 Merge online and local copies

    * 📌 Pin dependencies

    * 📌 Pin `ruff` version

    ---------

    Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
    Co-authored-by: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>

commit 52792bc
Author: Alon Alexander <alon008@gmail.com>
Date:   Wed Oct 2 12:51:52 2024 +0300

    🐛 Fix `CITATION.cff` to Make it Compatible with `cffconvert`

    - To cite this repository using `cffconvert`, the conversion fails as the citation file was invalid.
    - This PR fixes the citation file to a valid ORCID.

commit 3714ef7
Author: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>
Date:   Wed Oct 2 10:22:26 2024 +0100

    :pushpin: Pin `bokeh` version to less than 3.6.0 (#870)

    - Pin `bokeh` version to less than `3.6.0`
    - `bokeh` 3.6.0 has breaking changes
@mbasheer04
Merge branch 'develop' of https://github.com/mbasheer04/tiatoolbox in…
5f8032d 
…to develop
@mbasheer04
Integrating SAM into bokeh
a543460
@mbasheer04
Added save file for GeneralSegmentor output
c620ca4
@mbasheer04
Added on-click prompt segementation to TIAViz
9217b69
@mbasheer04
Added multi-prompt segmentation & bounding-box
60f4d5f
@mbasheer04
Added scores to masks
09a79fd
@mbasheer04
Attempting to add resolution/window-based segmentation
1a4a76c
@mbasheer04
Successfully implemented window-based segmentation
b322539
@mbasheer04
Fixing issues
35237c4
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
6ad56fe 
for more information, see https://pre-commit.ci
@shaneahmed
Merge branch 'develop' into sam-viz
2cb7d9c
@mbasheer04
Preparing for PR
7b403c9 
Removing large files

Removing unneccessary files

Fixing pre-commit

[pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Fixing PR issues

:technologist: pre-commit autoupdate (#910)

* 🧑‍💻 pre-commit autoupdate

updates:
- [github.com/executablebooks/mdformat: 0.7.21 → 0.7.22](hukkin/mdformat@0.7.21...0.7.22)
- [github.com/astral-sh/ruff-pre-commit: v0.8.6 → v0.9.4](astral-sh/ruff-pre-commit@v0.8.6...v0.9.4)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 📌 Update `ruff` dependency

* 🔥 TIAToolbox does not support Python > 3.12 yet
- There is no need for this check as this will be tested while upgrading to Python 3.13

* ♻️ Refactor `typing` to `type_hints`.

* 🐛 Fix `mypy` workflow

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>

📝 Update Documentation Structure (#909)

- Use `Python 3.12` for docs build
- Update `copyright` year to `2025`
- Landing page now shows text from README
- Update documentation structure
- Update `readthedocs` Build
- Remove `usage.rst`
- Rename Jupyter Notebooks to Usage Examples
- Show README for Usage Examples instead of TOC
- Reduce TOC depth for basic functionalities and pipelines
- Improve `README` quality.

🐛 Fix in `test_arch_mapde` and `test_arch_sccnn` (#911)

- If cuda is available model should be moved to cuda otherwise tests will fail as test data is moved to cuda.

[skip ci] 📝 Improve Documentation (#913)

- Update CONTRIBUTING.rst
- Bug fix in conf.py to fix notebook links
- Update `examples/README.md`
- Update `docs/installation.rst`
- Update `docs/visualization.rst`

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: adamshephard <39619155+adamshephard@users.noreply.github.com>

:bug: Fix `MapDe` `dist_filter` Shape  (#914)

- Fix `dist_filter` in `MapDe` model for multi-class output.

Explanation:
Previously, if we set `num_class` to more than 1, the model would still output 1 channel. This was because the `dist_filter` always had size of 1 in its first dimension, however the first dimension determines the number of output channels in the tensor produced by `torch.functional.F.conv2d`.
This PR changes this by repeating the filters the match the number of output classes.

:technologist: pre-commit autoupdate (#916)

* 🧑‍💻 pre-commit autoupdate

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.9.4 → v0.9.9](astral-sh/ruff-pre-commit@v0.9.4...v0.9.9)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 🔨 Update `ruff` version

* 🔨 Update noqa for Unused static method argument

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>

Add FsspecJsonWSIReader class. (#897)

The `FsspecJsonWSIReader` reads fsspec json file which represents SVS or TIFF whole slide image. The images are accessible by HTTP range requests, eg:

`https://api.gdc.cancer.gov/data/73c69d24-6f9e-44e2-bfe5-a608d4cf5c27`

The whole image can be downloaded like:

`curl -C - -o TCGA-22-1017-01Z-00-DX1.9562FE79-A261-42D3-B394-F3E0E2FF7DDA.svs   https://api.gdc.cancer.gov/data/73c69d24-6f9e-44e2-bfe5-a608d4cf5c27`

The `FsspecJsonWSIReader` class has a `_zarr_store` field which is created by reading json file using `fsspec`:

```
mapper = fsspec.get_mapper(
            "reference://", fo=str(input_img), target_protocol="file"
        )
self._zarr_array = zarr.open(mapper, mode="r")
self._zarr_store = self._zarr_array.store

self._zarr_lru_cache = zarr.LRUStoreCache(self._zarr_store, max_size=cache_size)
self._zarr_group = zarr.open(self._zarr_lru_cache)
```

This is equivalent to `TIFFWSIReader` code:

```
self._zarr_store = tifffile.imread(
            self.input_path,
            series=self.series_n,
            aszarr=True,
        )
        self._zarr_lru_cache = zarr.LRUStoreCache(self._zarr_store, max_size=cache_size)
        self._zarr_group = zarr.open(self._zarr_lru_cache)
```

Both FsspecJsonWSIReader and TIFFWSIReader forward calls to `read_bounds` and  `read_rect` methods of the`TIFFWSIReaderDelegate` delegate instance.

The method `_info` of the`TIFFWSIReaderDelegate` reads SVS metadata which is stored in the root group metadata like:
```
{
  ".zattrs": {
    "multiscales": [
      {
        "metadata": {
          "objective_power": 40,
          "vendor": "Aperio",
          "mpp": [0.2525, 0.2525]
        }
      }
    ]
  }
}
```

To test, execute from the root dir:
```
pip install -r requirements/requirements_dev.txt
mkdir -p samples/slides
mkdir -p samples/fsspec
cd samples/slides
curl -C - -o TCGA-22-1017-01Z-00-DX1.9562FE79-A261-42D3-B394-F3E0E2FF7DDA.svs   https://api.gdc.cancer.gov/data/73c69d24-6f9e-44e2-bfe5-a608d4cf5c27
cd ../../
cp tiatoolbox/utils/tiff_to_fsspec.py .
python tiff_to_fsspec.py "samples/slides/TCGA-22-1017-01Z-00-DX1.9562FE79-A261-42D3-B394-F3E0E2FF7DDA.svs"  "samples/fsspec/73c69d24-6f9e-44e2-bfe5-a608d4cf5c27_fsspec.json" "https://api.gdc.cancer.gov/data/73c69d24-6f9e-44e2-bfe5-a608d4cf5c27"
```

Create `tileserver.py` inside of the project root:

```
from flask_cors import CORS

from tiatoolbox.visualization import TileServer
from tiatoolbox.wsicore.wsireader import FsspecJsonWSIReader

wsi = FsspecJsonWSIReader.open(
    "./samples/fsspec/73c69d24-6f9e-44e2-bfe5-a608d4cf5c27_fsspec.json"
)

tile_server = TileServer(
    title="Tiatoolbox TileServer",
    layers={"layer": wsi},
)
CORS(tile_server, send_wildcard=True)

tile_server.run(host="127.0.0.1", port=5000)
```

Open `http://127.0.0.1:5000/` and verify that it works.
---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>

✨ Support for Additional Foundation Models (#906)

- Add support for additional foundation models as feature extractors using the TimmBackbone.
- Added models include: UNI2, Virchow, Virchow2, kaiko and H-optimus-1.
- Add more information to docstrings.
- Allow foundation models with additional parameters.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>

Fixing PR issues
@mbasheer04
Merge branch 'sam-viz' of https://github.com/mbasheer04/tiatoolbox in…
a8f2938 
…to sam-viz
@mbasheer04
Restoring notebook changes
6e8b859
@mbasheer04
Pre-commit fixes
51ecbe9
@mbasheer04
Added centroid extraction
fd3a2c2
@shaneahmed
Merge branch 'develop' into sam-viz
315d2c8
@adamshephard
Merge branch 'develop' into sam-viz
a5d5971
@mbasheer04
Improving Engine
ec9b76c
@mbasheer04
Removing eval files
5db9657
@mbasheer04
Merge branch 'sam-viz' of https://github.com/mbasheer04/tiatoolbox in…
e193646 
…to sam-viz
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
effdadc 
for more information, see https://pre-commit.ci
@mbasheer04
Cleaning up code
f69dfb7
@mbasheer04
Merge branch 'sam-viz' of https://github.com/mbasheer04/tiatoolbox in…
c02f153 
…to sam-viz
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
c1cb059 
for more information, see https://pre-commit.ci
@mbasheer04
Added docstrings
71eedb2
@mbasheer04
Added SAM to requirements.txt
0878a66
github-code-quality[bot]
github-code-quality Bot found potential problems Mar 9, 2026
View reviewed changes
Comment thread tests/test_app_bokeh.py Fixed
@shaneahmed
Copy link
Copy Markdown
Member

shaneahmed commented Mar 9, 2026 •
edited
Loading

Clear Overlays works fine now but the reset button does not clear the annotations. Is this intended design?

@measty
Copy link
Copy Markdown
Collaborator Author

measty commented Mar 9, 2026

Clear Overlays works fine now but the reset button does not clear the annotations. Is this intended design?

The reset button isn't intended for anything, it's just there by default in bokeh. We can remove it as it doesn't really have much use in TIAViz. The buttons I added in this PR are the 'Clear overlays' button that you requested to remove existing annotations, and the 'clear' button that was added before that clears only manual annotations (the points/boxes that you can draw).

shaneahmed
shaneahmed approved these changes Mar 10, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@shaneahmed shaneahmed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @measty

@shaneahmed shaneahmed changed the title 🆕 Simplified SAM segmentation in TIAViz 🆕 Simplified SAM Segmentation in TIAViz Mar 10, 2026
@shaneahmed shaneahmed merged commit fcbd3d1 into develop Mar 10, 2026
31 checks passed
@shaneahmed shaneahmed deleted the sam-viz-fix branch March 10, 2026 12:10
@shaneahmed shaneahmed mentioned this pull request Mar 11, 2026
Merged
shaneahmed added a commit that referenced this pull request Mar 11, 2026
@shaneahmed @measty
@Jiaqi-Lv @adamshephard @mostafajahanifar @John-P @pre-commit-ci @YijieZhu15 @aacic @AbishekRajVG @behnazelhaminia @gozdeg @github-code-quality @github-advanced-security @mbasheer04 @vqdang
🔖 Release 2.0.0 (#1031)
2275bf0 
## TIAToolbox v2.0.0 (2026-03-11)

### ✨ Major Updates and Feature Improvements

#### ⚙️ Engine Redesign (PR #578)
TIAToolbox 2.0.0 introduces a completely re-engineered inference engine designed for significant performance, scalability, and memory-efficiency improvements.

#### Key Enhancements
- A modern processing stack built on **Dask** (parallel/distributed execution) and **Zarr** (chunked, out-of-core storage)
- **Standardised output formats** across all engines:
  - Python `dict`
  - **Zarr**
  - **AnnotationStore** (SQLite-backed)
  - **QuPath JSON**
- Cleaner runtime behavior with reduced warning noise and a unified progress bar
- More predictable memory usage through chunked streaming
- Broader test coverage across engine components

### 🗺️ Improved QuPath Support
Enhancements include:

- Better handling of **GeoJSON**
- Support for **multipoint geometries** (#841)
- Improved semantic output helpers:
  - `dict_to_store_semantic_segmentor` (#926)
  - OME-TIFF probability overlays (#929)

### 🔬 New Nucleus Detection Engine
A dedicated nucleus detection pipeline has been added, built on the redesigned engine for improved accuracy and efficient large-scale processing.

#### 🧠 KongNet Model Family
TIAToolbox 2.0.0 introduces **KongNet**, a high-performance architecture that achieved top results across multiple international challenges:

- 🥇 **1st place: MONKEY Challenge (overall detection)**
- 🥇 **1st place: MIDOG (mitosis detection)**
- ⭐ Top-tier performance on **PUMA**

Multiple pretrained variants are available (CoNIC, PanNuke, MONKEY, PUMA, MIDOG), each with standardised IO configurations.

### 🧬 Expanded Foundation Model Support
Additional foundation models are now supported (#906), broadening the range of high-capacity architectures available for feature extraction and downstream tasks.

### 🖼️ SAM Segmentation in TIAViz
TIAViz now integrates Meta’s Segment Anything Model (SAM), enabling:

- Interactive segmentation
- Rapid region extraction
- Exploratory annotation workflows

Simplified SAM usage (#968) streamlines its integration into analysis pipelines.

### 🧩 Enhanced WSIReader & Metadata Handling
Major improvements include:

- More robust cross-vendor **metadata extraction** (#1001)
- **Multichannel image support** (PR #825) for immunofluorescence and non-RGB modalities
- Simplified Windows installation using `openslide-bin` (no manual DLL steps)
- macOS Tileserver fix (#976)
- Improved DICOM reading (#934)

### ☁️ New Cloud-Native Reader: FsspecJSONWSIReader (PR #897)
A new reader supporting **fsspec-compatible filesystems**, enabling seamless access to WSIs stored on:

- S3
- GCS
- Azure
- HPC clusters
- Any fsspec-supported backend

This enables cloud-native and distributed data workflows.
Contributed by @aacic

### 🤗 Pretrained Models Migrated to Hugging Face
All pretrained models and sample assets have been migrated (#945, #983), improving:

- Download reliability
- Versioning and reproducibility
- Caching and CI integration
- Licensing clarity per model family

### 🛡️ Security, Compatibility & Tooling

#### 🔐 Security & Dependency Updates
- Dependency upgrades
- Internal security improvements
- Explicit workflow permissions added (#1021, #1023)

#### 🐍 Python Version Support
- **Dropped:** Python **3.9**
- **Added:** Python **3.13**
- **Supported:** Python 3.10–3.13
- Updated CUDA wheel source to **cu126**

#### 🛠️ Developer Tooling & CI/CD
- Expanded **mypy** type-checking coverage (#912, #931, #935, #951)
- Updated pre-commit hooks and general formatting
- CI uses **CPU-only PyTorch** for faster, more reliable builds (#974, #979)
- Updated pip install workflow (#1013)
- Added new **Python 3.13 Docker images** (#1014, #1019)

### 🧹 Bug Fixes & Stability Improvements
- Fixed multi-GPU behaviour with `torch.compile` (#923)
- Fixed DICOM reading issue (#934)
- Fixed annotation contour handling with holes (#956)
- Fixed consecutive annotation load bug (#927)
- Fixed SCCNN model issues (#970)
- Fixed MapDe `dist_filter` shape issue (#914)
- Improved notebook reliability on Colab (#1026–#1030)
- macOS TileServer issues resolved (#976)

### 🧭 Migration Guide for Users

#### 🔄 Updating from 1.x to 2.0.0

#### Update calls: replace `.predict()` with `.run()`
```python
# Old
results = segmentor.predict(imgs=[...], ioconfig=config)

# New
results = segmentor.run(images=[...], ioconfig=config)
```

#### Use `patch_mode`: replace `mode="patch"` with `patch_mode=True` and `mode="tile"` or "wsi" with `patch_mode=False`
```python
# Old
results = segmentor.predict(imgs=[...], mode="patch", ioconfig=config)

# New
results = segmentor.run(images=[...], patch_mode=True, ioconfig=config)
```

```python
# Old
results = segmentor.predict(imgs=[...], mode="wsi", ioconfig=config)

# New
results = segmentor.run(images=[...], patch_mode=False, ioconfig=config)

```

#### Use the new I/O configs
```python
from tiatoolbox.models.engine.io_config import IOSegmentorConfig

config = IOSegmentorConfig(
    patch_input_shape=(256, 256),
    stride_shape=(240, 240),
    input_resolutions=[{"resolution": 0.25, "units": "mpp"}],
    save_resolution={"units": "baseline", "resolution": 1.0}
)
```

#### Specify the output format
```python
results = segmentor.run(
    images=[...],
    ioconfig=ioconfig,
    output_type="zarr",  # or "dict", "annotationstore", "qupath"
    save_dir="outputs/"
)
```

#### Update imports
- `tiatoolbox.typing` → `tiatoolbox.type_hints`

#### Install requirements
- Python **3.10+** required
- On Windows: install OpenSlide via `pip install openslide-bin`

**Full Changelog:** v1.6.0...v2.0.0

---------

Signed-off-by: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>
Co-authored-by: measty <20169086+measty@users.noreply.github.com>
Co-authored-by: Jiaqi-Lv <60471431+Jiaqi-Lv@users.noreply.github.com>
Co-authored-by: adamshephard <39619155+adamshephard@users.noreply.github.com>
Co-authored-by: Mostafa Jahanifar <74412979+mostafajahanifar@users.noreply.github.com>
Co-authored-by: John Pocock <John-P@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Yijie Zhu <120978607+YijieZhu15@users.noreply.github.com>
Co-authored-by: Aleksandar Acic <32873451+aacic@users.noreply.github.com>
Co-authored-by: Abdol A <u2271662@live.warwick.ac.uk>
Co-authored-by: Abishek <abishekraj6797@gmail.com>
Co-authored-by: behnazelhaminia <30952176+behnazelhaminia@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Adam Shephard <adam.shephard@warwick.ac.uk>
Co-authored-by: gozdeg <gozdegunesli@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: mbasheer04 <78800844+mbasheer04@users.noreply.github.com>
Co-authored-by: vqdang <24943262+vqdang@users.noreply.github.com>
@shaneahmed shaneahmed mentioned this pull request Mar 11, 2026
Merged
shaneahmed added a commit that referenced this pull request Mar 12, 2026
@shaneahmed @measty
@Jiaqi-Lv @adamshephard @mostafajahanifar @John-P @pre-commit-ci @YijieZhu15 @aacic @AbishekRajVG @behnazelhaminia @gozdeg @github-code-quality @github-advanced-security @mbasheer04 @vqdang
🔖 Release 2.0.0 (#1033)
ce9a2d0 
🔖 Release 2.0.0 (#1031)
## TIAToolbox v2.0.0 (2026-03-11)

### ✨ Major Updates and Feature Improvements

#### ⚙️ Engine Redesign (PR #578)
TIAToolbox 2.0.0 introduces a completely re-engineered inference engine designed for significant performance, scalability, and memory-efficiency improvements.

#### Key Enhancements
- A modern processing stack built on **Dask** (parallel/distributed execution) and **Zarr** (chunked, out-of-core storage)
- **Standardised output formats** across all engines:
  - Python `dict`
  - **Zarr**
  - **AnnotationStore** (SQLite-backed)
  - **QuPath JSON**
- Cleaner runtime behavior with reduced warning noise and a unified progress bar
- More predictable memory usage through chunked streaming
- Broader test coverage across engine components

### 🗺️ Improved QuPath Support
Enhancements include:

- Better handling of **GeoJSON**
- Support for **multipoint geometries** (#841)
- Improved semantic output helpers:
  - `dict_to_store_semantic_segmentor` (#926)
  - OME-TIFF probability overlays (#929)

### 🔬 New Nucleus Detection Engine
A dedicated nucleus detection pipeline has been added, built on the redesigned engine for improved accuracy and efficient large-scale processing.

#### 🧠 KongNet Model Family
TIAToolbox 2.0.0 introduces **KongNet**, a high-performance architecture that achieved top results across multiple international challenges:

- 🥇 **1st place: MONKEY Challenge (overall detection)**
- 🥇 **1st place: MIDOG (mitosis detection)**
- ⭐ Top-tier performance on **PUMA**

Multiple pretrained variants are available (CoNIC, PanNuke, MONKEY, PUMA, MIDOG), each with standardised IO configurations.

### 🧬 Expanded Foundation Model Support
Additional foundation models are now supported (#906), broadening the range of high-capacity architectures available for feature extraction and downstream tasks.

### 🖼️ SAM Segmentation in TIAViz
TIAViz now integrates Meta’s Segment Anything Model (SAM), enabling:

- Interactive segmentation
- Rapid region extraction
- Exploratory annotation workflows

Simplified SAM usage (#968) streamlines its integration into analysis pipelines.

### 🧩 Enhanced WSIReader & Metadata Handling
Major improvements include:

- More robust cross-vendor **metadata extraction** (#1001)
- **Multichannel image support** (PR #825) for immunofluorescence and non-RGB modalities
- Simplified Windows installation using `openslide-bin` (no manual DLL steps)
- macOS Tileserver fix (#976)
- Improved DICOM reading (#934)

### ☁️ New Cloud-Native Reader: FsspecJSONWSIReader (PR #897)
A new reader supporting **fsspec-compatible filesystems**, enabling seamless access to WSIs stored on:

- S3
- GCS
- Azure
- HPC clusters
- Any fsspec-supported backend

This enables cloud-native and distributed data workflows.
Contributed by @aacic

### 🤗 Pretrained Models Migrated to Hugging Face
All pretrained models and sample assets have been migrated (#945, #983), improving:

- Download reliability
- Versioning and reproducibility
- Caching and CI integration
- Licensing clarity per model family

### 🛡️ Security, Compatibility & Tooling

#### 🔐 Security & Dependency Updates
- Dependency upgrades
- Internal security improvements
- Explicit workflow permissions added (#1021, #1023)

#### 🐍 Python Version Support
- **Dropped:** Python **3.9**
- **Added:** Python **3.13**
- **Supported:** Python 3.10–3.13
- Updated CUDA wheel source to **cu126**

#### 🛠️ Developer Tooling & CI/CD
- Expanded **mypy** type-checking coverage (#912, #931, #935, #951)
- Updated pre-commit hooks and general formatting
- CI uses **CPU-only PyTorch** for faster, more reliable builds (#974, #979)
- Updated pip install workflow (#1013)
- Added new **Python 3.13 Docker images** (#1014, #1019)

### 🧹 Bug Fixes & Stability Improvements
- Fixed multi-GPU behaviour with `torch.compile` (#923)
- Fixed DICOM reading issue (#934)
- Fixed annotation contour handling with holes (#956)
- Fixed consecutive annotation load bug (#927)
- Fixed SCCNN model issues (#970)
- Fixed MapDe `dist_filter` shape issue (#914)
- Improved notebook reliability on Colab (#1026–#1030)
- macOS TileServer issues resolved (#976)

### 🧭 Migration Guide for Users

#### 🔄 Updating from 1.x to 2.0.0

#### Update calls: replace `.predict()` with `.run()`
```python
# Old
results = segmentor.predict(imgs=[...], ioconfig=config)

# New
results = segmentor.run(images=[...], ioconfig=config)
```

#### Use `patch_mode`: replace `mode="patch"` with `patch_mode=True` and `mode="tile"` or "wsi" with `patch_mode=False`
```python
# Old
results = segmentor.predict(imgs=[...], mode="patch", ioconfig=config)

# New
results = segmentor.run(images=[...], patch_mode=True, ioconfig=config)
```

```python
# Old
results = segmentor.predict(imgs=[...], mode="wsi", ioconfig=config)

# New
results = segmentor.run(images=[...], patch_mode=False, ioconfig=config)

```

#### Use the new I/O configs
```python
from tiatoolbox.models.engine.io_config import IOSegmentorConfig

config = IOSegmentorConfig(
    patch_input_shape=(256, 256),
    stride_shape=(240, 240),
    input_resolutions=[{"resolution": 0.25, "units": "mpp"}],
    save_resolution={"units": "baseline", "resolution": 1.0}
)
```

#### Specify the output format
```python
results = segmentor.run(
    images=[...],
    ioconfig=ioconfig,
    output_type="zarr",  # or "dict", "annotationstore", "qupath"
    save_dir="outputs/"
)
```

#### Update imports
- `tiatoolbox.typing` → `tiatoolbox.type_hints`

#### Install requirements
- Python **3.10+** required
- On Windows: install OpenSlide via `pip install openslide-bin`

**Full Changelog:** v1.6.0...v2.0.0

---------

Signed-off-by: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>
Co-authored-by: measty <20169086+measty@users.noreply.github.com>
Co-authored-by: Jiaqi-Lv <60471431+Jiaqi-Lv@users.noreply.github.com>
Co-authored-by: adamshephard <39619155+adamshephard@users.noreply.github.com>
Co-authored-by: Mostafa Jahanifar <74412979+mostafajahanifar@users.noreply.github.com>
Co-authored-by: John Pocock <John-P@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Yijie Zhu <120978607+YijieZhu15@users.noreply.github.com>
Co-authored-by: Aleksandar Acic <32873451+aacic@users.noreply.github.com>
Co-authored-by: Abdol A <u2271662@live.warwick.ac.uk>
Co-authored-by: Abishek <abishekraj6797@gmail.com>
Co-authored-by: behnazelhaminia <30952176+behnazelhaminia@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Adam Shephard <adam.shephard@warwick.ac.uk>
Co-authored-by: gozdeg <gozdegunesli@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: mbasheer04 <78800844+mbasheer04@users.noreply.github.com>
Co-authored-by: vqdang <24943262+vqdang@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@shaneahmed shaneahmed shaneahmed approved these changes

@adamshephard adamshephard adamshephard approved these changes

+1 more reviewer

@github-code-quality github-code-quality[bot] github-code-quality[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

@measty measty

Labels

enhancement New feature or request

Projects

None yet

Milestone

Release v2.0.0

Development

Successfully merging this pull request may close these issues.

5 participants

@measty @shaneahmed @adamshephard @mbasheer04