Skip to content
This repository was archived by the owner on May 9, 2024. It is now read-only.
This repository was archived by the owner on May 9, 2024. It is now read-only.

TF Dataset to TF Examples List in FDW Utils: Implementation is slow and consumes excessive RAM #31

@CLSchmitz

Description

@CLSchmitz

The current implementation of TF Dataset to TF Examples List uses a lot of RAM and runs very slowly on large datasets.

In this colab, the CelebA dataset, as processed by the fairness indicators tutorial, can't be processed on a default hosted runtime. After 18-20 minutes, the runtime crashes because RAM requirements are exceeded.

Because of all the preprocessing required, it's not easy to create a smaller reproducible example.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions