The current implementation of get_data directly calls save, which introduces a risk when used in multithreaded contexts. The disk I/O performed by save is not protected, but locking around it has serious performance implications.
In my own private implementation of a REST API client for PokeAPI in python (sorry just found out about this project), I solved this by separating in‑memory caching from disk persistence. My approach was:
- Keep an in‑memory cache updated freely across threads.
- Disable automatic disk saving during concurrent operations.
- Manually invoke a disk save step once threads have finished, ensuring serialization and consistency.
One note is it would likely involve not using shelve as I believe that is not thread safe. But other features may be more suited.
With approval, I am willing to begin migrating my code to use pokebase, implement and test a option to disable auto‑save and give manual control.
The current implementation of get_data directly calls save, which introduces a risk when used in multithreaded contexts. The disk I/O performed by save is not protected, but locking around it has serious performance implications.
In my own private implementation of a REST API client for PokeAPI in python (sorry just found out about this project), I solved this by separating in‑memory caching from disk persistence. My approach was:
One note is it would likely involve not using
shelveas I believe that is not thread safe. But other features may be more suited.With approval, I am willing to begin migrating my code to use pokebase, implement and test a option to disable auto‑save and give manual control.