-
- Opinionated web-interface around the music organizer beets -
- - - -## Features - - - -- Autogenerate previews before importing -- Auto-Import good matches -- Import via GUI -- Undo imports -- Web-Terminal -- Monitor multiple inboxes -- Library view and search - - - -https://github.com/user-attachments/assets/b7c7ae32-36ac-4950-b2ed-186b80d7075b - -## Motivation - - - -Autotagging music with beets is great. Beets identifies metadata correctly _most_ of the time, and if you are not a control-freak, there is hardly any reason to check the found metadata. - -However, if you do want a bit more control, things could be more convenient. - -This is the main idea with beets-flask: For all folders in your inbox, we generate a preview of what beets _would do_ and show you those previews. Then it's easy to go through them and import the correct ones, or pick other candidates for those that were not to your liking. - - - -## Quickstart - -We provide a docker image with the full beeets-flask setup. You can run it with docker-compose or docker. We recommend using the `stable` tag, alternatively you may use `latest` for the most recent build. - - -### Setup container - -**Using docker** - - - -```sh -docker run -d -p 5001:5001 \ - -e TZ=Europe/Berlin \ - -e USER_ID=1000 \ - -e GROUP_ID=1000 \ - -v /wherever/config/:/config \ - -v /music_path/inbox/:/music_path/inbox/ \ - -v /music_path/clean/:/music_path/clean/ \ - --name beets-flask \ - pspitzner/beets-flask:stable -``` - - - -**Using docker compose** - -```yaml -services: - beets-flask: - image: pspitzner/beets-flask:stable - restart: unless-stopped - ports: - - "5001:5001" - environment: - # 502 is default on macos, 1000 on linux - TZ: Europe/Berlin - USER_ID: 1000 - GROUP_ID: 1000 - volumes: - - /wherever/config/:/config - # for music folders, match paths inside and out of container! - - /music_path/inbox/:/music_path/inbox/ - - /music_path/clean/:/music_path/clean/ -``` - -This will create a container with the following folder structure: - -``` -├── music_path -│ ├── inbox -│ └── clean -└── config - ├── beets - │ ├── config.yaml - │ └── library.db - └── beets-flask - ├── config.yaml - └── beets-flask-sqlite.db -``` - -Check our [**documentation**](https://beets-flask.readthedocs.io/en/latest/) for more information! - -## Limitations - -There are a few known issues, see the corresponding [docs section](https://beets-flask.readthedocs.io/latest/limitations.html) for details. - -## Support the Project - -If you enjoy this project, there are a few ways you can support us: - -- **Contribute code:** Pull requests, bug reports, and feature suggestions are always welcome! -- **Spread the word:** Share the project with friends or on social media. -- **Donate:** Every contribution helps fuel more coffee-powered coding sessions! - - **Donate ETH:** [0x81927e76f2f0fAA9e7fD92176a473955DB20Ce55](ethereum:0x81927e76f2f0fAA9e7fD92176a473955DB20Ce55) + ++
+ Opinionated web-interface around the music organizer beets +
+ + + +## Features + + + +- Autogenerate previews before importing +- Auto-Import good matches +- Import via GUI +- Undo imports +- Web-Terminal +- Monitor multiple inboxes +- Library view and search + + + +https://github.com/user-attachments/assets/b7c7ae32-36ac-4950-b2ed-186b80d7075b + +## Motivation + + + +Autotagging music with beets is great. Beets identifies metadata correctly _most_ of the time, and if you are not a control-freak, there is hardly any reason to check the found metadata. + +However, if you do want a bit more control, things could be more convenient. + +This is the main idea with beets-flask: For all folders in your inbox, we generate a preview of what beets _would do_ and show you those previews. Then it's easy to go through them and import the correct ones, or pick other candidates for those that were not to your liking. + + + +## Quickstart + +We provide a docker image with the full beeets-flask setup. You can run it with docker-compose or docker. We recommend using the `stable` tag, alternatively you may use `latest` for the most recent build. + + +### Setup container + +**Using docker** + + + +```sh +docker run -d -p 5001:5001 \ + -e TZ=Europe/Berlin \ + -e USER_ID=1000 \ + -e GROUP_ID=1000 \ + -v /wherever/config/:/config \ + -v /music_path/inbox/:/music_path/inbox/ \ + -v /music_path/clean/:/music_path/clean/ \ + --name beets-flask \ + pspitzner/beets-flask:stable +``` + + + +**Using docker compose** + +```yaml +services: + beets-flask: + image: pspitzner/beets-flask:stable + restart: unless-stopped + ports: + - "5001:5001" + environment: + # 502 is default on macos, 1000 on linux + TZ: Europe/Berlin + USER_ID: 1000 + GROUP_ID: 1000 + volumes: + - /wherever/config/:/config + # for music folders, match paths inside and out of container! + - /music_path/inbox/:/music_path/inbox/ + - /music_path/clean/:/music_path/clean/ +``` + +This will create a container with the following folder structure: + +``` +├── music_path +│ ├── inbox +│ └── clean +└── config + ├── beets + │ ├── config.yaml + │ └── library.db + └── beets-flask + ├── config.yaml + └── beets-flask-sqlite.db +``` + +Check our [**documentation**](https://beets-flask.readthedocs.io/en/latest/) for more information! + +## Limitations + +There are a few known issues, see the corresponding [docs section](https://beets-flask.readthedocs.io/latest/limitations.html) for details. + +## Support the Project + +If you enjoy this project, there are a few ways you can support us: + +- **Contribute code:** Pull requests, bug reports, and feature suggestions are always welcome! +- **Spread the word:** Share the project with friends or on social media. +- **Donate:** Every contribution helps fuel more coffee-powered coding sessions! + - **Donate ETH:** [0x81927e76f2f0fAA9e7fD92176a473955DB20Ce55](ethereum:0x81927e76f2f0fAA9e7fD92176a473955DB20Ce55) - **Donate BTC:** [bc1qw5e0deust6uq94e5s58au82wrakcjmlemw3cy4](bitcoin:bc1qw5e0deust6uq94e5s58au82wrakcjmlemw3cy4) \ No newline at end of file diff --git a/backend/beets_flask/__init__.py b/backend/beets_flask/__init__.py index dce63995..acd8d0da 100644 --- a/backend/beets_flask/__init__.py +++ b/backend/beets_flask/__init__.py @@ -1,3 +1,3 @@ -from .logger import log - -__all__ = ["log"] +from .logger import log + +__all__ = ["log"] diff --git a/backend/beets_flask/config/__init__.py b/backend/beets_flask/config/__init__.py index 645d4c2f..47e047c9 100644 --- a/backend/beets_flask/config/__init__.py +++ b/backend/beets_flask/config/__init__.py @@ -1,19 +1,19 @@ -from .beets_config import get_config -from .flask_config import ( - DeploymentDocker, - DevelopmentDocker, - DevelopmentLocal, - ServerConfig, - Testing, - get_flask_config, -) - -__all__ = [ - "get_config", - "get_flask_config", - "ServerConfig", - "Testing", - "DevelopmentLocal", - "DevelopmentDocker", - "DeploymentDocker", -] +from .beets_config import get_config +from .flask_config import ( + DeploymentDocker, + DevelopmentDocker, + DevelopmentLocal, + ServerConfig, + Testing, + get_flask_config, +) + +__all__ = [ + "get_config", + "get_flask_config", + "ServerConfig", + "Testing", + "DevelopmentLocal", + "DevelopmentDocker", + "DeploymentDocker", +] diff --git a/backend/beets_flask/config/beets_config.py b/backend/beets_flask/config/beets_config.py index 196640ea..1b65ac93 100644 --- a/backend/beets_flask/config/beets_config.py +++ b/backend/beets_flask/config/beets_config.py @@ -1,265 +1,265 @@ -"""Overload for beets configuration. - -We support setting config values either via your beets config file, under the `gui` section, or via environment variables in the Docker compose. - -Use double underscore to separate nested values: -https://confuse.readthedocs.io/en/latest/usage.html#environment-variables - -We prefix all environment variables with `IB` to avoid conflicts with other services. - -To set a custom file path to a yaml that gets inserted into (and overwrites) the -beets config, set the `IB_GUI_CONFIGPATH` environment variable. -Note that this does not remove list keys from the lower priority default config -(e.g. if you configure different inbox folders in your beets config, and the ib config, -all of them will be added). - -# Example: - -```bash -export IB_GUI__TAGS=first -``` - -```python -from beets_flask.beets_config.config import config -print(config["gui"]["tags"].get(default="default_value")) -``` -""" - -import os -import sys -from typing import cast - -from beets import IncludeLazyConfig as BeetsConfig -from beets.plugins import _instances as plugin_instances -from beets.plugins import get_plugin_names, load_plugins -from confuse import YamlSource - -from beets_flask.logger import log - - -def _copy_file(src, dest): - with open(src) as src_file, open(dest, "w") as dest_file: - dest_file.write(src_file.read()) - - -class Singleton(type): - _instances: dict = {} - - def __call__(cls, *args, **kwargs): - """Singleton pattern implementation.""" - if cls not in cls._instances: - cls._instances[cls] = super().__call__(*args, **kwargs) - return cls._instances[cls] - - -class InteractiveBeetsConfig(BeetsConfig, metaclass=Singleton): - """Singleton class to handle the beets config. - - This class is a subclass of the beets config and adds some - interactive beets specific functionality. - """ - - def __init__(self): - """Initialize the config object with the default values. - - Loads config and some interactive beets specific tweaks. - """ - super().__init__("beets", "beets") - self.reset() - - @staticmethod - def get_beets_flask_config_path() -> str: - """Get the path to the beets-flask config file.""" - - ib_folder = os.getenv("BEETSFLASKDIR") - if ib_folder is None: - ib_folder = os.path.expanduser("~/.config/beets-flask") - os.makedirs(ib_folder, exist_ok=True) - ib_config_path = os.path.join(ib_folder, "config.yaml") - return ib_config_path - - @staticmethod - def get_beets_config_path() -> str: - """Get the path to the beets config file.""" - # TODO: maybe there is a beets function to get the path - beets_folder = os.getenv("BEETSDIR") - if beets_folder is None: - beets_folder = os.path.expanduser("~/.config/beets") - beets_config_path = os.path.join(beets_folder, "config.yaml") - return beets_config_path - - def reset(self): - """Recreate the config object. - - As if the app was just started. - """ - # vanilla beets reset - log.debug(f"Reading beets config from default location") - self.clear() - self.read() - - # read the default config just in case the user config is missing - # or malformed - ib_defaults_path = os.path.join( - os.path.dirname(__file__), "config_bf_default.yaml" - ) - log.debug(f"Reading IB config defaults from {ib_defaults_path}") - default_source = YamlSource(ib_defaults_path, default=True) - self.add(default_source) # .add inserts with lowest priority - - # then apply our needed tweaks - # enable env variables - self.set_env(prefix="IB") - - # Load config from default location (set via env var) - # if it is set otherwise use the default location - ib_config_path = self.get_beets_flask_config_path() - - # Check if the user config exists - # if not, copy the example config to the user config location - if not os.path.exists(ib_config_path): - # Copy the default config to the user config location - log.debug(f"Beets-flask config not found at {ib_config_path}") - log.debug(f"Copying default config to {ib_config_path}") - ib_example_path = os.path.join( - os.path.dirname(__file__), "config_bf_example.yaml" - ) - _copy_file(ib_example_path, ib_config_path) - - # Same check for beets config and copy our default - # if it does not exist - beets_config_path = self.get_beets_config_path() - if not os.path.exists(beets_config_path): - log.debug(f"Beets config not found at {beets_config_path}") - log.debug(f"Copying default config to {beets_config_path}") - beets_example_path = os.path.join( - os.path.dirname(__file__), "config_b_example.yaml" - ) - _copy_file(beets_example_path, beets_config_path) - - # Inserts user config at highest priority - log.debug(f"Reading beets-flask config from {ib_config_path}") - self.set(YamlSource(ib_config_path, default=False)) - - # add placeholders for required keys if they are not configured, - # so the docker container starts and can show some help. - - # beets does not create a config file automatically for the user. Customizations are added as extra layers on the config. - sources = [s for s in beets.config["directory"].resolve()] - if len(sources) == 1: - log.debug( - "Beets is not using a user config. Overwriting the default `directory`." - ) - self["directory"] = "/music/imported" - - # TODO: would be nice to have this in the default config, - # and simply remove it here if other inbox folders have been configured. - # but: I do not know how to remove (some) elements from a confuse config. - if len(self["gui"]["inbox"]["folders"].keys()) == 0: - self["gui"]["inbox"]["folders"]["Placeholder"] = { - "name": "Please check your config!", - "path": "/music/inbox", - "autotag": False, - } - - # make sure to remove trailing slashes from user configured inbox paths - for folder in self["gui"]["inbox"]["folders"].values(): - fp: str = folder["path"].as_str() # type: ignore - if fp.endswith("/"): - folder["path"] = fp.rstrip("/") - log.debug(f"Removed trailing slash from inbox path: {folder['path']}") - - @property - def ignore_globs(self) -> list[str]: - """ - Get the list of ignore globs from the config. - - If user does not set this in their beets flask config, we use whats in beets. - (We do this via a placeholder string "_use_beets_ignore") - If the user sets an empty list [], that means no files are ignored. - - """ - gui_globs: list[str] | str = get_config()["gui"]["inbox"]["ignore"].get() # type: ignore - if gui_globs is None or gui_globs == "_use_beets_ignore": - gui_globs: list[str] = self["ignore"].as_str_seq() # type: ignore - elif isinstance(gui_globs, str): - gui_globs = [gui_globs] - elif isinstance(gui_globs, list): - gui_globs = gui_globs - return cast(list[str], gui_globs) - - -# Monkey patch the beets config -import beets - -config: InteractiveBeetsConfig | None = None - - -def refresh_config(): - """Refresh the config object. - - This is useful if you want to reload the config after it has been changed. - """ - global config - - # Keep reference to old config - old_config = getattr(beets, "config", None) - - config = InteractiveBeetsConfig() - - beets.config = config - sys.modules["beets"].config = config # type: ignore - - # Hack: We have to manually load the plugins as this - # is normally done by beets. Clear the list to force - # actual reload. - plugin_instances.clear() - load_plugins() - log.debug(f"Loading plugins: {get_plugin_names()}") - - # Update any existing references in other modules - for module_name, mod in list(sys.modules.items()): - if mod is None: - continue - - if not ( - module_name.startswith("beets") # includes beets and beetsplug - ): - continue - - for attr_name in dir(mod): - try: - if getattr(mod, attr_name) is old_config: - setattr(mod, attr_name, config) - log.debug(f"Updated config in {module_name}.{attr_name}") - except Exception as e: - log.debug(f"Could not check {module_name}.{attr_name}", exc_info=e) - continue - - return config - - -def get_config(force_refresh=False) -> InteractiveBeetsConfig: - """Get the config object. - - This is useful if you want to access the config from another module. - - The result of this function is still the global object that you can mutate! - - Parameters - ---------- - force_refresh : bool - Force a refresh of the config object. - This is useful if you want to be sure that the config is up to date, - should normally only be called if the config was changed in another process. - """ - global config - - if config is None or force_refresh: - return refresh_config() - return config - - -__all__ = ["refresh_config", "get_config"] - -# raise NotImplementedError("This module should not be imported.") +"""Overload for beets configuration. + +We support setting config values either via your beets config file, under the `gui` section, or via environment variables in the Docker compose. + +Use double underscore to separate nested values: +https://confuse.readthedocs.io/en/latest/usage.html#environment-variables + +We prefix all environment variables with `IB` to avoid conflicts with other services. + +To set a custom file path to a yaml that gets inserted into (and overwrites) the +beets config, set the `IB_GUI_CONFIGPATH` environment variable. +Note that this does not remove list keys from the lower priority default config +(e.g. if you configure different inbox folders in your beets config, and the ib config, +all of them will be added). + +# Example: + +```bash +export IB_GUI__TAGS=first +``` + +```python +from beets_flask.beets_config.config import config +print(config["gui"]["tags"].get(default="default_value")) +``` +""" + +import os +import sys +from typing import cast + +from beets import IncludeLazyConfig as BeetsConfig +from beets.plugins import _instances as plugin_instances +from beets.plugins import get_plugin_names, load_plugins +from confuse import YamlSource + +from beets_flask.logger import log + + +def _copy_file(src, dest): + with open(src) as src_file, open(dest, "w") as dest_file: + dest_file.write(src_file.read()) + + +class Singleton(type): + _instances: dict = {} + + def __call__(cls, *args, **kwargs): + """Singleton pattern implementation.""" + if cls not in cls._instances: + cls._instances[cls] = super().__call__(*args, **kwargs) + return cls._instances[cls] + + +class InteractiveBeetsConfig(BeetsConfig, metaclass=Singleton): + """Singleton class to handle the beets config. + + This class is a subclass of the beets config and adds some + interactive beets specific functionality. + """ + + def __init__(self): + """Initialize the config object with the default values. + + Loads config and some interactive beets specific tweaks. + """ + super().__init__("beets", "beets") + self.reset() + + @staticmethod + def get_beets_flask_config_path() -> str: + """Get the path to the beets-flask config file.""" + + ib_folder = os.getenv("BEETSFLASKDIR") + if ib_folder is None: + ib_folder = os.path.expanduser("~/.config/beets-flask") + os.makedirs(ib_folder, exist_ok=True) + ib_config_path = os.path.join(ib_folder, "config.yaml") + return ib_config_path + + @staticmethod + def get_beets_config_path() -> str: + """Get the path to the beets config file.""" + # TODO: maybe there is a beets function to get the path + beets_folder = os.getenv("BEETSDIR") + if beets_folder is None: + beets_folder = os.path.expanduser("~/.config/beets") + beets_config_path = os.path.join(beets_folder, "config.yaml") + return beets_config_path + + def reset(self): + """Recreate the config object. + + As if the app was just started. + """ + # vanilla beets reset + log.debug(f"Reading beets config from default location") + self.clear() + self.read() + + # read the default config just in case the user config is missing + # or malformed + ib_defaults_path = os.path.join( + os.path.dirname(__file__), "config_bf_default.yaml" + ) + log.debug(f"Reading IB config defaults from {ib_defaults_path}") + default_source = YamlSource(ib_defaults_path, default=True) + self.add(default_source) # .add inserts with lowest priority + + # then apply our needed tweaks + # enable env variables + self.set_env(prefix="IB") + + # Load config from default location (set via env var) + # if it is set otherwise use the default location + ib_config_path = self.get_beets_flask_config_path() + + # Check if the user config exists + # if not, copy the example config to the user config location + if not os.path.exists(ib_config_path): + # Copy the default config to the user config location + log.debug(f"Beets-flask config not found at {ib_config_path}") + log.debug(f"Copying default config to {ib_config_path}") + ib_example_path = os.path.join( + os.path.dirname(__file__), "config_bf_example.yaml" + ) + _copy_file(ib_example_path, ib_config_path) + + # Same check for beets config and copy our default + # if it does not exist + beets_config_path = self.get_beets_config_path() + if not os.path.exists(beets_config_path): + log.debug(f"Beets config not found at {beets_config_path}") + log.debug(f"Copying default config to {beets_config_path}") + beets_example_path = os.path.join( + os.path.dirname(__file__), "config_b_example.yaml" + ) + _copy_file(beets_example_path, beets_config_path) + + # Inserts user config at highest priority + log.debug(f"Reading beets-flask config from {ib_config_path}") + self.set(YamlSource(ib_config_path, default=False)) + + # add placeholders for required keys if they are not configured, + # so the docker container starts and can show some help. + + # beets does not create a config file automatically for the user. Customizations are added as extra layers on the config. + sources = [s for s in beets.config["directory"].resolve()] + if len(sources) == 1: + log.debug( + "Beets is not using a user config. Overwriting the default `directory`." + ) + self["directory"] = "/music/imported" + + # TODO: would be nice to have this in the default config, + # and simply remove it here if other inbox folders have been configured. + # but: I do not know how to remove (some) elements from a confuse config. + if len(self["gui"]["inbox"]["folders"].keys()) == 0: + self["gui"]["inbox"]["folders"]["Placeholder"] = { + "name": "Please check your config!", + "path": "/music/inbox", + "autotag": False, + } + + # make sure to remove trailing slashes from user configured inbox paths + for folder in self["gui"]["inbox"]["folders"].values(): + fp: str = folder["path"].as_str() # type: ignore + if fp.endswith("/"): + folder["path"] = fp.rstrip("/") + log.debug(f"Removed trailing slash from inbox path: {folder['path']}") + + @property + def ignore_globs(self) -> list[str]: + """ + Get the list of ignore globs from the config. + + If user does not set this in their beets flask config, we use whats in beets. + (We do this via a placeholder string "_use_beets_ignore") + If the user sets an empty list [], that means no files are ignored. + + """ + gui_globs: list[str] | str = get_config()["gui"]["inbox"]["ignore"].get() # type: ignore + if gui_globs is None or gui_globs == "_use_beets_ignore": + gui_globs: list[str] = self["ignore"].as_str_seq() # type: ignore + elif isinstance(gui_globs, str): + gui_globs = [gui_globs] + elif isinstance(gui_globs, list): + gui_globs = gui_globs + return cast(list[str], gui_globs) + + +# Monkey patch the beets config +import beets + +config: InteractiveBeetsConfig | None = None + + +def refresh_config(): + """Refresh the config object. + + This is useful if you want to reload the config after it has been changed. + """ + global config + + # Keep reference to old config + old_config = getattr(beets, "config", None) + + config = InteractiveBeetsConfig() + + beets.config = config + sys.modules["beets"].config = config # type: ignore + + # Hack: We have to manually load the plugins as this + # is normally done by beets. Clear the list to force + # actual reload. + plugin_instances.clear() + load_plugins() + log.debug(f"Loading plugins: {get_plugin_names()}") + + # Update any existing references in other modules + for module_name, mod in list(sys.modules.items()): + if mod is None: + continue + + if not ( + module_name.startswith("beets") # includes beets and beetsplug + ): + continue + + for attr_name in dir(mod): + try: + if getattr(mod, attr_name) is old_config: + setattr(mod, attr_name, config) + log.debug(f"Updated config in {module_name}.{attr_name}") + except Exception as e: + log.debug(f"Could not check {module_name}.{attr_name}", exc_info=e) + continue + + return config + + +def get_config(force_refresh=False) -> InteractiveBeetsConfig: + """Get the config object. + + This is useful if you want to access the config from another module. + + The result of this function is still the global object that you can mutate! + + Parameters + ---------- + force_refresh : bool + Force a refresh of the config object. + This is useful if you want to be sure that the config is up to date, + should normally only be called if the config was changed in another process. + """ + global config + + if config is None or force_refresh: + return refresh_config() + return config + + +__all__ = ["refresh_config", "get_config"] + +# raise NotImplementedError("This module should not be imported.") diff --git a/backend/beets_flask/config/config_b_example.yaml b/backend/beets_flask/config/config_b_example.yaml index f55d5d49..cae22eab 100644 --- a/backend/beets_flask/config/config_b_example.yaml +++ b/backend/beets_flask/config/config_b_example.yaml @@ -1,129 +1,129 @@ -# ------------------------------------------------------------------------------------ # -# BEETS CONFIG # -# ------------------------------------------------------------------------------------ # -# Opinionated example beets configuration. This file was automatically copied to -# /config/beets/config.yaml. Feel free to edit this file to customize the beets -# configuration. For more information on the beets configuration, see -# https://beets.readthedocs.io/en/stable/reference/config.html - -plugins: [ - info, - the, - fetchart, - embedart, - ftintitle, - lastgenre, - missing, - albumtypes, - scrub, - zero, - mbsync, - duplicates, - convert, - fromfilename, - inline, - edit, - spotify, # needs authentication https://docs.beets.io/en/latest/plugins/spotify.html - musicbrainz, # needs to be enabled explicitly since beets 2.4.0 - ] - -directory: /music/imported -# library: /config/beets/library.db # default location in the container - -import: - move: no - copy: yes - write: yes - log: /music/last_beets_imports.log - quiet_fallback: skip - detail: yes - duplicate_action: ask # ask|skip|merge|keep|remove - -ui: - color: yes - -# fix up output file paths -replace: - '[\\]': "" - "[_]": "-" - "[/]": "-" - '^\.+': "" - '[\x00-\x1f]': "" - '[<>:"\?\*\|]': "" - '\.$': "" - '\s+$': "" - '^\s+': "" - "^-": "" - "’": "" - "′": "" - "″": "" - "‐": "-" - -per_disc_numbering: no -asciify_paths: yes - -# adjusting the `threaded` setting of beets should currently have no effect. -# we launch our own workers for previews and only have one -# import worker that runs one import (file moving) at a time. -threaded: no - -fetchart: - minwidth: 500 - enforce_ratio: 10px # yes|no or tolerance around 1:1 ratio - sources: coverart filesystem itunes amazon spotify albumart fanarttv - -embedart: - auto: yes - ifempty: yes # whether to avoid embedding album art for files that already have art embedded. - remove_art_file: yes - -ftintitle: - auto: yes - format: (feat. {0}) - -lastgenre: # get genres from last fm - auto: yes - count: 4 - prefer_specific: yes # Sort genres by the most to least specific, rather than most to least popular. - force: yes # By default, beets will always fetch new genres, even if the files already have one - source: track # album|track - separator: "; " - fallback: "" - -match: - # autotagger tolerance [0, 1], default 0.04 - # for example, 0.1 means 90% similarity required. - # inbox-folder autotag setting 'auto' respects this and imports above the threshold (like the beets cli would) - strong_rec_thresh: 0.1 - - # customize how penalties affect the match score - distance_weights: - # source: 2.0 - # artist: 3.0 - # album: 3.0 - # media: 1.0 - # mediums: 1.0 - # year: 1.0 - # country: 0.5 - # label: 0.5 - # catalognum: 0.5 - # albumdisambig: 0.5 - # album_id: 5.0 - # tracks: 2.0 - data_source: 0.0 # do not apply penalty to any data source plugin - missing_tracks: 0.2 # If your prefer not being so picky about missing tracks. default 0.9 - # unmatched_tracks: 0.6 - # track_title: 3.0 - # track_artist: 2.0 - # track_index: 1.0 - # track_length: 2.0 - # track_id: 5.0 - -musicbrainz: - external_ids: - discogs: yes - bandcamp: yes - spotify: yes - deezer: yes - beatport: yes - tidal: yes +# ------------------------------------------------------------------------------------ # +# BEETS CONFIG # +# ------------------------------------------------------------------------------------ # +# Opinionated example beets configuration. This file was automatically copied to +# /config/beets/config.yaml. Feel free to edit this file to customize the beets +# configuration. For more information on the beets configuration, see +# https://beets.readthedocs.io/en/stable/reference/config.html + +plugins: [ + info, + the, + fetchart, + embedart, + ftintitle, + lastgenre, + missing, + albumtypes, + scrub, + zero, + mbsync, + duplicates, + convert, + fromfilename, + inline, + edit, + spotify, # needs authentication https://docs.beets.io/en/latest/plugins/spotify.html + musicbrainz, # needs to be enabled explicitly since beets 2.4.0 + ] + +directory: /music/imported +# library: /config/beets/library.db # default location in the container + +import: + move: no + copy: yes + write: yes + log: /music/last_beets_imports.log + quiet_fallback: skip + detail: yes + duplicate_action: ask # ask|skip|merge|keep|remove + +ui: + color: yes + +# fix up output file paths +replace: + '[\\]': "" + "[_]": "-" + "[/]": "-" + '^\.+': "" + '[\x00-\x1f]': "" + '[<>:"\?\*\|]': "" + '\.$': "" + '\s+$': "" + '^\s+': "" + "^-": "" + "’": "" + "′": "" + "″": "" + "‐": "-" + +per_disc_numbering: no +asciify_paths: yes + +# adjusting the `threaded` setting of beets should currently have no effect. +# we launch our own workers for previews and only have one +# import worker that runs one import (file moving) at a time. +threaded: no + +fetchart: + minwidth: 500 + enforce_ratio: 10px # yes|no or tolerance around 1:1 ratio + sources: coverart filesystem itunes amazon spotify albumart fanarttv + +embedart: + auto: yes + ifempty: yes # whether to avoid embedding album art for files that already have art embedded. + remove_art_file: yes + +ftintitle: + auto: yes + format: (feat. {0}) + +lastgenre: # get genres from last fm + auto: yes + count: 4 + prefer_specific: yes # Sort genres by the most to least specific, rather than most to least popular. + force: yes # By default, beets will always fetch new genres, even if the files already have one + source: track # album|track + separator: "; " + fallback: "" + +match: + # autotagger tolerance [0, 1], default 0.04 + # for example, 0.1 means 90% similarity required. + # inbox-folder autotag setting 'auto' respects this and imports above the threshold (like the beets cli would) + strong_rec_thresh: 0.1 + + # customize how penalties affect the match score + distance_weights: + # source: 2.0 + # artist: 3.0 + # album: 3.0 + # media: 1.0 + # mediums: 1.0 + # year: 1.0 + # country: 0.5 + # label: 0.5 + # catalognum: 0.5 + # albumdisambig: 0.5 + # album_id: 5.0 + # tracks: 2.0 + data_source: 0.0 # do not apply penalty to any data source plugin + missing_tracks: 0.2 # If your prefer not being so picky about missing tracks. default 0.9 + # unmatched_tracks: 0.6 + # track_title: 3.0 + # track_artist: 2.0 + # track_index: 1.0 + # track_length: 2.0 + # track_id: 5.0 + +musicbrainz: + external_ids: + discogs: yes + bandcamp: yes + spotify: yes + deezer: yes + beatport: yes + tidal: yes diff --git a/backend/beets_flask/config/config_bf_default.yaml b/backend/beets_flask/config/config_bf_default.yaml index aae84399..c64ad687 100644 --- a/backend/beets_flask/config/config_bf_default.yaml +++ b/backend/beets_flask/config/config_bf_default.yaml @@ -1,22 +1,22 @@ -# ------------------------------------------------------------------------------------ # -# DO NOT EDIT THIS FILE # -# ------------------------------------------------------------------------------------ # -# these are the defaults for the gui. -# you must provide your own config and map it to /config/beets-flask/config.yaml -# to get started, see the auto-generated examples in /config/beets-flask/ - -gui: - num_preview_workers: 4 - - library: - readonly: no - artist_separators: [",", ";", "&"] - - terminal: - start_path: "/repo" - - inbox: - ignore: "_use_beets_ignore" - debounce_before_autotag: 30 - concat_nested_folders: yes - expand_files: no +# ------------------------------------------------------------------------------------ # +# DO NOT EDIT THIS FILE # +# ------------------------------------------------------------------------------------ # +# these are the defaults for the gui. +# you must provide your own config and map it to /config/beets-flask/config.yaml +# to get started, see the auto-generated examples in /config/beets-flask/ + +gui: + num_preview_workers: 4 + + library: + readonly: no + artist_separators: [",", ";", "&"] + + terminal: + start_path: "/repo" + + inbox: + ignore: "_use_beets_ignore" + debounce_before_autotag: 30 + concat_nested_folders: yes + expand_files: no diff --git a/backend/beets_flask/config/config_bf_example.yaml b/backend/beets_flask/config/config_bf_example.yaml index be92d1ac..5e3bb3bd 100644 --- a/backend/beets_flask/config/config_bf_example.yaml +++ b/backend/beets_flask/config/config_bf_example.yaml @@ -1,45 +1,45 @@ -# ------------------------------------------------------------------------------------ # -# BEETS GUI CONFIG # -# ------------------------------------------------------------------------------------ # -# Example file, this file was automatically copied to /config/beets-flask/config.yaml. -# Feel free to edit this file to customize the gui configuration. -# Especially the `folders` section in the `inbox` section are important to set up your inbox -# folders. You can add as many folders as you like, but don't forget to volume-map them in -# your docker-compose.yml. - -gui: - num_preview_workers: 4 # how many previews to generate in parallel - - library: - # Use to split artists in the library view if using multiple artists in a field. - # Set to an empty list to disable this feature. - artist_separators: [",", ";", "&"] - - terminal: - start_path: "/music/inbox" # the directory where to start new terminal sessions - - inbox: - folders: - # --------------------------------- README -------------------------------- # - # Before using the inbox feature, you need to create the folders - # and decide on an inbox type. Have a look at the examples below. - - Inbox1: - name: "Dummy inbox" - path: "/music/dummy" - autotag: no - # do not automatically trigger tagging and do not automatically import - Inbox2: - name: "Auto Inbox" - path: "/music/inbox_auto" - autotag: "auto" - # trigger tag and import if a good match is found based on `auto_threshold` - auto_threshold: null - # if set to null, uses the value in beets config (match.strong_rec_thresh) - # define the distance from a perfect match, i.e. set to 0.1 to import - # matches with 90% similarity or better. - Inbox3: - name: "An Inbox that only generates the previews" - path: "/music/inbox_preview" - autotag: "preview" - # trigger tag but do not import, recommended for most control +# ------------------------------------------------------------------------------------ # +# BEETS GUI CONFIG # +# ------------------------------------------------------------------------------------ # +# Example file, this file was automatically copied to /config/beets-flask/config.yaml. +# Feel free to edit this file to customize the gui configuration. +# Especially the `folders` section in the `inbox` section are important to set up your inbox +# folders. You can add as many folders as you like, but don't forget to volume-map them in +# your docker-compose.yml. + +gui: + num_preview_workers: 4 # how many previews to generate in parallel + + library: + # Use to split artists in the library view if using multiple artists in a field. + # Set to an empty list to disable this feature. + artist_separators: [",", ";", "&"] + + terminal: + start_path: "/music/inbox" # the directory where to start new terminal sessions + + inbox: + folders: + # --------------------------------- README -------------------------------- # + # Before using the inbox feature, you need to create the folders + # and decide on an inbox type. Have a look at the examples below. + + Inbox1: + name: "Dummy inbox" + path: "/music/dummy" + autotag: no + # do not automatically trigger tagging and do not automatically import + Inbox2: + name: "Auto Inbox" + path: "/music/inbox_auto" + autotag: "auto" + # trigger tag and import if a good match is found based on `auto_threshold` + auto_threshold: null + # if set to null, uses the value in beets config (match.strong_rec_thresh) + # define the distance from a perfect match, i.e. set to 0.1 to import + # matches with 90% similarity or better. + Inbox3: + name: "An Inbox that only generates the previews" + path: "/music/inbox_preview" + autotag: "preview" + # trigger tag but do not import, recommended for most control diff --git a/backend/beets_flask/config/flask_config.py b/backend/beets_flask/config/flask_config.py index 4895525a..0a7f761c 100644 --- a/backend/beets_flask/config/flask_config.py +++ b/backend/beets_flask/config/flask_config.py @@ -1,122 +1,122 @@ -"""Server configuration for flask app. - -We have different configurations classes for -different environments further you may create -a custom configuration class for your own needs. - -The configuration classes are parse in the `create_app` -function in the `__init__.py` file. -""" - -from __future__ import annotations - -import os -from collections.abc import Mapping - -from ..logger import log - -cwd = os.getcwd() - - -class ServerConfig: - DEBUG = False - TESTING = False - - # If the database should be reset on start - # Useful for development - RESET_DB_ON_START = False - - # If errors should be thrown or - # caught and logged - # Enable for production! - PROPAGATE_EXCEPTIONS = False - - # Database URI - DATABASE_URI = "sqlite:///beets-flask-sqlite.db" - - # Folder where the frontend build resources are stored - # FIXME: 2025-06-04 likely we only need this in production. - FRONTEND_DIST_DIR = "../frontend/dist/" - - # Not sure if this is even used! - SECRET_KEY = "secret" - - def as_dict(self) -> dict: - return { - "DEBUG": ServerConfig.DEBUG, - "TESTING": ServerConfig.TESTING, - "RESET_DB_ON_START": ServerConfig.RESET_DB_ON_START, - "PROPAGATE_EXCEPTIONS": ServerConfig.PROPAGATE_EXCEPTIONS, - "DATABASE_URI": ServerConfig.DATABASE_URI, - "SECRET_KEY": ServerConfig.SECRET_KEY, - } - - def __getitem__(self, key): - return getattr(self, key) - - -class Testing(ServerConfig): - TESTING = True - DATABASE_URI = "sqlite:///:memory:?cache=shared" - # temporary in-memory database - # DATABASE_URI = "sqlite://" - - -class DevelopmentLocal(ServerConfig): - RESET_DB_ON_START = True - DEBUG = True - DATABASE_URI = f"sqlite:///{cwd}/beets-flask-sqlite.db" - - -class DevelopmentDocker(ServerConfig): - DATABASE_URI = ( - f"sqlite:////{os.getenv('BEETSFLASKDIR')}/beets-flask-sqlite.db?timeout=5" - ) - DEBUG = True - - -# production -class DeploymentDocker(DevelopmentDocker): - DEBUG = False - TESTING = False - PROPAGATE_EXCEPTIONS = True - FRONTEND_DIST_DIR = "/repo/frontend/dist/" - - -def init_server_config(input_config: str | ServerConfig | None = None) -> ServerConfig: - global config - - if isinstance(input_config, ServerConfig): - config = input_config - else: - if input_config is None: - input_config = os.environ.get("IB_SERVER_CONFIG", "dev_local") - switch: Mapping[str, type[ServerConfig]] = { - "dev_local": DevelopmentLocal, - "dev_docker": DevelopmentDocker, - "test": Testing, - "prod": DeploymentDocker, - } - if isinstance(input_config, str) and input_config not in switch: - raise ValueError(f"Invalid config: {config}") - log.debug(f"Using config: {input_config}") - # we still have to initalize! - config = switch[input_config]() - - return config - - -# if "RQ_WORKER_ID" in os.environ: -# not elegant, but we also need to initalize the config in workers, -# where the app init is not called -# and for some reason it is needed in global space. revisit this in quartz port - - -config: ServerConfig | None = None - - -def get_flask_config() -> ServerConfig: - global config - if not config: - config = init_server_config() - return config +"""Server configuration for flask app. + +We have different configurations classes for +different environments further you may create +a custom configuration class for your own needs. + +The configuration classes are parse in the `create_app` +function in the `__init__.py` file. +""" + +from __future__ import annotations + +import os +from collections.abc import Mapping + +from ..logger import log + +cwd = os.getcwd() + + +class ServerConfig: + DEBUG = False + TESTING = False + + # If the database should be reset on start + # Useful for development + RESET_DB_ON_START = False + + # If errors should be thrown or + # caught and logged + # Enable for production! + PROPAGATE_EXCEPTIONS = False + + # Database URI + DATABASE_URI = "sqlite:///beets-flask-sqlite.db" + + # Folder where the frontend build resources are stored + # FIXME: 2025-06-04 likely we only need this in production. + FRONTEND_DIST_DIR = "../frontend/dist/" + + # Not sure if this is even used! + SECRET_KEY = "secret" + + def as_dict(self) -> dict: + return { + "DEBUG": ServerConfig.DEBUG, + "TESTING": ServerConfig.TESTING, + "RESET_DB_ON_START": ServerConfig.RESET_DB_ON_START, + "PROPAGATE_EXCEPTIONS": ServerConfig.PROPAGATE_EXCEPTIONS, + "DATABASE_URI": ServerConfig.DATABASE_URI, + "SECRET_KEY": ServerConfig.SECRET_KEY, + } + + def __getitem__(self, key): + return getattr(self, key) + + +class Testing(ServerConfig): + TESTING = True + DATABASE_URI = "sqlite:///:memory:?cache=shared" + # temporary in-memory database + # DATABASE_URI = "sqlite://" + + +class DevelopmentLocal(ServerConfig): + RESET_DB_ON_START = True + DEBUG = True + DATABASE_URI = f"sqlite:///{cwd}/beets-flask-sqlite.db" + + +class DevelopmentDocker(ServerConfig): + DATABASE_URI = ( + f"sqlite:////{os.getenv('BEETSFLASKDIR')}/beets-flask-sqlite.db?timeout=5" + ) + DEBUG = True + + +# production +class DeploymentDocker(DevelopmentDocker): + DEBUG = False + TESTING = False + PROPAGATE_EXCEPTIONS = True + FRONTEND_DIST_DIR = "/repo/frontend/dist/" + + +def init_server_config(input_config: str | ServerConfig | None = None) -> ServerConfig: + global config + + if isinstance(input_config, ServerConfig): + config = input_config + else: + if input_config is None: + input_config = os.environ.get("IB_SERVER_CONFIG", "dev_local") + switch: Mapping[str, type[ServerConfig]] = { + "dev_local": DevelopmentLocal, + "dev_docker": DevelopmentDocker, + "test": Testing, + "prod": DeploymentDocker, + } + if isinstance(input_config, str) and input_config not in switch: + raise ValueError(f"Invalid config: {config}") + log.debug(f"Using config: {input_config}") + # we still have to initalize! + config = switch[input_config]() + + return config + + +# if "RQ_WORKER_ID" in os.environ: +# not elegant, but we also need to initalize the config in workers, +# where the app init is not called +# and for some reason it is needed in global space. revisit this in quartz port + + +config: ServerConfig | None = None + + +def get_flask_config() -> ServerConfig: + global config + if not config: + config = init_server_config() + return config diff --git a/backend/beets_flask/database/__init__.py b/backend/beets_flask/database/__init__.py index 78d3b6a7..608a6175 100644 --- a/backend/beets_flask/database/__init__.py +++ b/backend/beets_flask/database/__init__.py @@ -1,7 +1,7 @@ -from .setup import db_session_factory, setup_database, with_db_session - -__all__ = [ - "setup_database", - "db_session_factory", - "with_db_session", -] +from .setup import db_session_factory, setup_database, with_db_session + +__all__ = [ + "setup_database", + "db_session_factory", + "with_db_session", +] diff --git a/backend/beets_flask/database/models/__init__.py b/backend/beets_flask/database/models/__init__.py index 0e0f4a34..c37eba94 100644 --- a/backend/beets_flask/database/models/__init__.py +++ b/backend/beets_flask/database/models/__init__.py @@ -1,10 +1,12 @@ -from .base import Base -from .states import CandidateStateInDb, FolderInDb, SessionStateInDb, TaskStateInDb - -__all__ = [ - "Base", - "FolderInDb", - "SessionStateInDb", - "TaskStateInDb", - "CandidateStateInDb", -] +from .base import Base +from .states import CandidateStateInDb, FolderInDb, SessionStateInDb, TaskStateInDb +from .stats import CachedStatInDb + +__all__ = [ + "Base", + "FolderInDb", + "SessionStateInDb", + "TaskStateInDb", + "CandidateStateInDb", + "CachedStatInDb", +] diff --git a/backend/beets_flask/database/models/base.py b/backend/beets_flask/database/models/base.py index 02ab64fb..446a1366 100644 --- a/backend/beets_flask/database/models/base.py +++ b/backend/beets_flask/database/models/base.py @@ -1,120 +1,120 @@ -from __future__ import annotations - -from collections.abc import Mapping, Sequence -from datetime import datetime -from typing import Any, Self -from uuid import uuid4 - -import pytz -from sqlalchemy import LargeBinary, select -from sqlalchemy.orm import ( - DeclarativeBase, - Mapped, - Session, - mapped_column, - reconstructor, - registry, -) -from sqlalchemy.sql import func - -from beets_flask.logger import log - -from .types import DictType, IntDictType, StrDictType - - -class Base(DeclarativeBase): - __abstract__ = True - - registry = registry( - type_annotation_map={ - bytes: LargeBinary, - dict[int, int]: IntDictType, - dict[str, str]: StrDictType, - dict[str, Any]: DictType, - } - ) - - id: Mapped[str] = mapped_column(primary_key=True) - - created_at: Mapped[datetime] = mapped_column(default=func.now(), index=True) - updated_at: Mapped[datetime] = mapped_column( - default=func.now(), onupdate=func.now() - ) - - def __init__(self, id: str | None = None): - self.id = str(id) if id is not None else str(uuid4()) - - @classmethod - def get_by(cls, *whereclause, session: Session | None = None) -> Self | None: - close_after = False - if session is None: - log.debug( - "No session provided, you will not be able to make changes to the database." - ) - close_after = True - from beets_flask.database.setup import session_factory - - session = session_factory() - - try: - stmt = select(cls).where(*whereclause) - item = session.execute(stmt).scalars().first() - return item - except: - raise - finally: - if close_after: - session.close() - - @classmethod - def get_all_by(cls, *whereclause, session: Session | None = None) -> Sequence[Self]: - close_after = False - if session is None: - log.debug( - "No session provided, you will not be able to make changes to the database." - ) - close_after = True - from beets_flask.database.setup import session_factory - - session = session_factory() - - try: - stmt = select(cls).where(*whereclause) - item = session.execute(stmt).scalars().all() - return item - except: - raise - finally: - if close_after: - session.close() - - @classmethod - def get_by_raise(cls, *whereclause, session: Session | None = None) -> Self: - """Get an item by whereclause or raise ValueError if not found.""" - item = cls.get_by(*whereclause, session=session) - if item is None: - raise ValueError(f"{cls.__name__} not found.") - return item - - @classmethod - def exist_all_ids(cls, ids: list[str], session: Session) -> bool: - """Check if an item exists for every given id.""" - _ids = set(ids) - stmt = select(func.count()).select_from(cls).where(cls.id.in_(_ids)) - count = session.execute(stmt).scalar() - if count is None or count != len(_ids): - return False - return True - - def to_dict(self) -> Mapping: - return {c.name: getattr(self, c.name) for c in self.__table__.columns} - - @reconstructor - def _sqlalchemy_reconstructor(self): - # Set timezone info for created_at and updated_at - # Seems a bit hacky but is the only way to ensure that - # datetime objects are timezone-aware after deserialization - if self.created_at and self.created_at.tzinfo is None: - self.created_at = self.created_at.replace(tzinfo=pytz.UTC) - if self.updated_at and self.updated_at.tzinfo is None: - self.updated_at = self.updated_at.replace(tzinfo=pytz.UTC) +from __future__ import annotations + +from collections.abc import Mapping, Sequence +from datetime import datetime +from typing import Any, Self +from uuid import uuid4 + +import pytz +from sqlalchemy import LargeBinary, select +from sqlalchemy.orm import ( + DeclarativeBase, + Mapped, + Session, + mapped_column, + reconstructor, + registry, +) +from sqlalchemy.sql import func + +from beets_flask.logger import log + +from .types import DictType, IntDictType, StrDictType + + +class Base(DeclarativeBase): + __abstract__ = True + + registry = registry( + type_annotation_map={ + bytes: LargeBinary, + dict[int, int]: IntDictType, + dict[str, str]: StrDictType, + dict[str, Any]: DictType, + } + ) + + id: Mapped[str] = mapped_column(primary_key=True) + + created_at: Mapped[datetime] = mapped_column(default=func.now(), index=True) + updated_at: Mapped[datetime] = mapped_column( + default=func.now(), onupdate=func.now() + ) + + def __init__(self, id: str | None = None): + self.id = str(id) if id is not None else str(uuid4()) + + @classmethod + def get_by(cls, *whereclause, session: Session | None = None) -> Self | None: + close_after = False + if session is None: + log.debug( + "No session provided, you will not be able to make changes to the database." + ) + close_after = True + from beets_flask.database.setup import session_factory + + session = session_factory() + + try: + stmt = select(cls).where(*whereclause) + item = session.execute(stmt).scalars().first() + return item + except: + raise + finally: + if close_after: + session.close() + + @classmethod + def get_all_by(cls, *whereclause, session: Session | None = None) -> Sequence[Self]: + close_after = False + if session is None: + log.debug( + "No session provided, you will not be able to make changes to the database." + ) + close_after = True + from beets_flask.database.setup import session_factory + + session = session_factory() + + try: + stmt = select(cls).where(*whereclause) + item = session.execute(stmt).scalars().all() + return item + except: + raise + finally: + if close_after: + session.close() + + @classmethod + def get_by_raise(cls, *whereclause, session: Session | None = None) -> Self: + """Get an item by whereclause or raise ValueError if not found.""" + item = cls.get_by(*whereclause, session=session) + if item is None: + raise ValueError(f"{cls.__name__} not found.") + return item + + @classmethod + def exist_all_ids(cls, ids: list[str], session: Session) -> bool: + """Check if an item exists for every given id.""" + _ids = set(ids) + stmt = select(func.count()).select_from(cls).where(cls.id.in_(_ids)) + count = session.execute(stmt).scalar() + if count is None or count != len(_ids): + return False + return True + + def to_dict(self) -> Mapping: + return {c.name: getattr(self, c.name) for c in self.__table__.columns} + + @reconstructor + def _sqlalchemy_reconstructor(self): + # Set timezone info for created_at and updated_at + # Seems a bit hacky but is the only way to ensure that + # datetime objects are timezone-aware after deserialization + if self.created_at and self.created_at.tzinfo is None: + self.created_at = self.created_at.replace(tzinfo=pytz.UTC) + if self.updated_at and self.updated_at.tzinfo is None: + self.updated_at = self.updated_at.replace(tzinfo=pytz.UTC) diff --git a/backend/beets_flask/database/models/states.py b/backend/beets_flask/database/models/states.py index a4798d07..eb51700b 100644 --- a/backend/beets_flask/database/models/states.py +++ b/backend/beets_flask/database/models/states.py @@ -1,587 +1,587 @@ -"""Minimal state model for the beets_flask application. - -Allows to resume a import at any time using our state dataclasses, -see importer/state.py for more information. - -Why not just have State and StateInDb in the same class? -- ORM ideally wants full mirroring of whats in RAM in the DB. This is hard to ensure - in our case, as we dont have full control over beets tasks etc. -- A lot of beets objects do not neatly translate to DB objects. -- Often we want states without having to think about a DB Session. -- Just a current motivation and choice, will revisit this later. -""" - -from __future__ import annotations - -import io -import pickle -from pathlib import Path -from typing import Any - -from beets.autotag import AlbumMatch -from beets.autotag.distance import Distance -from beets.importer import Action, ImportTask -from beets.library.models import Item as LibraryItem -from sqlalchemy import ( - ForeignKey, - UniqueConstraint, - select, -) -from sqlalchemy.orm import ( - Mapped, - Session, - mapped_column, - relationship, -) - -from beets_flask.database.models.base import Base -from beets_flask.disk import Archive, Folder -from beets_flask.importer.progress import Progress -from beets_flask.importer.states import ( - CandidateState, - SerializedCandidateState, - SerializedSessionState, - SerializedTaskState, - SessionState, - TaskState, -) -from beets_flask.importer.types import BeetsAlbumMatch, BeetsTrackMatch -from beets_flask.logger import log -from beets_flask.server.exceptions import SerializedException - - -class FolderInDb(Base): - """Represents a folder on disk, to keep track of changes. - - This folder does not necessarily have to exist on disk anymore. If the content - changed, a new folder object (new hash) should be created. - """ - - __tablename__ = "folder" - - # Composite primary key - full_path: Mapped[str] = mapped_column(index=True, primary_key=True) - - # checked -> yes | no or didnt check -> None - is_album: Mapped[bool | None] - - def __init__(self, path: Path | str, hash: str, is_album: bool | None = None): - """ - Create a FolderInDb object from a path. - - Convention: - /home/user/foo/ - abs path with trailing slash. - - Parameters - ---------- - path : Path - The path to create the object from. - """ - if isinstance(path, str): - path = Path(path) - self.full_path = str(path.resolve()) - self.hash = hash - self.is_album = is_album - - @classmethod - def from_live_folder(cls, folder: Folder | Archive) -> FolderInDb: - """Create a FolderInDb object from a Folder object.""" - f_in_db = cls( - path=folder.path, - hash=folder.hash, - ) - f_in_db.is_album = folder.is_album - - return f_in_db - - def to_live_folder(self) -> Folder: - """Recreate the live Folder object from its stored version in the db.""" - return Folder( - children=[], - full_path=self.full_path, - hash=self.hash, - is_album=self.is_album or False, - ) - - def as_tuple(self) -> tuple[Path, str]: - """Recreate the live Folder object from its stored version in the db.""" - return ( - self.path, - self.hash, - ) - - @property - def hash(self) -> str: - """ - Convenience property to get the id. - - Note: Although the id is just the hash, when querying the db, you **must** use `FolderInDb.id == hash`. Sqlalchemy does not resolve properties. - """ - return self.id - - @hash.setter - def hash(self, value: str): - self.id = value - - @property - def path(self) -> Path: - return Path(self.full_path) - - @classmethod - def get_current_on_disk(cls, hash: str, path: Path | str) -> Folder | Archive: - """ - Check that a folders hash is still the same, as you have previously determined. - - If changed, a new instance of FolderInDb is created and stored in the DB. - - Returns - ------- - Folder: The live folder object on disk, with the potentially new (current) hash. - """ - from beets_flask.database.setup import db_session_factory - - with db_session_factory() as db_session: - if isinstance(path, str): - path = Path(path) - # Check if archive - f_on_disk: Folder | Archive - if path.is_dir(): - f_on_disk = Folder.from_path(path) - else: - f_on_disk = Archive.from_path(path) - - f_in_db = FolderInDb.get_by(FolderInDb.id == hash, session=db_session) - if f_in_db is None: - f_in_db = FolderInDb.from_live_folder(f_on_disk) - db_session.merge(f_in_db) - db_session.commit() - - if f_in_db.hash != f_on_disk.hash: - log.debug( - f"Hash mismatch {path=} {f_in_db.hash=} {f_on_disk.hash=}" - + "This indicatest that the folder has changed." - ) - return f_on_disk - - -class SessionStateInDb(Base): - """Represents an import session. - - Normally a session has one task but in theory and edge cases - we could have multiple tasks per session. - - Beets uses sessions for the back-and-forth dialog with the user, - where one session may have multiple tasks. - We wrap the beets session in our SessionState to better handle its progress. - And our SessionState has a representation in our database, the SessionStateInDb. - - Example: - ``` - # Create - s_live_state = SessionState(Path("path")) - session = PreviewSession(s_live_state) - s_live_state = session.run_sync() - s_db_state = SessionStateInDb.from_live_state(s_live_state) - - # Search - select(SessionStateInDb).where(TaskStateInDb.id == "some path").first() - s_db_state = SessionStateInDb.get_by( - ``` - """ - - __tablename__ = "session" - - tasks: Mapped[list[TaskStateInDb]] = relationship( - back_populates="session", - # all: All operations cascade i.e. session.merge! - # delete-orphan: Automatic deletion of tasks if not referenced - # by a session anymore - # See also https://docs.sqlalchemy.org/en/20/orm/cascades.html#unitofwork-cascades - cascade="all, delete-orphan", - ) - - folder: Mapped[FolderInDb] = relationship() - folder_hash: Mapped[str] = mapped_column(ForeignKey("folder.id")) - folder_revision: Mapped[int] = mapped_column(default=0) - __table_args__ = ( - UniqueConstraint( - "folder_hash", "folder_revision", name="uq_folder_hash_revision" - ), - ) - # We have folder revisions to allow multiple sessions for the same folder hash, - # the purpose being that we want to keep old sessions around. E.g. to not loose - # old data when regenerating previews. - # but at the same time, we want a soft 1:1 mapping between folder hash and session. - # Thus, revisions are needed: the session-hash link always uses the highest revision. - - # FIXME: This should be a getter for the which queries the tasks - progress: Mapped[Progress] - - # If an session run fails we want to store the exception - exc: Mapped[bytes | None] - - def __init__( - self, - folder: FolderInDb, - id: str | None = None, - tasks: list[TaskStateInDb] = [], - progress: Progress = Progress.NOT_STARTED, - exc: SerializedException | None = None, - ): - super().__init__(id) - self.folder = folder - self.tasks = tasks - self.progress = progress - self.exc = pickle.dumps(exc) if exc else None - - @classmethod - def from_live_state(cls, state: SessionState) -> SessionStateInDb: - """Create the DB representation of a live SessionState..""" - - session = cls( - folder=FolderInDb(state.folder_path, state.folder_hash), - id=state.id, - tasks=[TaskStateInDb.from_live_state(ts) for ts in state.task_states], - progress=state.progress.progress, - exc=state.exc, - ) - - return session - - @property - def folder_path(self) -> Path: - return self.folder.path - - def to_live_state(self, new_folder=True) -> SessionState: - """Recreate the live SessionState with underlying task from its stored version in the db. - - HACK: new_folder param is a bit hacky, as if we do not include the children if we - are not recomputing the folder hash. Might lead to some issues down the line. - """ - - if new_folder: - s_state = SessionState(self.folder.path) - else: - s_state = SessionState(self.folder.to_live_folder()) - - if s_state.folder_hash != self.folder.hash: - log.warning( - f"Folder hash mismatch for {self.folder.path}. " - f"Expected {self.folder.hash} but got {s_state.folder_hash}." - ) - s_state.id = self.id - s_state.created_at = self.created_at - s_state.updated_at = self.updated_at - s_state._task_states = [task.to_live_state(s_state) for task in self.tasks] - s_state.exc = pickle.loads(self.exc) if self.exc else None - return s_state - - def to_dict(self) -> SerializedSessionState: - return self.to_live_state(False).serialize() - - @classmethod - def get_by_hash_and_path( - cls, - hash: str | None, - path: Path | str | None, - db_session: Session | None = None, - ) -> SessionStateInDb | None: - """ - Get a session by its hash and if this fails, try its path. - - If multiple matches, returns the most recent one. - """ - from beets_flask.database import db_session_factory - - with db_session_factory(db_session) as db_session: - item = None - if hash is not None: - query = ( - select(cls) - .where(cls.folder_hash == hash) - # hash+revision combos have unique constraints - # and sessions always point to the latest / highest revision. - .order_by(cls.folder_revision.desc()) - ) - item = db_session.execute(query).scalars().first() - if item is None and path is not None: - # Try to get by path - # paths do not have revisions, always use last updated session - query = ( - select(cls) - .join(cls.folder) - .where(FolderInDb.full_path == str(path)) - .order_by(cls.updated_at.desc(), cls.folder_revision.desc()) - ) - item = db_session.execute(query).scalars().first() - - return item - - @property - def exception(self) -> SerializedException | None: - """Returns the exception of the session if it failed.""" - return pickle.loads(self.exc) if self.exc else None - - -class TaskStateInDb(Base): - """Represents an import task. - - More precisely, beets uses one task per album that goes through a bunch of stages. - We wrap the beets task in our TaskState to better handle its progress. - And this TaskState has a representation in our database, the TaskStateInDb. - """ - - __tablename__ = "task" - - # Relationships - session_id: Mapped[str] = mapped_column(ForeignKey("session.id")) - session: Mapped[SessionStateInDb] = relationship( - back_populates="tasks", - foreign_keys=[session_id], - ) - - candidates: Mapped[list[CandidateStateInDb]] = relationship( - back_populates="task", - foreign_keys="[CandidateStateInDb.task_id]", - cascade="all, delete-orphan", - ) - # Set at the end of the import session - chosen_candidate_id: Mapped[str | None] = mapped_column(ForeignKey("candidate.id")) - chosen_candidate: Mapped[CandidateStateInDb | None] = relationship( - back_populates="task", - foreign_keys=[chosen_candidate_id], - ) - - toppath: Mapped[bytes | None] - - # To reconstruct the beets task we need to store a few of its attributes - paths: Mapped[bytes] - old_paths: Mapped[bytes | None] - # old_paths contain original file paths, but are only set when files are moved. - # (which breaks some deep links that before were identical to paths, but no more!) - items: Mapped[bytes] - choice_flag: Mapped[Action | None] - - # To allow for continue we need to store the current artist and album - # TODO: REMOVE this is not needed!! We can look at the asis candidate for this! - # E.g. frontend component to compare two candidates - cur_artist: Mapped[str | None] - cur_album: Mapped[str | None] - - progress: Mapped[Progress] - - def __init__( - self, - id: str | None = None, - toppath: bytes | None = None, - paths: list[bytes] = [], - old_paths: list[bytes] | None = None, - items: list[LibraryItem] = [], - candidates: list[CandidateStateInDb] = [], - chosen_candidate_id: str | None = None, - progress: Progress = Progress.NOT_STARTED, - choice_flag: Action | None = None, - cur_artist: str | None = None, - cur_album: str | None = None, - ): - super().__init__(id) - self.toppath = toppath - self.paths = pickle.dumps(paths) - self.old_paths = pickle.dumps(old_paths) if old_paths else None - - for item in items: - # Remove db from all items as it can't be pickled - item._db = None - item._Item__album = None - - self.items = pickle.dumps(items) - self.candidates = candidates - self.chosen_candidate_id = chosen_candidate_id - self.progress = progress - self.choice_flag = choice_flag - self.cur_artist = cur_artist - self.cur_album = cur_album - - @classmethod - def from_live_state(cls, state: TaskState) -> TaskStateInDb: - """Create the DB representation of a live TaskState.""" - if hasattr(state.task, "old_paths"): - old_paths = state.task.old_paths - else: - old_paths = None - - task = cls( - id=state.id, - toppath=str(state.toppath).encode("utf-8") if state.toppath else None, - paths=state.task.paths, - items=state.task.items, - candidates=[ - CandidateStateInDb.from_live_state(c) for c in state.candidate_states - ], - chosen_candidate_id=state.chosen_candidate_state_id, - progress=state.progress.progress, - choice_flag=state.task.choice_flag, - cur_artist=state.task.cur_artist, - cur_album=state.task.cur_album, - old_paths=old_paths, - ) - return task - - def to_live_state(self, session_state: SessionState | None = None) -> TaskState: - """Recreate the live TaskState with underlying task from its stored version in the db.""" - - # We just assume it is a normal import task - beets_task = ImportTask( - toppath=self.toppath, - paths=pickle.loads(self.paths), - items=pickle.loads(self.items), - ) - beets_task.choice_flag = self.choice_flag - beets_task.cur_artist = self.cur_artist - beets_task.cur_album = self.cur_album - old_paths: list[bytes] | None = ( - pickle.loads(self.old_paths) if self.old_paths else None - ) - # TODO: Update type hints once beets is updated - beets_task.old_paths = old_paths # type: ignore - - live_state = TaskState(beets_task) - live_state.id = self.id - live_state.created_at = self.created_at - live_state.updated_at = self.updated_at - live_state.candidate_states = [ - c.to_live_state(live_state) for c in self.candidates - ] - live_state.chosen_candidate_state_id = self.chosen_candidate_id - live_state.progress.progress = self.progress - - # Set candidate of beets_task - live_state.task.candidates = [c.match for c in live_state.candidate_states] - - return live_state - - def to_dict(self) -> SerializedTaskState: - return self.to_live_state().serialize() - - -class CandidateStateInDb(Base): - """Represents a candidate (potential match) for an import task. - - Again: Beets-Candidate > CandidateState > CandidateStateInDb - """ - - __tablename__ = "candidate" - - task_id: Mapped[str] = mapped_column(ForeignKey("task.id")) - task: Mapped[TaskStateInDb] = relationship( - back_populates="candidates", - foreign_keys=[task_id], - ) - - # Should deserialize to AlbumMatch|TrackMatch - # ~4kb per match - match: Mapped[bytes] - - # Duplicate ids (if any) (beets_id) - duplicate_ids: Mapped[str] - - # association between tracks online and items on disk, from int to int - mapping: Mapped[dict[int, int]] - - def __init__( - self, - match: BeetsAlbumMatch | BeetsTrackMatch, - mapping: dict[int, int], - duplicate_ids: list[str] = [], - id: str | None = None, - ): - super().__init__(id) - - # Remove db from all items as it can't be pickled - # FIXME: this should go into beets __getstate__ method - # see https://github.com/beetbox/beets/pull/5641 - if isinstance(match, BeetsAlbumMatch): - for item in match.mapping.keys(): - item._db = None - item._Item__album = None - for item in match.extra_items: - item._db = None - item._Item__album = None - - self.match = pickle.dumps(match) - self.duplicate_ids = ";".join(map(str, duplicate_ids)) - self.mapping = mapping - - @classmethod - def from_live_state(cls, state: CandidateState) -> CandidateStateInDb: - """Create the DB representation of a live CandidateState.""" - return cls( - id=state.id, - match=state.match, - duplicate_ids=state.duplicate_ids, - mapping=state._mapping, - ) - - def to_live_state(self, task_state: TaskState | None) -> CandidateState: - """Recreate the live CandidateState with underlying task from its stored version in the db.""" - if task_state is None: - task_state = self.task.to_live_state() - live_state = CandidateState( - CustomUnpickler(io.BytesIO(self.match)).load(), - task_state, - mapping=self.mapping, - ) - live_state.id = self.id - live_state.created_at = self.created_at - live_state.updated_at = self.updated_at - live_state.duplicate_ids = ( - # edge case: "".split() gives [''] - [] if len(self.duplicate_ids) == 0 else self.duplicate_ids.split(";") - ) - return live_state - - def to_dict(self) -> SerializedCandidateState: - return self.to_live_state(self.task.to_live_state()).serialize() - - -# Hotfix for match unpickler to resolve beets distance moved -# This is needed because various beets updates changed class implementations -# and we want to rebuild the newer versions of some beets classes from old pickles. -# TODO: We should fix this in general and not pickle beets objects -class CustomUnpickler(pickle.Unpickler): - def find_class(self, module, name): - """Override the find_class method to redirect Distance class references.""" - # Redirect Distance class from beets.autotag.hooks to beets.distance (2.4.0) - if module == "beets.autotag.hooks" and name == "Distance": - return Distance - - # For all other classes, use the default lookup mechanism - return super().find_class(module, name) - - def load(self) -> Any: - object = super().load() - if isinstance(object, Distance): - self.patch_distance(object) - - if isinstance(object, AlbumMatch): - self.patch_distance(object.distance) - - return object - - def patch_distance(self, distance: Distance) -> Distance: - # Rewrite "source" penalty to "data_source" penalty (2.5.0) - if "source" in distance._penalties: - log.debug( - "Converting old distance.source to distance.data_source (changed in beets 2.5.0)" - ) - distance._penalties["data_source"] = distance._penalties["source"] - del distance._penalties["source"] - - # Potential infinite recursion, ah well - for track, d in distance.tracks.items(): - self.patch_distance(d) - return distance - - -__all__ = ["SessionStateInDb", "TaskStateInDb", "CandidateStateInDb"] +"""Minimal state model for the beets_flask application. + +Allows to resume a import at any time using our state dataclasses, +see importer/state.py for more information. + +Why not just have State and StateInDb in the same class? +- ORM ideally wants full mirroring of whats in RAM in the DB. This is hard to ensure + in our case, as we dont have full control over beets tasks etc. +- A lot of beets objects do not neatly translate to DB objects. +- Often we want states without having to think about a DB Session. +- Just a current motivation and choice, will revisit this later. +""" + +from __future__ import annotations + +import io +import pickle +from pathlib import Path +from typing import Any + +from beets.autotag import AlbumMatch +from beets.autotag.distance import Distance +from beets.importer import Action, ImportTask +from beets.library.models import Item as LibraryItem +from sqlalchemy import ( + ForeignKey, + UniqueConstraint, + select, +) +from sqlalchemy.orm import ( + Mapped, + Session, + mapped_column, + relationship, +) + +from beets_flask.database.models.base import Base +from beets_flask.disk import Archive, Folder +from beets_flask.importer.progress import Progress +from beets_flask.importer.states import ( + CandidateState, + SerializedCandidateState, + SerializedSessionState, + SerializedTaskState, + SessionState, + TaskState, +) +from beets_flask.importer.types import BeetsAlbumMatch, BeetsTrackMatch +from beets_flask.logger import log +from beets_flask.server.exceptions import SerializedException + + +class FolderInDb(Base): + """Represents a folder on disk, to keep track of changes. + + This folder does not necessarily have to exist on disk anymore. If the content + changed, a new folder object (new hash) should be created. + """ + + __tablename__ = "folder" + + # Composite primary key + full_path: Mapped[str] = mapped_column(index=True, primary_key=True) + + # checked -> yes | no or didnt check -> None + is_album: Mapped[bool | None] + + def __init__(self, path: Path | str, hash: str, is_album: bool | None = None): + """ + Create a FolderInDb object from a path. + + Convention: + /home/user/foo/ + abs path with trailing slash. + + Parameters + ---------- + path : Path + The path to create the object from. + """ + if isinstance(path, str): + path = Path(path) + self.full_path = str(path.resolve()) + self.hash = hash + self.is_album = is_album + + @classmethod + def from_live_folder(cls, folder: Folder | Archive) -> FolderInDb: + """Create a FolderInDb object from a Folder object.""" + f_in_db = cls( + path=folder.path, + hash=folder.hash, + ) + f_in_db.is_album = folder.is_album + + return f_in_db + + def to_live_folder(self) -> Folder: + """Recreate the live Folder object from its stored version in the db.""" + return Folder( + children=[], + full_path=self.full_path, + hash=self.hash, + is_album=self.is_album or False, + ) + + def as_tuple(self) -> tuple[Path, str]: + """Recreate the live Folder object from its stored version in the db.""" + return ( + self.path, + self.hash, + ) + + @property + def hash(self) -> str: + """ + Convenience property to get the id. + + Note: Although the id is just the hash, when querying the db, you **must** use `FolderInDb.id == hash`. Sqlalchemy does not resolve properties. + """ + return self.id + + @hash.setter + def hash(self, value: str): + self.id = value + + @property + def path(self) -> Path: + return Path(self.full_path) + + @classmethod + def get_current_on_disk(cls, hash: str, path: Path | str) -> Folder | Archive: + """ + Check that a folders hash is still the same, as you have previously determined. + + If changed, a new instance of FolderInDb is created and stored in the DB. + + Returns + ------- + Folder: The live folder object on disk, with the potentially new (current) hash. + """ + from beets_flask.database.setup import db_session_factory + + with db_session_factory() as db_session: + if isinstance(path, str): + path = Path(path) + # Check if archive + f_on_disk: Folder | Archive + if path.is_dir(): + f_on_disk = Folder.from_path(path) + else: + f_on_disk = Archive.from_path(path) + + f_in_db = FolderInDb.get_by(FolderInDb.id == hash, session=db_session) + if f_in_db is None: + f_in_db = FolderInDb.from_live_folder(f_on_disk) + db_session.merge(f_in_db) + db_session.commit() + + if f_in_db.hash != f_on_disk.hash: + log.debug( + f"Hash mismatch {path=} {f_in_db.hash=} {f_on_disk.hash=}" + + "This indicatest that the folder has changed." + ) + return f_on_disk + + +class SessionStateInDb(Base): + """Represents an import session. + + Normally a session has one task but in theory and edge cases + we could have multiple tasks per session. + + Beets uses sessions for the back-and-forth dialog with the user, + where one session may have multiple tasks. + We wrap the beets session in our SessionState to better handle its progress. + And our SessionState has a representation in our database, the SessionStateInDb. + + Example: + ``` + # Create + s_live_state = SessionState(Path("path")) + session = PreviewSession(s_live_state) + s_live_state = session.run_sync() + s_db_state = SessionStateInDb.from_live_state(s_live_state) + + # Search + select(SessionStateInDb).where(TaskStateInDb.id == "some path").first() + s_db_state = SessionStateInDb.get_by( + ``` + """ + + __tablename__ = "session" + + tasks: Mapped[list[TaskStateInDb]] = relationship( + back_populates="session", + # all: All operations cascade i.e. session.merge! + # delete-orphan: Automatic deletion of tasks if not referenced + # by a session anymore + # See also https://docs.sqlalchemy.org/en/20/orm/cascades.html#unitofwork-cascades + cascade="all, delete-orphan", + ) + + folder: Mapped[FolderInDb] = relationship() + folder_hash: Mapped[str] = mapped_column(ForeignKey("folder.id")) + folder_revision: Mapped[int] = mapped_column(default=0) + __table_args__ = ( + UniqueConstraint( + "folder_hash", "folder_revision", name="uq_folder_hash_revision" + ), + ) + # We have folder revisions to allow multiple sessions for the same folder hash, + # the purpose being that we want to keep old sessions around. E.g. to not loose + # old data when regenerating previews. + # but at the same time, we want a soft 1:1 mapping between folder hash and session. + # Thus, revisions are needed: the session-hash link always uses the highest revision. + + # FIXME: This should be a getter for the which queries the tasks + progress: Mapped[Progress] + + # If an session run fails we want to store the exception + exc: Mapped[bytes | None] + + def __init__( + self, + folder: FolderInDb, + id: str | None = None, + tasks: list[TaskStateInDb] = [], + progress: Progress = Progress.NOT_STARTED, + exc: SerializedException | None = None, + ): + super().__init__(id) + self.folder = folder + self.tasks = tasks + self.progress = progress + self.exc = pickle.dumps(exc) if exc else None + + @classmethod + def from_live_state(cls, state: SessionState) -> SessionStateInDb: + """Create the DB representation of a live SessionState..""" + + session = cls( + folder=FolderInDb(state.folder_path, state.folder_hash), + id=state.id, + tasks=[TaskStateInDb.from_live_state(ts) for ts in state.task_states], + progress=state.progress.progress, + exc=state.exc, + ) + + return session + + @property + def folder_path(self) -> Path: + return self.folder.path + + def to_live_state(self, new_folder=True) -> SessionState: + """Recreate the live SessionState with underlying task from its stored version in the db. + + HACK: new_folder param is a bit hacky, as if we do not include the children if we + are not recomputing the folder hash. Might lead to some issues down the line. + """ + + if new_folder: + s_state = SessionState(self.folder.path) + else: + s_state = SessionState(self.folder.to_live_folder()) + + if s_state.folder_hash != self.folder.hash: + log.warning( + f"Folder hash mismatch for {self.folder.path}. " + f"Expected {self.folder.hash} but got {s_state.folder_hash}." + ) + s_state.id = self.id + s_state.created_at = self.created_at + s_state.updated_at = self.updated_at + s_state._task_states = [task.to_live_state(s_state) for task in self.tasks] + s_state.exc = pickle.loads(self.exc) if self.exc else None + return s_state + + def to_dict(self) -> SerializedSessionState: + return self.to_live_state(False).serialize() + + @classmethod + def get_by_hash_and_path( + cls, + hash: str | None, + path: Path | str | None, + db_session: Session | None = None, + ) -> SessionStateInDb | None: + """ + Get a session by its hash and if this fails, try its path. + + If multiple matches, returns the most recent one. + """ + from beets_flask.database import db_session_factory + + with db_session_factory(db_session) as db_session: + item = None + if hash is not None: + query = ( + select(cls) + .where(cls.folder_hash == hash) + # hash+revision combos have unique constraints + # and sessions always point to the latest / highest revision. + .order_by(cls.folder_revision.desc()) + ) + item = db_session.execute(query).scalars().first() + if item is None and path is not None: + # Try to get by path + # paths do not have revisions, always use last updated session + query = ( + select(cls) + .join(cls.folder) + .where(FolderInDb.full_path == str(path)) + .order_by(cls.updated_at.desc(), cls.folder_revision.desc()) + ) + item = db_session.execute(query).scalars().first() + + return item + + @property + def exception(self) -> SerializedException | None: + """Returns the exception of the session if it failed.""" + return pickle.loads(self.exc) if self.exc else None + + +class TaskStateInDb(Base): + """Represents an import task. + + More precisely, beets uses one task per album that goes through a bunch of stages. + We wrap the beets task in our TaskState to better handle its progress. + And this TaskState has a representation in our database, the TaskStateInDb. + """ + + __tablename__ = "task" + + # Relationships + session_id: Mapped[str] = mapped_column(ForeignKey("session.id")) + session: Mapped[SessionStateInDb] = relationship( + back_populates="tasks", + foreign_keys=[session_id], + ) + + candidates: Mapped[list[CandidateStateInDb]] = relationship( + back_populates="task", + foreign_keys="[CandidateStateInDb.task_id]", + cascade="all, delete-orphan", + ) + # Set at the end of the import session + chosen_candidate_id: Mapped[str | None] = mapped_column(ForeignKey("candidate.id")) + chosen_candidate: Mapped[CandidateStateInDb | None] = relationship( + back_populates="task", + foreign_keys=[chosen_candidate_id], + ) + + toppath: Mapped[bytes | None] + + # To reconstruct the beets task we need to store a few of its attributes + paths: Mapped[bytes] + old_paths: Mapped[bytes | None] + # old_paths contain original file paths, but are only set when files are moved. + # (which breaks some deep links that before were identical to paths, but no more!) + items: Mapped[bytes] + choice_flag: Mapped[Action | None] + + # To allow for continue we need to store the current artist and album + # TODO: REMOVE this is not needed!! We can look at the asis candidate for this! + # E.g. frontend component to compare two candidates + cur_artist: Mapped[str | None] + cur_album: Mapped[str | None] + + progress: Mapped[Progress] + + def __init__( + self, + id: str | None = None, + toppath: bytes | None = None, + paths: list[bytes] = [], + old_paths: list[bytes] | None = None, + items: list[LibraryItem] = [], + candidates: list[CandidateStateInDb] = [], + chosen_candidate_id: str | None = None, + progress: Progress = Progress.NOT_STARTED, + choice_flag: Action | None = None, + cur_artist: str | None = None, + cur_album: str | None = None, + ): + super().__init__(id) + self.toppath = toppath + self.paths = pickle.dumps(paths) + self.old_paths = pickle.dumps(old_paths) if old_paths else None + + for item in items: + # Remove db from all items as it can't be pickled + item._db = None + item._Item__album = None + + self.items = pickle.dumps(items) + self.candidates = candidates + self.chosen_candidate_id = chosen_candidate_id + self.progress = progress + self.choice_flag = choice_flag + self.cur_artist = cur_artist + self.cur_album = cur_album + + @classmethod + def from_live_state(cls, state: TaskState) -> TaskStateInDb: + """Create the DB representation of a live TaskState.""" + if hasattr(state.task, "old_paths"): + old_paths = state.task.old_paths + else: + old_paths = None + + task = cls( + id=state.id, + toppath=str(state.toppath).encode("utf-8") if state.toppath else None, + paths=state.task.paths, + items=state.task.items, + candidates=[ + CandidateStateInDb.from_live_state(c) for c in state.candidate_states + ], + chosen_candidate_id=state.chosen_candidate_state_id, + progress=state.progress.progress, + choice_flag=state.task.choice_flag, + cur_artist=state.task.cur_artist, + cur_album=state.task.cur_album, + old_paths=old_paths, + ) + return task + + def to_live_state(self, session_state: SessionState | None = None) -> TaskState: + """Recreate the live TaskState with underlying task from its stored version in the db.""" + + # We just assume it is a normal import task + beets_task = ImportTask( + toppath=self.toppath, + paths=pickle.loads(self.paths), + items=pickle.loads(self.items), + ) + beets_task.choice_flag = self.choice_flag + beets_task.cur_artist = self.cur_artist + beets_task.cur_album = self.cur_album + old_paths: list[bytes] | None = ( + pickle.loads(self.old_paths) if self.old_paths else None + ) + # TODO: Update type hints once beets is updated + beets_task.old_paths = old_paths # type: ignore + + live_state = TaskState(beets_task) + live_state.id = self.id + live_state.created_at = self.created_at + live_state.updated_at = self.updated_at + live_state.candidate_states = [ + c.to_live_state(live_state) for c in self.candidates + ] + live_state.chosen_candidate_state_id = self.chosen_candidate_id + live_state.progress.progress = self.progress + + # Set candidate of beets_task + live_state.task.candidates = [c.match for c in live_state.candidate_states] + + return live_state + + def to_dict(self) -> SerializedTaskState: + return self.to_live_state().serialize() + + +class CandidateStateInDb(Base): + """Represents a candidate (potential match) for an import task. + + Again: Beets-Candidate > CandidateState > CandidateStateInDb + """ + + __tablename__ = "candidate" + + task_id: Mapped[str] = mapped_column(ForeignKey("task.id")) + task: Mapped[TaskStateInDb] = relationship( + back_populates="candidates", + foreign_keys=[task_id], + ) + + # Should deserialize to AlbumMatch|TrackMatch + # ~4kb per match + match: Mapped[bytes] + + # Duplicate ids (if any) (beets_id) + duplicate_ids: Mapped[str] + + # association between tracks online and items on disk, from int to int + mapping: Mapped[dict[int, int]] + + def __init__( + self, + match: BeetsAlbumMatch | BeetsTrackMatch, + mapping: dict[int, int], + duplicate_ids: list[str] = [], + id: str | None = None, + ): + super().__init__(id) + + # Remove db from all items as it can't be pickled + # FIXME: this should go into beets __getstate__ method + # see https://github.com/beetbox/beets/pull/5641 + if isinstance(match, BeetsAlbumMatch): + for item in match.mapping.keys(): + item._db = None + item._Item__album = None + for item in match.extra_items: + item._db = None + item._Item__album = None + + self.match = pickle.dumps(match) + self.duplicate_ids = ";".join(map(str, duplicate_ids)) + self.mapping = mapping + + @classmethod + def from_live_state(cls, state: CandidateState) -> CandidateStateInDb: + """Create the DB representation of a live CandidateState.""" + return cls( + id=state.id, + match=state.match, + duplicate_ids=state.duplicate_ids, + mapping=state._mapping, + ) + + def to_live_state(self, task_state: TaskState | None) -> CandidateState: + """Recreate the live CandidateState with underlying task from its stored version in the db.""" + if task_state is None: + task_state = self.task.to_live_state() + live_state = CandidateState( + CustomUnpickler(io.BytesIO(self.match)).load(), + task_state, + mapping=self.mapping, + ) + live_state.id = self.id + live_state.created_at = self.created_at + live_state.updated_at = self.updated_at + live_state.duplicate_ids = ( + # edge case: "".split() gives [''] + [] if len(self.duplicate_ids) == 0 else self.duplicate_ids.split(";") + ) + return live_state + + def to_dict(self) -> SerializedCandidateState: + return self.to_live_state(self.task.to_live_state()).serialize() + + +# Hotfix for match unpickler to resolve beets distance moved +# This is needed because various beets updates changed class implementations +# and we want to rebuild the newer versions of some beets classes from old pickles. +# TODO: We should fix this in general and not pickle beets objects +class CustomUnpickler(pickle.Unpickler): + def find_class(self, module, name): + """Override the find_class method to redirect Distance class references.""" + # Redirect Distance class from beets.autotag.hooks to beets.distance (2.4.0) + if module == "beets.autotag.hooks" and name == "Distance": + return Distance + + # For all other classes, use the default lookup mechanism + return super().find_class(module, name) + + def load(self) -> Any: + object = super().load() + if isinstance(object, Distance): + self.patch_distance(object) + + if isinstance(object, AlbumMatch): + self.patch_distance(object.distance) + + return object + + def patch_distance(self, distance: Distance) -> Distance: + # Rewrite "source" penalty to "data_source" penalty (2.5.0) + if "source" in distance._penalties: + log.debug( + "Converting old distance.source to distance.data_source (changed in beets 2.5.0)" + ) + distance._penalties["data_source"] = distance._penalties["source"] + del distance._penalties["source"] + + # Potential infinite recursion, ah well + for track, d in distance.tracks.items(): + self.patch_distance(d) + return distance + + +__all__ = ["SessionStateInDb", "TaskStateInDb", "CandidateStateInDb"] diff --git a/backend/beets_flask/database/models/stats.py b/backend/beets_flask/database/models/stats.py new file mode 100644 index 00000000..31e47457 --- /dev/null +++ b/backend/beets_flask/database/models/stats.py @@ -0,0 +1,51 @@ +"""Cached filesystem statistics model. + +Stores pre-computed dir size and file counts so that the stats endpoints +(home page) can return instantly without running expensive subprocesses. + +Values are written by: +- The watchdog after its debounce fires (inbox stats) +- The import workers after an import completes (library stats) +- On watchdog startup for each configured inbox +""" + +from __future__ import annotations + +from pathlib import Path + +from sqlalchemy.orm import Mapped, mapped_column +from sqlalchemy.sql import func + +from beets_flask.database.models.base import Base + + +class CachedStatInDb(Base): + """Cached filesystem stats for a single directory path. + + The ``id`` (inherited primary key) is set to the resolved absolute path + string so each path maps to exactly one row. + """ + + __tablename__ = "cached_stats" + + # Size of the directory tree in bytes (du -sb) + size_bytes: Mapped[int | None] = mapped_column(default=None) + # Total number of files/dirs under the path (find | wc -l) + n_files: Mapped[int | None] = mapped_column(default=None) + # When these values were last computed + computed_at: Mapped[float] = mapped_column(default=func.now()) + + def __init__( + self, + path: Path | str, + size_bytes: int | None = None, + n_files: int | None = None, + ): + path = Path(path) + super().__init__(id=str(path.resolve())) + self.size_bytes = size_bytes + self.n_files = n_files + + @property + def path(self) -> Path: + return Path(self.id) diff --git a/backend/beets_flask/database/models/types.py b/backend/beets_flask/database/models/types.py index 4d558a60..ad108efa 100644 --- a/backend/beets_flask/database/models/types.py +++ b/backend/beets_flask/database/models/types.py @@ -1,67 +1,67 @@ -import json -from typing import Any - -from sqlalchemy import types - - -class DictType(types.TypeDecorator): - """Stores a dict[str, Any] as a JSON-encoded string in the database. - - Allows for flexible storage of dictionaries with string keys and values of - any (serializable) type. - """ - - impl = types.Text - cache_ok = True - - allowed_keys_types: tuple[type, ...] = (str,) - allowed_values_types: tuple[type | Any, ...] = (Any,) - - def process_bind_param(self, value, dialect): - if value is None: - return None - if not isinstance(value, dict): - raise ValueError("Value must be a dict") - - # Any type needs some special handling - allowed_types_v: tuple[type, ...] = tuple( - filter(lambda x: x is not Any, self.allowed_values_types) - ) - - if not len(allowed_types_v) == 0: - if not all(isinstance(v, allowed_types_v) for v in value.values()): - raise ValueError( - f"Value must be a dict with values of type {allowed_types_v}. Got: {value.values()}" - ) - - if not all(isinstance(k, self.allowed_keys_types) for k in value.keys()): - raise ValueError(f"Keys must be of type {self.allowed_keys_types}.") - - return json.dumps({str(k): v for k, v in value.items()}) - - def process_result_value(self, value, dialect): - if value is None: - return None - return json.loads(value) - - def copy(self, **kw): - return self.__class__(self.impl.length) # type: ignore - - -class IntDictType(DictType): - """Stores a dict[int, int] as a JSON-encoded string in the database.""" - - allowed_keys_types = (int,) - allowed_values_types = (int,) - - def process_result_value(self, value, dialect): - if value is None: - return None - return {int(k): int(v) for k, v in json.loads(value).items()} - - -class StrDictType(DictType): - """Stores a dict[str, str] as a JSON-encoded string in the database.""" - - allowed_keys_types = (str,) - allowed_values_types = (str,) +import json +from typing import Any + +from sqlalchemy import types + + +class DictType(types.TypeDecorator): + """Stores a dict[str, Any] as a JSON-encoded string in the database. + + Allows for flexible storage of dictionaries with string keys and values of + any (serializable) type. + """ + + impl = types.Text + cache_ok = True + + allowed_keys_types: tuple[type, ...] = (str,) + allowed_values_types: tuple[type | Any, ...] = (Any,) + + def process_bind_param(self, value, dialect): + if value is None: + return None + if not isinstance(value, dict): + raise ValueError("Value must be a dict") + + # Any type needs some special handling + allowed_types_v: tuple[type, ...] = tuple( + filter(lambda x: x is not Any, self.allowed_values_types) + ) + + if not len(allowed_types_v) == 0: + if not all(isinstance(v, allowed_types_v) for v in value.values()): + raise ValueError( + f"Value must be a dict with values of type {allowed_types_v}. Got: {value.values()}" + ) + + if not all(isinstance(k, self.allowed_keys_types) for k in value.keys()): + raise ValueError(f"Keys must be of type {self.allowed_keys_types}.") + + return json.dumps({str(k): v for k, v in value.items()}) + + def process_result_value(self, value, dialect): + if value is None: + return None + return json.loads(value) + + def copy(self, **kw): + return self.__class__(self.impl.length) # type: ignore + + +class IntDictType(DictType): + """Stores a dict[int, int] as a JSON-encoded string in the database.""" + + allowed_keys_types = (int,) + allowed_values_types = (int,) + + def process_result_value(self, value, dialect): + if value is None: + return None + return {int(k): int(v) for k, v in json.loads(value).items()} + + +class StrDictType(DictType): + """Stores a dict[str, str] as a JSON-encoded string in the database.""" + + allowed_keys_types = (str,) + allowed_values_types = (str,) diff --git a/backend/beets_flask/database/setup.py b/backend/beets_flask/database/setup.py index 09a3bb63..fc771cf1 100644 --- a/backend/beets_flask/database/setup.py +++ b/backend/beets_flask/database/setup.py @@ -1,128 +1,128 @@ -from contextlib import contextmanager -from functools import wraps - -from quart import Quart -from sqlalchemy import Engine, create_engine -from sqlalchemy.orm import Session, scoped_session, sessionmaker - -from beets_flask.config import get_flask_config -from beets_flask.logger import log - -from .models import Base - -engine: Engine | None = None -session_factory: scoped_session[Session] - - -def setup_database(app: Quart | None = None) -> None: - """Set up the database connection and session factory for the FLask application. - - This function initializes the global `engine` and `session_factory` variables - using the database URI specified in the application's configuration. It also - sets up a teardown hook to gracefully close the database session when the - application context is torn down. - - Args: - app (Quart): The Quart application instance. - - Returns - ------- - None - """ - __setup_factory() - if get_flask_config()["RESET_DB_ON_START"]: - log.warning("Resetting database due to RESET_DB=True in config") - _reset_database() - - _create_tables(engine) - - if app is not None: - # Gracefully shutdown the database session, if launched - # from within a Flask app context. - @app.teardown_appcontext - def shutdown_session(exception=None) -> None: - session_factory.remove() - - -def __setup_factory(): - global engine - global session_factory - - engine = create_engine(get_flask_config()["DATABASE_URI"]) - session_factory = scoped_session(sessionmaker(bind=engine, expire_on_commit=False)) - - -@contextmanager -def db_session_factory(session: Session | None = None): - """Databases session as context. - - Makes sure sessions are closed at the end. - If an existing session is provided, it will not be closed at the end. - This allows to wrap multiple `with db_session()` blocks around each other without closing the outer session. - - Example: - ``` - with db_session() as session: - tag.foo = "bar" - session.merge(tag) - return tag.to_dict() - - existingSession = session_factory() - with db_session(session) as s: - tag.foo = "bar" - s.merge(tag) - return tag.to_dict() - ``` - """ - is_outermost = session is None - if is_outermost: - try: - session = session_factory() - except NameError: - __setup_factory() - session = session_factory() - - try: - # mypy does not resolve our try/catch for None-Type check. ignore type errors.`` - yield session - session.commit() # type: ignore - except: - session.rollback() # type: ignore - raise - finally: - if is_outermost: - session.close() # type: ignore - - -def with_db_session(func): - """Decorate a function with a db session as a keyword argument to the function. - - Example - ``` - @with_db_session - def my_function(session=None): - tag.foo = "bar" - session.merge(tag) - return tag.to_dict() - ``` - """ - - @wraps(func) - def wrapper(*args, **kwargs): - with db_session_factory() as session: - kwargs.setdefault("session", session) - return func(*args, **kwargs) - - return wrapper - - -def _create_tables(engine) -> None: - Base.metadata.create_all(bind=engine) - - -def _reset_database(): - # Removes all data from the database but keeps schema - for t in reversed(Base.metadata.sorted_tables): - with db_session_factory() as session: - session.execute(t.delete()) - session.commit() +from contextlib import contextmanager +from functools import wraps + +from quart import Quart +from sqlalchemy import Engine, create_engine +from sqlalchemy.orm import Session, scoped_session, sessionmaker + +from beets_flask.config import get_flask_config +from beets_flask.logger import log + +from .models import Base + +engine: Engine | None = None +session_factory: scoped_session[Session] + + +def setup_database(app: Quart | None = None) -> None: + """Set up the database connection and session factory for the FLask application. + + This function initializes the global `engine` and `session_factory` variables + using the database URI specified in the application's configuration. It also + sets up a teardown hook to gracefully close the database session when the + application context is torn down. + + Args: + app (Quart): The Quart application instance. + + Returns + ------- + None + """ + __setup_factory() + if get_flask_config()["RESET_DB_ON_START"]: + log.warning("Resetting database due to RESET_DB=True in config") + _reset_database() + + _create_tables(engine) + + if app is not None: + # Gracefully shutdown the database session, if launched + # from within a Flask app context. + @app.teardown_appcontext + def shutdown_session(exception=None) -> None: + session_factory.remove() + + +def __setup_factory(): + global engine + global session_factory + + engine = create_engine(get_flask_config()["DATABASE_URI"]) + session_factory = scoped_session(sessionmaker(bind=engine, expire_on_commit=False)) + + +@contextmanager +def db_session_factory(session: Session | None = None): + """Databases session as context. + + Makes sure sessions are closed at the end. + If an existing session is provided, it will not be closed at the end. + This allows to wrap multiple `with db_session()` blocks around each other without closing the outer session. + + Example: + ``` + with db_session() as session: + tag.foo = "bar" + session.merge(tag) + return tag.to_dict() + + existingSession = session_factory() + with db_session(session) as s: + tag.foo = "bar" + s.merge(tag) + return tag.to_dict() + ``` + """ + is_outermost = session is None + if is_outermost: + try: + session = session_factory() + except NameError: + __setup_factory() + session = session_factory() + + try: + # mypy does not resolve our try/catch for None-Type check. ignore type errors.`` + yield session + session.commit() # type: ignore + except: + session.rollback() # type: ignore + raise + finally: + if is_outermost: + session.close() # type: ignore + + +def with_db_session(func): + """Decorate a function with a db session as a keyword argument to the function. + + Example + ``` + @with_db_session + def my_function(session=None): + tag.foo = "bar" + session.merge(tag) + return tag.to_dict() + ``` + """ + + @wraps(func) + def wrapper(*args, **kwargs): + with db_session_factory() as session: + kwargs.setdefault("session", session) + return func(*args, **kwargs) + + return wrapper + + +def _create_tables(engine) -> None: + Base.metadata.create_all(bind=engine) + + +def _reset_database(): + # Removes all data from the database but keeps schema + for t in reversed(Base.metadata.sorted_tables): + with db_session_factory() as session: + session.execute(t.delete()) + session.commit() diff --git a/backend/beets_flask/dirhash_custom.py b/backend/beets_flask/dirhash_custom.py index 3555b496..006f6a7a 100644 --- a/backend/beets_flask/dirhash_custom.py +++ b/backend/beets_flask/dirhash_custom.py @@ -1,92 +1,92 @@ -import os -from hashlib import md5 -from pathlib import Path -from re import Pattern - -from cachetools import Cache - - -def dirhash_c( - dirname: str | Path, - cache: Cache[str, bytes] | None, - filter_regex: Pattern[str] | None = None, -) -> bytes: - """Compute a hash for a directory. - - The hash is computed by hashing the path of each file and using - its filesystem metadata (size, mtime, ctime). - - Parameters - ---------- - dirname: str - The path to the directory - cache: dict, optional - A cache object to store intermediate results. If None, no caching is used. - filter_regex: re.Pattern, optional - When calculating checksum contributon for files, only consider - those that match the provided pattern. - """ - if isinstance(dirname, Path): - dirname = str(dirname.resolve()) - - if cache is not None and dirname in cache: - return cache[dirname] - - hash = md5() - - # Hash for each entry in the directory - for entry in os.scandir(dirname): - if entry.is_dir(): - hash.update(dirhash_c(entry.path, cache, filter_regex)) - else: - # Skip files that do not match the filter - if filter_regex is not None and not filter_regex.match(entry.name): - continue - - fs = os.stat(entry.path) - hash.update(fs.st_size.to_bytes(8, byteorder="big")) - hash.update(fs.st_ino.to_bytes(8, byteorder="big")) - hash.update(str(fs.st_mtime).encode()) - hash.update(entry.name.encode()) - - # dirstats - fs = os.stat(dirname) - hash.update(dirname.encode()) - # hash.update(fs.st_size.to_bytes(8, byteorder="big")) - # hash.update(fs.st_ino.to_bytes(8, byteorder="big")) - # hash.update(str(fs.st_mtime).encode()) - - # for dirs we should use very little info. - # For instance, mtime, ino, size get changed when e.g. a file - # is added directly inside - and this becomes somewhat inconsistent with - # regex ignore patterns. (adding an ignored file would still modify the hash) - - return hash.digest() - - -def archive_hash( - f_path: str | Path, - cache: Cache[str, bytes] | None = None, -) -> bytes: - """Compute a hash for an archive file.""" - - if isinstance(f_path, Path): - f_path = str(f_path.resolve()) - - if cache is not None and f_path in cache: - return cache[f_path] - - hash = md5() - fs = os.stat(f_path) - hash.update(fs.st_size.to_bytes(8, byteorder="big")) - hash.update(fs.st_ino.to_bytes(8, byteorder="big")) - hash.update(str(fs.st_mtime).encode()) - hash.update(os.path.basename(f_path).encode()) - - if cache is not None: - cache[f_path] = hash.digest() - - return hash.digest() - - -__all__ = ["dirhash_c", "archive_hash"] +import os +from hashlib import md5 +from pathlib import Path +from re import Pattern + +from cachetools import Cache + + +def dirhash_c( + dirname: str | Path, + cache: Cache[str, bytes] | None, + filter_regex: Pattern[str] | None = None, +) -> bytes: + """Compute a hash for a directory. + + The hash is computed by hashing the path of each file and using + its filesystem metadata (size, mtime, ctime). + + Parameters + ---------- + dirname: str + The path to the directory + cache: dict, optional + A cache object to store intermediate results. If None, no caching is used. + filter_regex: re.Pattern, optional + When calculating checksum contributon for files, only consider + those that match the provided pattern. + """ + if isinstance(dirname, Path): + dirname = str(dirname.resolve()) + + if cache is not None and dirname in cache: + return cache[dirname] + + hash = md5() + + # Hash for each entry in the directory + for entry in os.scandir(dirname): + if entry.is_dir(): + hash.update(dirhash_c(entry.path, cache, filter_regex)) + else: + # Skip files that do not match the filter + if filter_regex is not None and not filter_regex.match(entry.name): + continue + + fs = os.stat(entry.path) + hash.update(fs.st_size.to_bytes(8, byteorder="big")) + hash.update(fs.st_ino.to_bytes(8, byteorder="big")) + hash.update(str(fs.st_mtime).encode()) + hash.update(entry.name.encode()) + + # dirstats + fs = os.stat(dirname) + hash.update(dirname.encode()) + # hash.update(fs.st_size.to_bytes(8, byteorder="big")) + # hash.update(fs.st_ino.to_bytes(8, byteorder="big")) + # hash.update(str(fs.st_mtime).encode()) + + # for dirs we should use very little info. + # For instance, mtime, ino, size get changed when e.g. a file + # is added directly inside - and this becomes somewhat inconsistent with + # regex ignore patterns. (adding an ignored file would still modify the hash) + + return hash.digest() + + +def archive_hash( + f_path: str | Path, + cache: Cache[str, bytes] | None = None, +) -> bytes: + """Compute a hash for an archive file.""" + + if isinstance(f_path, Path): + f_path = str(f_path.resolve()) + + if cache is not None and f_path in cache: + return cache[f_path] + + hash = md5() + fs = os.stat(f_path) + hash.update(fs.st_size.to_bytes(8, byteorder="big")) + hash.update(fs.st_ino.to_bytes(8, byteorder="big")) + hash.update(str(fs.st_mtime).encode()) + hash.update(os.path.basename(f_path).encode()) + + if cache is not None: + cache[f_path] = hash.digest() + + return hash.digest() + + +__all__ = ["dirhash_c", "archive_hash"] diff --git a/backend/beets_flask/disk.py b/backend/beets_flask/disk.py index 3e71e31f..fcaaf815 100644 --- a/backend/beets_flask/disk.py +++ b/backend/beets_flask/disk.py @@ -1,474 +1,546 @@ -from __future__ import annotations - -import os -import re -import subprocess -from abc import ABC, abstractmethod -from collections.abc import Iterator, Sequence -from dataclasses import dataclass -from fnmatch import fnmatch -from pathlib import Path -from typing import ( - Literal, -) - -from beets.importer import ( - ArchiveImportTask, -) -from beets.importer.tasks import ( - MULTIDISC_MARKERS, - MULTIDISC_PAT_FMT, - albums_in_dir, -) -from beets.util import bytestring_path -from cachetools import Cache, TTLCache, cached -from natsort import os_sorted - -from beets_flask.config import get_config -from beets_flask.dirhash_custom import archive_hash, dirhash_c -from beets_flask.logger import log -from beets_flask.utility import AUDIO_EXTENSIONS - -# Regex pattern to exclude hidden files (files starting with ".") -audio_regex = re.compile( - r".*\.(" + "|".join(AUDIO_EXTENSIONS) + ")$", - re.IGNORECASE, -) - - -@dataclass -class FileSystemItem(ABC): - """Base class for file system items.""" - - type: Literal["file", "directory", "archive"] - full_path: str - hash: str - - # If beets has marked this folder as an album, or, if its a file, archives can be - # imported. Singletons (importing music files directly) is not supported yet. - is_album: bool - - @property - def path(self) -> Path: - # For convenience, to get `full_path` as a Path object, - # but in the frontend and sqlite database, we use strings (full_path). - return Path(self.full_path) - - @path.setter - def path(self, value: Path) -> None: - self.full_path = str(value) - - @classmethod - @abstractmethod - def from_path( - cls, path: Path | str, cache: Cache[str, bytes] | None = None - ) -> FileSystemItem: - """Create a FileSystemItem object from a path.""" - raise NotImplementedError("This method should be implemented in subclasses.") - - -def fs_item_from_path( - path: Path | str, cache: Cache[str, bytes] | None = None, subdirs: bool = True -) -> File | Folder | Archive: - """Create a _specific_ FileSystemItem from a path.""" - if isinstance(path, str): - path = Path(path) - - if path.is_dir(): - return Folder.from_path(path, cache=cache, subdirs=subdirs) - elif is_archive_file(path): - return Archive.from_path(path, cache=cache) - else: - return File.from_path(path, cache=cache) - - -def _matches_patterns(s: str, patterns: list[str]) -> bool: - """Check if a string matches any of the given patterns.""" - return any(fnmatch(s, pat) for pat in patterns) - - -@dataclass -class Folder(FileSystemItem): - children: Sequence[FileSystemItem] - - def __init__( - self, - children: Sequence[FileSystemItem], - full_path: str, - hash: str, - is_album: bool = False, - ): - super().__init__( - full_path=full_path, - hash=hash, - is_album=is_album, - type="directory", - ) - self.children = children - - @classmethod - def from_path( - cls, - path: Path | str, - cache: Cache[str, bytes] | None = None, - subdirs=True, - ) -> Folder: - """Create a Folder object from a path.""" - - ignore_globs = get_config().ignore_globs - - if isinstance(path, str): - path = Path(path) - - if not path.is_dir(): - raise FileNotFoundError(f"Path `{path}` does not exist or is no directory.") - - path = path.resolve() - - album_folders = all_album_folders(path, subdirs=subdirs) - - # Cache for the dirhash function - if cache is None: - cache = Cache(maxsize=2**16) - - # Object that contains all tree elements, because - # we need to fill top down but iterate buttom up - lookup: dict[str, Folder] = dict() - - # Iterate over all directories from bottom to top - for dirpath, dirnames, filenames in os.walk(path, topdown=False): - if _matches_patterns(os.path.basename(dirpath), ignore_globs): - continue - - # Skip ignored files - # TODO: I think we could optimize this by - # compiling to regex - _dirnames = filter( - lambda d: not _matches_patterns(d, ignore_globs), dirnames - ) - _filenames = filter( - lambda f: not _matches_patterns(f, ignore_globs), filenames - ) - - # As we iterate from bottom to top, we can access the elements from - # the lookup table as they are already created - children: list[FileSystemItem] = [ - lookup[os.path.join(dirpath, sub_dir)] - for sub_dir in os_sorted(_dirnames) - ] - - # Add all files to children - for filename in os_sorted(_filenames): - full_path = os.path.join(dirpath, filename) - # Here, we know this not a folder, so we can use fs_item_from_path. - children.append( - fs_item_from_path(path=os.path.abspath(full_path), cache=cache) - ) - - # Add current directory to lookup - lookup[dirpath] = Folder( - children=children, - full_path=os.path.abspath(dirpath), - hash=dirhash_c( - dirpath, - cache, - filter_regex=audio_regex, # Only hash audio files - ).hex(), - is_album=Path(dirpath) in album_folders, - ) - - return lookup[str(path)] - - def walk(self) -> Iterator[FileSystemItem]: - """Walk the folder and yield all files and folders.""" - yield self - for child in self.children: - if isinstance(child, Folder): - yield from child.walk() - else: - yield child - - -@dataclass -class Archive(FileSystemItem): - # Defaults to true, as we assume that we can import an archive as an album - is_album: bool = True - - def __init__(self, full_path: str, hash: str): - super().__init__( - full_path=full_path, - hash=hash, - is_album=True, # Archives are always considered albums - type="archive", - ) - - @classmethod - def from_path( - cls, path: Path | str, cache: Cache[str, bytes] | None = None - ) -> Archive: - """Create an Archive object from a path.""" - if isinstance(path, str): - path = Path(path) - - if not is_archive_file(path): - raise FileNotFoundError(f"Path `{path}` is not an archive file.") - - if cache is None: - cache = Cache(maxsize=2**16) - - return cls( - full_path=str(path.resolve()), - hash=archive_hash(path, cache=cache).hex(), - ) - - -def is_archive_file(path: Path | str) -> bool: - """Check if a file is an archive file based on its extension.""" - return ArchiveImportTask.is_archive(str(path)) - - -@dataclass -class File(FileSystemItem): - def __init__(self, full_path: str): - super().__init__( - full_path=full_path, - hash="", # Files do not have a hash atm (maybe later we can add a hash) - is_album=False, # Files are not considered albums - type="file", - ) - - @classmethod - def from_path( - cls, path: Path | str, cache: Cache[str, bytes] | None = None - ) -> File: - """Create a File object from a path.""" - if isinstance(path, str): - path = Path(path) - - if not path.is_file(): - raise FileNotFoundError(f"Path `{path}` is not a file.") - - full_path = str(path.resolve()) - - return cls( - full_path=full_path, - ) - - -@cached(cache=TTLCache(maxsize=1024, ttl=900), info=True) -def path_to_folder(root_dir: Path | str, subdirs=True) -> Folder: - """Generate our nested dict structure for the specified path. - - Parameters - ---------- - root_dir : str - The root directory to start from. - subdirs : bool, optional - Whether to mark qualifying subfolders of an album as album folders themselves. If true, e.g. for `/album/CD1/track.mp3` both `/album/` and `/album/CD1/` are flagged. Defaults to True. - - Returns - ------- - dict: The nested dict structure. - """ - - return Folder.from_path(root_dir, subdirs=subdirs) - - -def album_folders_from_track_paths( - track_paths: list[Path] | list[str], use_parent_for_multidisc: bool = True -) -> list[Path]: - """Get all album folders from a list of paths to files. - - Parameters - ---------- - track_paths : list[Path] - list of track paths, e.g. mp3 files. - use_parent_for_multidisc : bool, optional - When files are in an album folder that might be a multi-disc folder (e.g. `/album/cd1`), - return the parent (`/album`) instead of the lowest-level-folder (`/cd1`). Defaults to True. - - Returns - ------- - list[str]: album folders - """ - - folders_to_check: set[Path] = set() - album_folders: set[Path] = set() - for path in track_paths: - # FIXME: For backwards compatibility, we allow a string as input - if isinstance(path, str): - path = Path(path) - - if is_archive_file(path): - album_folders.add(path.resolve()) - elif path.is_file(): - folders_to_check.add(path.parent.resolve()) - else: - # just to be nice and manage directories instead of files - folders_to_check.add(path.resolve()) - - for folder in folders_to_check: - afs = all_album_folders(folder, subdirs=True) - for af in afs: - album_folders.add(af) - - if use_parent_for_multidisc: - parents: set[Path] = set() - children: set[Path] = set() - for folder in album_folders: - if is_within_multi_dir(folder): - parents.add(folder.parent) - children.add(folder) - - album_folders = album_folders - children - album_folders = album_folders.union(parents) - - return sorted(album_folders, key=lambda s: str(s).lower()) - - -def is_album_folder(path: Path | str): - """Check if a path is an album folder. - - Returns true if the path is detected as an album by beets, or if it is an archive file. - ------- - path : Path | str - The path to check, can be a folder, file or archive. - - Note - ---- - Except in tests, we dont use this function yet. - Its logic is duplicated in `all_album_folders`. (We should consolidate.) - """ - if isinstance(path, str): - path = Path(path).absolute() - if is_archive_file(path): - return True - for paths, items in albums_in_dir(bytestring_path(path)): - if all(is_archive_file(i.decode("utf-8")) for i in items): - continue - if str(path).encode("utf-8") in paths: - return True - return False - - -def all_album_folders(root_dir: Path | str, subdirs: bool = False) -> list[Path]: - """ - Get all album folders from a given root dir. - - Parameters - ---------- - root_dir : str - toppath, highest level to start searching. - subdirs : bool, optional - Whether to return subfolders of an album that themselves would qualify. - E.g. a `CD1` folder. Defaults to False. - - Returns - ------- - list[Path] - """ - - # FIXME: For backwards compatibility, we allow a string as input - if isinstance(root_dir, str): - root_dir = Path(root_dir) - - folders: list[bytes] = [] - for paths, items in albums_in_dir(bytestring_path(root_dir.absolute())): - # Our choice on handling archives: - # - archives are always simple albums. no multi-disc logic supported, - # all discs need to be _inside_ the archive. - # - if a folder contains only archives, it will never be considered an - # album folder - # - if a folder contains a mix of archives and music files, it will be - # considered an album folder (as we think archives might be metadata or additional files e.g. cover art) - - if all(is_archive_file(i.decode("utf-8")) for i in items): - folders.extend(items) - continue - - if subdirs: - folders.extend(p for p in paths) - else: - # the top-level path is always the first in the list - # however, there is an edgecase, if we have a rogue element in a multi-disc folder: - # - artist/album/should_not_be_here.mp3 - # - artist/album/CD1/track.mp3 - # - artist/album/CD2/track.mp3 - # -> then albums_in_dir returns [album], [CD1, CD2] so that picking the first element is wrong. - # we would want all 3: album, CD1 and CD2. but in this case, the parent `album` should already - # be in our set when we check [CD1, CD2] - if os.path.dirname(paths[0]) in folders: - folders.extend(p for p in paths) - else: - folders.append(paths[0]) - - return [Path(f.decode("utf-8")) for f in folders] - - -def is_within_multi_dir(path: Path | str) -> bool: - """ - Minimal version of beets heuristic to check if a string matches a multi-disc pattern. - - E.g. "My Album CD1" or "Disc 2" will return True - """ - - # FIXME: For backwards compatibility, we allow a string as input - if isinstance(path, str): - path = Path(path) - - path_str = path.name # Use pathlib to get the basename - - for marker in MULTIDISC_MARKERS: - p = MULTIDISC_PAT_FMT.replace(b"%s", marker) - marker_pat = re.compile(p, re.I) - match = marker_pat.match(path_str.encode("utf-8")) - if match: - return True - return False - - -@cached(cache=TTLCache(maxsize=1024, ttl=60), info=True) -def dir_size(path: Path) -> int: - """Size of a dir in bytes, including content.""" - try: - result = subprocess.run( - ["du", "-sb", str(path.resolve())], - capture_output=True, - text=True, - check=True, - ) - size = int(result.stdout.split()[0]) - return size - except Exception as e: - # this happens e.g. if the directory does not exist. - log.error(e) - return -1 - - -@cached(cache=TTLCache(maxsize=1024, ttl=60), info=True) -def dir_files(path: Path) -> int: - """Count the number of files in a directory.""" - try: - result = subprocess.run( - [f"find {str(path.resolve())} | wc -l"], - capture_output=True, - text=True, - check=True, - shell=True, - ) - count = int(result.stdout) - return count - except Exception as e: - # this happens e.g. if the directory does not exist. - log.error(e) - return -1 - - -def clear_cache(): - """Clear the cache for all cached functions.""" - path_to_folder.cache.clear() # type: ignore - dir_size.cache.clear() # type: ignore - dir_files.cache.clear() # type: ignore - - -__all__ = ["dir_size", "fs_item_from_path"] +from __future__ import annotations + +import asyncio +import os +import re +import subprocess +from abc import ABC, abstractmethod +from collections.abc import Iterator, Sequence +from dataclasses import dataclass +from fnmatch import fnmatch +from pathlib import Path +from typing import ( + TYPE_CHECKING, + Literal, +) + +if TYPE_CHECKING: + from beets_flask.database.models.stats import CachedStatInDb + +from beets.importer import ( + ArchiveImportTask, +) +from beets.importer.tasks import ( + MULTIDISC_MARKERS, + MULTIDISC_PAT_FMT, + albums_in_dir, +) +from beets.util import bytestring_path +from cachetools import Cache, TTLCache, cached +from natsort import os_sorted + +from beets_flask.config import get_config +from beets_flask.dirhash_custom import archive_hash, dirhash_c +from beets_flask.logger import log +from beets_flask.utility import AUDIO_EXTENSIONS + +# Regex pattern to exclude hidden files (files starting with ".") +audio_regex = re.compile( + r".*\.(" + "|".join(AUDIO_EXTENSIONS) + ")$", + re.IGNORECASE, +) + + +@dataclass +class FileSystemItem(ABC): + """Base class for file system items.""" + + type: Literal["file", "directory", "archive"] + full_path: str + hash: str + + # If beets has marked this folder as an album, or, if its a file, archives can be + # imported. Singletons (importing music files directly) is not supported yet. + is_album: bool + + @property + def path(self) -> Path: + # For convenience, to get `full_path` as a Path object, + # but in the frontend and sqlite database, we use strings (full_path). + return Path(self.full_path) + + @path.setter + def path(self, value: Path) -> None: + self.full_path = str(value) + + @classmethod + @abstractmethod + def from_path( + cls, path: Path | str, cache: Cache[str, bytes] | None = None + ) -> FileSystemItem: + """Create a FileSystemItem object from a path.""" + raise NotImplementedError("This method should be implemented in subclasses.") + + +def fs_item_from_path( + path: Path | str, cache: Cache[str, bytes] | None = None, subdirs: bool = True +) -> File | Folder | Archive: + """Create a _specific_ FileSystemItem from a path.""" + if isinstance(path, str): + path = Path(path) + + if path.is_dir(): + return Folder.from_path(path, cache=cache, subdirs=subdirs) + elif is_archive_file(path): + return Archive.from_path(path, cache=cache) + else: + return File.from_path(path, cache=cache) + + +def _matches_patterns(s: str, patterns: list[str]) -> bool: + """Check if a string matches any of the given patterns.""" + return any(fnmatch(s, pat) for pat in patterns) + + +@dataclass +class Folder(FileSystemItem): + children: Sequence[FileSystemItem] + + def __init__( + self, + children: Sequence[FileSystemItem], + full_path: str, + hash: str, + is_album: bool = False, + ): + super().__init__( + full_path=full_path, + hash=hash, + is_album=is_album, + type="directory", + ) + self.children = children + + @classmethod + def from_path( + cls, + path: Path | str, + cache: Cache[str, bytes] | None = None, + subdirs=True, + ) -> Folder: + """Create a Folder object from a path.""" + + ignore_globs = get_config().ignore_globs + + if isinstance(path, str): + path = Path(path) + + if not path.is_dir(): + raise FileNotFoundError(f"Path `{path}` does not exist or is no directory.") + + path = path.resolve() + + album_folders = all_album_folders(path, subdirs=subdirs) + + # Cache for the dirhash function + if cache is None: + cache = Cache(maxsize=2**16) + + # Object that contains all tree elements, because + # we need to fill top down but iterate buttom up + lookup: dict[str, Folder] = dict() + + # Iterate over all directories from bottom to top + for dirpath, dirnames, filenames in os.walk(path, topdown=False): + if _matches_patterns(os.path.basename(dirpath), ignore_globs): + continue + + # Skip ignored files + # TODO: I think we could optimize this by + # compiling to regex + _dirnames = filter( + lambda d: not _matches_patterns(d, ignore_globs), dirnames + ) + _filenames = filter( + lambda f: not _matches_patterns(f, ignore_globs), filenames + ) + + # As we iterate from bottom to top, we can access the elements from + # the lookup table as they are already created + children: list[FileSystemItem] = [ + lookup[os.path.join(dirpath, sub_dir)] + for sub_dir in os_sorted(_dirnames) + ] + + # Add all files to children + for filename in os_sorted(_filenames): + full_path = os.path.join(dirpath, filename) + # Here, we know this not a folder, so we can use fs_item_from_path. + children.append( + fs_item_from_path(path=os.path.abspath(full_path), cache=cache) + ) + + # Add current directory to lookup + lookup[dirpath] = Folder( + children=children, + full_path=os.path.abspath(dirpath), + hash=dirhash_c( + dirpath, + cache, + filter_regex=audio_regex, # Only hash audio files + ).hex(), + is_album=Path(dirpath) in album_folders, + ) + + return lookup[str(path)] + + def walk(self) -> Iterator[FileSystemItem]: + """Walk the folder and yield all files and folders.""" + yield self + for child in self.children: + if isinstance(child, Folder): + yield from child.walk() + else: + yield child + + +@dataclass +class Archive(FileSystemItem): + # Defaults to true, as we assume that we can import an archive as an album + is_album: bool = True + + def __init__(self, full_path: str, hash: str): + super().__init__( + full_path=full_path, + hash=hash, + is_album=True, # Archives are always considered albums + type="archive", + ) + + @classmethod + def from_path( + cls, path: Path | str, cache: Cache[str, bytes] | None = None + ) -> Archive: + """Create an Archive object from a path.""" + if isinstance(path, str): + path = Path(path) + + if not is_archive_file(path): + raise FileNotFoundError(f"Path `{path}` is not an archive file.") + + if cache is None: + cache = Cache(maxsize=2**16) + + return cls( + full_path=str(path.resolve()), + hash=archive_hash(path, cache=cache).hex(), + ) + + +def is_archive_file(path: Path | str) -> bool: + """Check if a file is an archive file based on its extension.""" + return ArchiveImportTask.is_archive(str(path)) + + +@dataclass +class File(FileSystemItem): + def __init__(self, full_path: str): + super().__init__( + full_path=full_path, + hash="", # Files do not have a hash atm (maybe later we can add a hash) + is_album=False, # Files are not considered albums + type="file", + ) + + @classmethod + def from_path( + cls, path: Path | str, cache: Cache[str, bytes] | None = None + ) -> File: + """Create a File object from a path.""" + if isinstance(path, str): + path = Path(path) + + if not path.is_file(): + raise FileNotFoundError(f"Path `{path}` is not a file.") + + full_path = str(path.resolve()) + + return cls( + full_path=full_path, + ) + + +@cached(cache=TTLCache(maxsize=1024, ttl=900), info=True) +def path_to_folder(root_dir: Path | str, subdirs=True) -> Folder: + """Generate our nested dict structure for the specified path. + + Parameters + ---------- + root_dir : str + The root directory to start from. + subdirs : bool, optional + Whether to mark qualifying subfolders of an album as album folders themselves. If true, e.g. for `/album/CD1/track.mp3` both `/album/` and `/album/CD1/` are flagged. Defaults to True. + + Returns + ------- + dict: The nested dict structure. + """ + + return Folder.from_path(root_dir, subdirs=subdirs) + + +def album_folders_from_track_paths( + track_paths: list[Path] | list[str], use_parent_for_multidisc: bool = True +) -> list[Path]: + """Get all album folders from a list of paths to files. + + Parameters + ---------- + track_paths : list[Path] + list of track paths, e.g. mp3 files. + use_parent_for_multidisc : bool, optional + When files are in an album folder that might be a multi-disc folder (e.g. `/album/cd1`), + return the parent (`/album`) instead of the lowest-level-folder (`/cd1`). Defaults to True. + + Returns + ------- + list[str]: album folders + """ + + folders_to_check: set[Path] = set() + album_folders: set[Path] = set() + for path in track_paths: + # FIXME: For backwards compatibility, we allow a string as input + if isinstance(path, str): + path = Path(path) + + if is_archive_file(path): + album_folders.add(path.resolve()) + elif path.is_file(): + folders_to_check.add(path.parent.resolve()) + else: + # just to be nice and manage directories instead of files + folders_to_check.add(path.resolve()) + + for folder in folders_to_check: + afs = all_album_folders(folder, subdirs=True) + for af in afs: + album_folders.add(af) + + if use_parent_for_multidisc: + parents: set[Path] = set() + children: set[Path] = set() + for folder in album_folders: + if is_within_multi_dir(folder): + parents.add(folder.parent) + children.add(folder) + + album_folders = album_folders - children + album_folders = album_folders.union(parents) + + return sorted(album_folders, key=lambda s: str(s).lower()) + + +def is_album_folder(path: Path | str): + """Check if a path is an album folder. + + Returns true if the path is detected as an album by beets, or if it is an archive file. + ------- + path : Path | str + The path to check, can be a folder, file or archive. + + Note + ---- + Except in tests, we dont use this function yet. + Its logic is duplicated in `all_album_folders`. (We should consolidate.) + """ + if isinstance(path, str): + path = Path(path).absolute() + if is_archive_file(path): + return True + for paths, items in albums_in_dir(bytestring_path(path)): + if all(is_archive_file(i.decode("utf-8")) for i in items): + continue + if str(path).encode("utf-8") in paths: + return True + return False + + +def all_album_folders(root_dir: Path | str, subdirs: bool = False) -> list[Path]: + """ + Get all album folders from a given root dir. + + Parameters + ---------- + root_dir : str + toppath, highest level to start searching. + subdirs : bool, optional + Whether to return subfolders of an album that themselves would qualify. + E.g. a `CD1` folder. Defaults to False. + + Returns + ------- + list[Path] + """ + + # FIXME: For backwards compatibility, we allow a string as input + if isinstance(root_dir, str): + root_dir = Path(root_dir) + + folders: list[bytes] = [] + for paths, items in albums_in_dir(bytestring_path(root_dir.absolute())): + # Our choice on handling archives: + # - archives are always simple albums. no multi-disc logic supported, + # all discs need to be _inside_ the archive. + # - if a folder contains only archives, it will never be considered an + # album folder + # - if a folder contains a mix of archives and music files, it will be + # considered an album folder (as we think archives might be metadata or additional files e.g. cover art) + + if all(is_archive_file(i.decode("utf-8")) for i in items): + folders.extend(items) + continue + + if subdirs: + folders.extend(p for p in paths) + else: + # the top-level path is always the first in the list + # however, there is an edgecase, if we have a rogue element in a multi-disc folder: + # - artist/album/should_not_be_here.mp3 + # - artist/album/CD1/track.mp3 + # - artist/album/CD2/track.mp3 + # -> then albums_in_dir returns [album], [CD1, CD2] so that picking the first element is wrong. + # we would want all 3: album, CD1 and CD2. but in this case, the parent `album` should already + # be in our set when we check [CD1, CD2] + if os.path.dirname(paths[0]) in folders: + folders.extend(p for p in paths) + else: + folders.append(paths[0]) + + return [Path(f.decode("utf-8")) for f in folders] + + +def is_within_multi_dir(path: Path | str) -> bool: + """ + Minimal version of beets heuristic to check if a string matches a multi-disc pattern. + + E.g. "My Album CD1" or "Disc 2" will return True + """ + + # FIXME: For backwards compatibility, we allow a string as input + if isinstance(path, str): + path = Path(path) + + path_str = path.name # Use pathlib to get the basename + + for marker in MULTIDISC_MARKERS: + p = MULTIDISC_PAT_FMT.replace(b"%s", marker) + marker_pat = re.compile(p, re.I) + match = marker_pat.match(path_str.encode("utf-8")) + if match: + return True + return False + + +def _dir_size_subprocess(path: Path) -> int: + """Run ``du -sb`` and return bytes. Uncached.""" + try: + result = subprocess.run( + ["du", "-sb", str(path.resolve())], + capture_output=True, + text=True, + check=True, + ) + return int(result.stdout.split()[0]) + except Exception as e: + # this happens e.g. if the directory does not exist. + log.error(e) + return -1 + + +def _dir_files_subprocess(path: Path) -> int: + """Run ``find | wc -l`` and return count. Uncached.""" + try: + result = subprocess.run( + [f"find {str(path.resolve())} | wc -l"], + capture_output=True, + text=True, + check=True, + shell=True, + ) + return int(result.stdout) + except Exception as e: + # this happens e.g. if the directory does not exist. + log.error(e) + return -1 + + +@cached(cache=TTLCache(maxsize=1024, ttl=60), info=True) +def dir_size(path: Path) -> int: + """Size of a dir in bytes, including content. + + Cached for 60 s. Prefer :func:`get_cached_dir_stats` for hot paths — + it reads a pre-computed value from the database instead of running a + subprocess on every cache miss. + """ + return _dir_size_subprocess(path) + + +@cached(cache=TTLCache(maxsize=1024, ttl=60), info=True) +def dir_files(path: Path) -> int: + """Count the number of files in a directory. + + Cached for 60 s. Prefer :func:`get_cached_dir_stats` for hot paths. + """ + return _dir_files_subprocess(path) + + +def clear_cache(): + """Clear the directory-tree cache. + + Only clears :func:`path_to_folder` so that the inbox tree view stays + current after filesystem changes. The ``dir_size`` / ``dir_files`` + TTL caches are left alone — those are now fallbacks only; the + authoritative values live in the ``cached_stats`` DB table and are + updated by :func:`compute_and_store_dir_stats`. + """ + path_to_folder.cache.clear() # type: ignore + + +# --------------------------------------------------------------------------- +# DB-backed stats helpers +# --------------------------------------------------------------------------- + + +async def compute_and_store_dir_stats(path: Path) -> CachedStatInDb: + """Compute dir size + file count and persist to the database. + + Runs the subprocess calls in a thread so the async event loop is not + blocked. Call this from the watchdog (after debounce) and from import + workers after a successful import so that the stats endpoints always + return a pre-computed value. + + Parameters + ---------- + path: + Directory to measure. Will be resolved to an absolute path. + """ + from beets_flask.database.models.stats import CachedStatInDb + from beets_flask.database.setup import db_session_factory + + resolved = path.resolve() + size = await asyncio.to_thread(_dir_size_subprocess, resolved) + n_files = await asyncio.to_thread(_dir_files_subprocess, resolved) + + stat = CachedStatInDb(path=resolved, size_bytes=size, n_files=n_files) + with db_session_factory() as session: + session.merge(stat) + + log.info(f"Stats updated for {resolved}: {n_files} files, {size} bytes") + return stat + + +def get_cached_dir_stats(path: Path) -> CachedStatInDb | None: + """Return the most recently stored stats for *path*, or ``None``.""" + from beets_flask.database.models.stats import CachedStatInDb + from beets_flask.database.setup import db_session_factory + + with db_session_factory() as session: + row = session.get(CachedStatInDb, str(path.resolve())) + if row is not None: + # Ensure attributes are loaded while the session is still open + _ = row.size_bytes, row.n_files, row.computed_at + return row + + +__all__ = ["dir_size", "fs_item_from_path", "compute_and_store_dir_stats", "get_cached_dir_stats"] diff --git a/backend/beets_flask/importer/__init__.py b/backend/beets_flask/importer/__init__.py index 01ea95ed..65415afe 100644 --- a/backend/beets_flask/importer/__init__.py +++ b/backend/beets_flask/importer/__init__.py @@ -1,7 +1,7 @@ -from . import types -from .states import SessionState - -__all__ = [ - "SessionState", - "types", -] +from . import types +from .states import SessionState + +__all__ = [ + "SessionState", + "types", +] diff --git a/backend/beets_flask/importer/pipeline.py b/backend/beets_flask/importer/pipeline.py index 533cea41..b401f02e 100644 --- a/backend/beets_flask/importer/pipeline.py +++ b/backend/beets_flask/importer/pipeline.py @@ -1,124 +1,124 @@ -"""Beets pipeline overloads. - -Added type hints to decorators because it is annoying to have to look up -the type of the function usage. - -This is a temporary implementation. Async Generators are an antipattern -(use coroutines instead). We only use them here because the older beets pipeline -used Generators, and we still rely on parts of it. -""" - -import asyncio -from collections.abc import ( - AsyncGenerator, - AsyncIterable, - Coroutine, - Generator, - Iterable, - Sequence, -) -from typing import ( - Any, - Generic, - Literal, - TypeVar, -) - -from beets.util.pipeline import MultiMessage, _allmsgs - -# Generics for Generators -Y = TypeVar("Y") # yield -S = TypeVar("S") # send -R = TypeVar("R") # return - -# --------------------------------- Pipeline --------------------------------- # - -# yield : Task or None, send : Task, return : None -Task = TypeVar("Task", bound=Any) -Stage = ( - Generator[Task | MultiMessage | Literal["__PIPELINE_BUBBLE__"] | None, Task, R] - | AsyncGenerator[Task | None, Task] -) - - -class AsyncPipeline(Generic[Task, R]): - start_tasks: AsyncIterable[Task] - stages: list[Stage[Task, R]] - - # Original: stages = [start_task, *stages] - def __init__( - self, - start_tasks: Iterable[Task] | AsyncIterable[Task] | Task, - stages: Sequence[Stage[Task, R]] = [], - ) -> None: - if isinstance(start_tasks, Iterable): - self.start_tasks = _async_iterable_from_iterable(start_tasks) - elif isinstance(start_tasks, AsyncIterable): - self.start_tasks = start_tasks - else: - self.start_tasks = _async_iterable_from_iterable([start_tasks]) - - self.stages = list(stages) - - def add_stage(self, *stage: Stage[Task, R]) -> None: - """Add a stage to the pipeline.""" - for s in stage: - self.stages.append(s) - - async def pull_async(self) -> AsyncGenerator[Task, None]: - """Pull items through the pipeline. - - If item is coroutine, await it. - """ - # Priming -> Wait for first send i.e. right side of `task = yield` - for stage in self.stages: - await _next_resolve_async(stage) - - async for task in self.start_tasks: - msgs: list[Task] = _allmsgs(task) # returns a list of tasks - - for stage in self.stages: - next_coros: list[Coroutine] = [ - _send_resolve_async(stage, msg) for msg in msgs - ] - # override for input of next stage - msgs = [] - for out in await asyncio.gather(*next_coros): - msgs.extend(_allmsgs(out)) - - for msg in msgs: - yield msg - - async def run_async(self) -> None: - """Run the pipeline asynchronously. - - This just resolves the generator. - """ - async for _ in self.pull_async(): - # resolves the generator, and awaits each element after the previous - pass - - -async def _next_resolve_async(gen: Generator[Y, S, R] | AsyncGenerator[Y, S]): - """Call next on the generator.""" - if isinstance(gen, Generator): - return next(gen) - else: - return await anext(gen) - - -async def _send_resolve_async( - gen: Generator[Y, S, R] | AsyncGenerator[Y, S], *args, **kwargs -): - """Send to the generator.""" - if isinstance(gen, Generator): - return gen.send(*args, **kwargs) - else: - return await gen.asend(*args, **kwargs) - - -async def _async_iterable_from_iterable( - iterable: Iterable[Task], -) -> AsyncIterable[Task]: - for item in iterable: - yield item +"""Beets pipeline overloads. + +Added type hints to decorators because it is annoying to have to look up +the type of the function usage. + +This is a temporary implementation. Async Generators are an antipattern +(use coroutines instead). We only use them here because the older beets pipeline +used Generators, and we still rely on parts of it. +""" + +import asyncio +from collections.abc import ( + AsyncGenerator, + AsyncIterable, + Coroutine, + Generator, + Iterable, + Sequence, +) +from typing import ( + Any, + Generic, + Literal, + TypeVar, +) + +from beets.util.pipeline import MultiMessage, _allmsgs + +# Generics for Generators +Y = TypeVar("Y") # yield +S = TypeVar("S") # send +R = TypeVar("R") # return + +# --------------------------------- Pipeline --------------------------------- # + +# yield : Task or None, send : Task, return : None +Task = TypeVar("Task", bound=Any) +Stage = ( + Generator[Task | MultiMessage | Literal["__PIPELINE_BUBBLE__"] | None, Task, R] + | AsyncGenerator[Task | None, Task] +) + + +class AsyncPipeline(Generic[Task, R]): + start_tasks: AsyncIterable[Task] + stages: list[Stage[Task, R]] + + # Original: stages = [start_task, *stages] + def __init__( + self, + start_tasks: Iterable[Task] | AsyncIterable[Task] | Task, + stages: Sequence[Stage[Task, R]] = [], + ) -> None: + if isinstance(start_tasks, Iterable): + self.start_tasks = _async_iterable_from_iterable(start_tasks) + elif isinstance(start_tasks, AsyncIterable): + self.start_tasks = start_tasks + else: + self.start_tasks = _async_iterable_from_iterable([start_tasks]) + + self.stages = list(stages) + + def add_stage(self, *stage: Stage[Task, R]) -> None: + """Add a stage to the pipeline.""" + for s in stage: + self.stages.append(s) + + async def pull_async(self) -> AsyncGenerator[Task, None]: + """Pull items through the pipeline. + + If item is coroutine, await it. + """ + # Priming -> Wait for first send i.e. right side of `task = yield` + for stage in self.stages: + await _next_resolve_async(stage) + + async for task in self.start_tasks: + msgs: list[Task] = _allmsgs(task) # returns a list of tasks + + for stage in self.stages: + next_coros: list[Coroutine] = [ + _send_resolve_async(stage, msg) for msg in msgs + ] + # override for input of next stage + msgs = [] + for out in await asyncio.gather(*next_coros): + msgs.extend(_allmsgs(out)) + + for msg in msgs: + yield msg + + async def run_async(self) -> None: + """Run the pipeline asynchronously. + + This just resolves the generator. + """ + async for _ in self.pull_async(): + # resolves the generator, and awaits each element after the previous + pass + + +async def _next_resolve_async(gen: Generator[Y, S, R] | AsyncGenerator[Y, S]): + """Call next on the generator.""" + if isinstance(gen, Generator): + return next(gen) + else: + return await anext(gen) + + +async def _send_resolve_async( + gen: Generator[Y, S, R] | AsyncGenerator[Y, S], *args, **kwargs +): + """Send to the generator.""" + if isinstance(gen, Generator): + return gen.send(*args, **kwargs) + else: + return await gen.asend(*args, **kwargs) + + +async def _async_iterable_from_iterable( + iterable: Iterable[Task], +) -> AsyncIterable[Task]: + for item in iterable: + yield item diff --git a/backend/beets_flask/importer/progress.py b/backend/beets_flask/importer/progress.py index 7e848a3f..f625d95e 100644 --- a/backend/beets_flask/importer/progress.py +++ b/backend/beets_flask/importer/progress.py @@ -1,124 +1,124 @@ -from __future__ import annotations - -from dataclasses import dataclass -from enum import Enum -from functools import total_ordering -from typing import TypedDict - -__all__ = [ - "FolderStatus", - "Progress", - "SerializedProgressState", - "ProgressState", -] - - -@total_ordering -class Progress(Enum): - """The progress of tasks in chronological order. - - The order roughly matches the stages you might expect. - - Allows to resume a import at any time using our state dataclasses. We might - also want to add the plugin stages or refine this. - """ - - NOT_STARTED = 0 - - # PreviewSession - READING_FILES = 10 - GROUPING_ALBUMS = 11 - LOOKING_UP_CANDIDATES = 12 - IDENTIFYING_DUPLICATES = 13 - - PREVIEW_COMPLETED = 20 # dummy, only for comparison and report, has no actual stage - DELETION_COMPLETED = 21 # dummy. after a successful deletion, we can restart import - - # ImportSession - OFFERING_MATCHES = 30 - MATCH_THRESHOLD = 31 - WAITING_FOR_USER_SELECTION = 32 - EARLY_IMPORTING = 33 - IMPORTING = 34 - MANIPULATING_FILES = 35 - - IMPORT_COMPLETED = 40 # also a dummy - - # UndoSession - DELETING = 50 - - def __lt__(self, other: Progress | ProgressState) -> bool: - if isinstance(other, ProgressState): - other = other.progress - return self.value < other.value - - def __sub__(self, other: int) -> Progress: - if not isinstance(other, int): - raise TypeError("Unsupported type for addition") - - other = max(min(self.value - other, 50), 0) - return Progress(other) - - def __add__(self, other: int) -> Progress: - return self.__sub__(-1 * other) - - -class SerializedProgressState(TypedDict): - # ugly to repeat, but no way to read the type hint from enum. - progress: Progress - message: str | None - plugin_name: str | None - - -@total_ordering -@dataclass(slots=True) -class ProgressState: - """Simple dataclass to hold a status message and a status code.""" - - progress: Progress = Progress.NOT_STARTED - - # Optional message to display to the user - message: str | None = None - - # Plugin specific - plugin_name: str | None = None - - def serialize(self) -> SerializedProgressState: - return SerializedProgressState( - progress=self.progress, - message=self.message, - plugin_name=self.plugin_name, - ) - - def __lt__(self, other: ProgressState | Progress) -> bool: - if isinstance(other, Progress): - other = ProgressState(other) - return self.progress < other.progress - - def __eq__(self, other: object) -> bool: - if isinstance(other, Progress): - return self.progress == other - if not isinstance(other, ProgressState): - return False - return self.progress == other.progress - - -class FolderStatus(int, Enum): - """The status of a folder. - - Order does not matter, but we need to be able to check equality - """ - - UNKNOWN = -2 - FAILED = -1 - NOT_STARTED = 0 - PENDING = 1 - PREVIEWING = 2 - PREVIEWED = 3 - IMPORTING = 4 - IMPORTED = 5 - DELETING = 6 - DELETED = 7 - - def __str__(self) -> str: - return self.name.lower() +from __future__ import annotations + +from dataclasses import dataclass +from enum import Enum +from functools import total_ordering +from typing import TypedDict + +__all__ = [ + "FolderStatus", + "Progress", + "SerializedProgressState", + "ProgressState", +] + + +@total_ordering +class Progress(Enum): + """The progress of tasks in chronological order. + + The order roughly matches the stages you might expect. + + Allows to resume a import at any time using our state dataclasses. We might + also want to add the plugin stages or refine this. + """ + + NOT_STARTED = 0 + + # PreviewSession + READING_FILES = 10 + GROUPING_ALBUMS = 11 + LOOKING_UP_CANDIDATES = 12 + IDENTIFYING_DUPLICATES = 13 + + PREVIEW_COMPLETED = 20 # dummy, only for comparison and report, has no actual stage + DELETION_COMPLETED = 21 # dummy. after a successful deletion, we can restart import + + # ImportSession + OFFERING_MATCHES = 30 + MATCH_THRESHOLD = 31 + WAITING_FOR_USER_SELECTION = 32 + EARLY_IMPORTING = 33 + IMPORTING = 34 + MANIPULATING_FILES = 35 + + IMPORT_COMPLETED = 40 # also a dummy + + # UndoSession + DELETING = 50 + + def __lt__(self, other: Progress | ProgressState) -> bool: + if isinstance(other, ProgressState): + other = other.progress + return self.value < other.value + + def __sub__(self, other: int) -> Progress: + if not isinstance(other, int): + raise TypeError("Unsupported type for addition") + + other = max(min(self.value - other, 50), 0) + return Progress(other) + + def __add__(self, other: int) -> Progress: + return self.__sub__(-1 * other) + + +class SerializedProgressState(TypedDict): + # ugly to repeat, but no way to read the type hint from enum. + progress: Progress + message: str | None + plugin_name: str | None + + +@total_ordering +@dataclass(slots=True) +class ProgressState: + """Simple dataclass to hold a status message and a status code.""" + + progress: Progress = Progress.NOT_STARTED + + # Optional message to display to the user + message: str | None = None + + # Plugin specific + plugin_name: str | None = None + + def serialize(self) -> SerializedProgressState: + return SerializedProgressState( + progress=self.progress, + message=self.message, + plugin_name=self.plugin_name, + ) + + def __lt__(self, other: ProgressState | Progress) -> bool: + if isinstance(other, Progress): + other = ProgressState(other) + return self.progress < other.progress + + def __eq__(self, other: object) -> bool: + if isinstance(other, Progress): + return self.progress == other + if not isinstance(other, ProgressState): + return False + return self.progress == other.progress + + +class FolderStatus(int, Enum): + """The status of a folder. + + Order does not matter, but we need to be able to check equality + """ + + UNKNOWN = -2 + FAILED = -1 + NOT_STARTED = 0 + PENDING = 1 + PREVIEWING = 2 + PREVIEWED = 3 + IMPORTING = 4 + IMPORTED = 5 + DELETING = 6 + DELETED = 7 + + def __str__(self) -> str: + return self.name.lower() diff --git a/backend/beets_flask/importer/session.py b/backend/beets_flask/importer/session.py index 8cad55ac..ec1edf2b 100644 --- a/backend/beets_flask/importer/session.py +++ b/backend/beets_flask/importer/session.py @@ -1,1142 +1,1142 @@ -""" -Session classes for the import pipeline. - -Sessions often take particular arguments, such as a duplicate action. In the simplest and most common case, -each session has one task (i.e. one album) to deal with. Sometimes, however, one session may have multiple tasks, -such as when one folder contains files from two albums. - -To account for this, we use the TaskMapping type. -They contain an action to take for each task (mapping a task_id as string to the action), and the default value -must be None (which means that the session uses the default action for that task, loaded from user config). - -When no mapping is given, the default action is used for all tasks. - -``` -TaskIdMappingArg = defaultdict[str, T | None] | None -``` - -If you want to pass a value to all tasks, you can omit looking up the task ids and -instead use "*" as the key, which will apply the action to all tasks of the session. - -``` -action_for_all : TaskIdMappingArg[DuplicateAction] = {"*": "remove"} -``` -""" - -from __future__ import annotations - -import asyncio -import logging -from abc import ABC, abstractmethod -from collections import defaultdict -from collections.abc import Callable -from copy import deepcopy -from enum import Enum -from pathlib import Path -from typing import Any, Literal, TypedDict, TypeGuard, TypeVar - -import nest_asyncio -from beets import autotag, importer, plugins -from beets.ui import UserError, _open_library -from beets.util import bytestring_path -from deprecated import deprecated - -from beets_flask.config import get_config -from beets_flask.disk import is_archive_file -from beets_flask.importer.progress import Progress, ProgressState -from beets_flask.importer.types import ( - BeetsAlbum, - BeetsLibrary, - DuplicateAction, -) -from beets_flask.logger import log -from beets_flask.server.exceptions import ( - ApiException, - DuplicateException, - IntegrityException, - NoCandidatesFoundException, - NotImportedException, - to_serialized_exception, -) -from beets_flask.utility import capture_stdout_stderr - -from .pipeline import AsyncPipeline -from .stages import ( - StageOrder, - finalize, - group_albums, - identify_duplicates, - lookup_candidates, - manipulate_files, - match_threshold, - plugin_stage, - read_tasks, - user_query, -) -from .states import ProgressState, SessionState - -nest_asyncio.apply() - -# ---------------------------------------------------------------------------- # -# Types and helpers # -# ---------------------------------------------------------------------------- # - -T = TypeVar("T") - -TaskIdMapping = defaultdict[str, T] -TaskIdMappingArg = dict[str, T | None] | None - - -def parse_task_id_mapping(mapping: TaskIdMappingArg[T], default: T) -> TaskIdMapping[T]: - """ - Convert the flexible arguments to stricter TaskIdMapping that sessions use internally. - - Parameters - ---------- - mapping : TaskIdMappingArg - For each task_id (key) which action to take (value). - If None, the default action is used for all tasks. - If "*" is used as key, this action is used for all tasks, and only one key-value - pair is allowed. - default : T - Default value to use for all tasks that are not in the mapping, or "*". - - - Note - ---- - TaskIdMappings are defaultdicts, which keeps the lower level logic simpler. - TaskIdMappingsArgs are just dicts, which are serializable trhough api and redis - thread bounds. - """ - - m: TaskIdMapping[T] = defaultdict(lambda: default) - if mapping is not None: - if "*" in mapping.keys(): - if len(mapping) > 1: - raise ValueError( - "If you use '*' as key, you cannot use any other keys in the mapping." - ) - else: - return defaultdict(lambda: mapping["*"] or default) - - for k, v in mapping.items(): - if v is None: - continue - m[k] = v - - return m - - -class CandidateChoiceFallback(Enum): - """ - Type for the candidate choice. - - Candidate Choices are either a string (the candidate id) or a special case - (asis candidate, or the best one). - """ - - ASIS = 1 - BEST = 2 - - -CandidateChoice = str | CandidateChoiceFallback - - -class Search(TypedDict): - """Search for a candidate. - - This is used to search for a candidate in the preview session. - """ - - search_ids: list[str] - search_artist: str | None - search_album: str | None - - -def _is_search(d: Any) -> TypeGuard[Search]: - """Check if the given dict is a Search object.""" - return ( - d is not None - and isinstance(d, dict) - and "search_ids" in d - and isinstance(d["search_ids"], list) - ) - - -# ---------------------------------------------------------------------------- # -# Sessions # -# ---------------------------------------------------------------------------- # - - -class BaseSession(importer.ImportSession, ABC): - """Base class for our GUI-based ImportSessions. - - Operates on single Albums / files. - - Parameters - ---------- - path : list[str] - list of album folders to import - config_overlay : str or dict - path to a config file to overlay on top of the default config. - Note that if `dict`, the lazyconfig notation e.g. `{import.default_action: skip}` - wont work reliably. Better nest the dicts: `{import: {default_action: skip}}` - - Note: It's a design choice to require that you manually create and pass the - `SessionState` object. Usually the states go into the database, which needs explizit - handling beyond the session. - """ - - # attributes needed to create a beetsTag instance for our database - # are contained in the associated SessionState -> TaskState -> CandidateStates - state: SessionState - - pipeline: AsyncPipeline[importer.ImportTask, Any] | None = None - config_overlay: dict - - # FIXME: only for typehint until we update beets - lib: BeetsLibrary # type: ignore - - def __init__( - self, - state: SessionState, - config_overlay: dict | None = None, - ): - if not state.path.exists(): - raise FileNotFoundError(f"Path {state.path} does not exist.") - if state.path.is_file() and not is_archive_file(state.path): - raise ValueError( - f"Path {state.path} is not an archive file. " - + "Importing singletons is not supported yet." - ) - - # FIXME: This is a super bad convention of the original beets. - # We do not want to pollute a global config object every time a session runs. - # This is fine for the cli tool, where each run creates only one session - # but not for our long-running webserver. - config = get_config() - if isinstance(config_overlay, dict): - config.set_args(config_overlay) - - self.config_overlay = config_overlay or {} - self.state = state - - super().__init__( - lib=_open_library(config), - paths=[bytestring_path(state.path)], - query=None, - loghandler=None, - ) - # Hacky workaround to use our logging, to allow plugins to communicate - self.logger.handlers = log.handlers - log.debug(f"Created new {self.__class__.__name__} for {state.path}") - - @property - def path(self) -> Path: - return self.state.path - - @deprecated - def run_and_capture_output(self) -> tuple[str, str]: - """Run the import session and capture the output. - - Uses the original beets import session run method, - with lots of overhead. - Sets self.preivew to output and error messages occuring during run. - - Returns - ------- - tuple[str, str]: out, err - """ - self.logger.debug(f"{self.paths}") - out, err, _ = capture_stdout_stderr(self.run) - self.preview = out + "\n\n" + err if err else out - return out, err - - def get_config_value(self, key: str, type_func: Callable | None = None) -> Any: - """Get a config value from the overlay or default. - - Use dots to separate levels. - """ - - path = key.split(".") - - overlay = self.config_overlay - for p in path: - overlay = overlay.get(p, {}) - - # overlay takes priority - if not isinstance(overlay, dict): - return type_func(overlay) if type_func else overlay - - # get settings from user settings, this is not a dict, but confuse config - # the confuse config views do not throw key errors, and their .get() is not - # the same as dict.get(), but rather resolves the value. - default = get_config() - for p in path: - default = default[p] - default = default.get(type_func) if type_func else default.get() - return default - - # -------------------------- State handling helpers -------------------------- # - - def set_task_progress( - self, task: importer.ImportTask, progress: ProgressState | Progress | str - ): - """Set the progress for a task belonging to the session. - - If string is given it is set as the message of the current progress. - Note: currently we only implement status on the level of the whole import session, - but should eventually do this per selection (task). - """ - - task_state = self.state.get_task_state_for_task_raise(task) - - task_state.set_progress(progress) - - def get_task_progress(self, task: importer.ImportTask) -> ProgressState | None: - """Get the progress of the task, via this sessions state.""" - task_state = self.state.get_task_state_for_task(task) - return task_state.progress if task_state else None - - # -------------------------------- Stages -------------------------------- # - - @property - @abstractmethod - def stages(self) -> StageOrder: - """Set the stages for the session. - - In Subclasses, define the order of stages here. - """ - raise NotImplementedError("Implement in subclass") - - def resolve_duplicate(self, task: importer.ImportTask, found_duplicates): - """Overload default resolve duplicate and skip it. - - This basically skips this stage. - """ - self.logger.warning( - "Skipping duplicate resolution. " - + f"Your session should implement this! -> {self.__class__.__name__}" - ) - task.set_choice(importer.Action.SKIP) - - def choose_item(self, task: importer.ImportTask): - """Overload default choose item and skip it. - - This session should not reach this stage. - """ - self.logger.debug(f"skipping choose_item {task}") - return importer.Action.SKIP - - def should_resume(self, path): - """Overload default should_resume and skip it. - - Should normally be no problem if the config is set correctly, but just - in case. - """ - self.logger.debug(f"skipping should_resume {path}") - return False - - def identify_duplicates(self, task: importer.ImportTask): - """For all candidates, check if they have duplicates in the library. - - This stage should only be run for preview sessions, but we still have - some old code in stages.py/user_query(). - """ - raise NotImplementedError( - f"This session should not reach this stage. {self.__class__.__name__}" - ) - - def lookup_candidates(self, task: importer.ImportTask): - """Lookup candidates for the task. - - This stage should only be run for preview sessions, but we still have - some old code in stages.py/user_query(). - """ - raise NotImplementedError( - f"This session should not reach this stage. {self.__class__.__name__}" - ) - - def finalize(self, task: importer.ImportTask): - """Last stage called and customizable any session.""" - self.logger.debug(f"Finalized {self} {task}") - - # ---------------------------------- Run --------------------------------- # - - def run_sync(self) -> SessionState: - """Run the import session synchronously.""" - return asyncio.run(self.run_async()) - - async def run_async(self) -> SessionState: - """Run the import session asynchronously. - - Does not set tasks to completed at the end. - Take care of this in subclasses. - """ - # For now, until we improve the upstream beets config logic, - # adhere to importer.ImportSession convention and create a local copy - # of the config. - config = get_config() - self.set_config(config["import"]) - - # TODO: check some config values. that are not compatible with our code. - self.pipeline = AsyncPipeline(start_tasks=read_tasks(self)) - - for s in self.stages.values(): - self.pipeline.add_stage(s) - - log.info(f"Running {self.__class__.__name__} on state<{self.state.id=}>.") - log.debug(f"Running {len(self.pipeline.stages)} stages.") - - # reset exception state - # TODO: To clear or not to clear, - # exception hierarchy and own table for exceptions/warnings - # self.state.exc = None - plugins.send("import_begin", session=self) - try: - assert self.pipeline is not None - await self.pipeline.run_async() - except importer.ImportAbortError: - log.debug(f"Interactive import session aborted by user") - except ApiException as e: - if e.persist_in_db: - log.debug(f"Persisting exception {e} in session state") - self.state.exc = to_serialized_exception(e) - raise e - except Exception as e: - self.state.exc = to_serialized_exception(e) - raise e - - log.info(f"Completed {self.__class__.__name__} on state<{self.state.id=}>.") - return self.state - - -class PreviewSession(BaseSession): - """Preview what would be imported. Only fetches candidates.""" - - group_albums: bool | None - autotag: bool | None - - def __init__( - self, - state: SessionState, - config_overlay: dict | None = None, - group_albums: bool | None = None, - autotag: bool | None = None, - **kwargs, - ): - """ - Create new PreviewSession. - - Parameters - ---------- - group_albums : bool | None - Whether to create multple tasks, one for each album found in the metadata - of the files. Set to true if you have multiple albums in a single folder. - If None: get value from beets config. - autotag : bool | None - Whether to look up metadata online. If None: get value from beets config. - """ - - super().__init__(state, config_overlay, **kwargs) - self.group_albums = group_albums - self.autotag = autotag - - # -------------------------------- Stages -------------------------------- # - - @property - def stages(self) -> StageOrder: - stages = StageOrder() - - if self.get_config_value("import.singletons"): - # beets tweaks the album grouping settings via overlay for singletons. - raise NotImplementedError("Singletons not implemented yet.") - - if self.group_albums or ( - self.group_albums is None and self.get_config_value("import.group_albums") - ): - stages.append(group_albums(self)) - - if self.autotag or ( - self.autotag is None and self.get_config_value("import.autotag") - ): - stages.append(lookup_candidates(self)) - - stages.append(identify_duplicates(self)) - stages.append(finalize(self)) - - return stages - - # --------------------------- Stage Definitions -------------------------- # - - def identify_duplicates(self, task: importer.ImportTask): - """For all candidates, check if they have duplicates in the library.""" - task_state = self.state.get_task_state_for_task_raise(task) - - for idx, cs in enumerate( - task_state.candidate_states + [task_state.asis_candidate] - ): - # This is a mutable operation i.e. candidate state is modfied here! - duplicates = cs.identify_duplicates(self.lib) - - if len(duplicates) > 0: - log.debug(f"Found duplicates for {cs.id=}: {duplicates}") - - def lookup_candidates(self, task: importer.ImportTask): - """Lookup candidates for the task.""" - - search_ids = self.config["search_ids"].as_str_seq() # might be an empty list - log.debug( - f"Looking up candidates for {task.paths}, using search ids {search_ids}" - ) - - task.lookup_candidates(search_ids) - - if len(task.candidates) == 0: - raise NoCandidatesFoundException(persist_in_db=True) - - # Update our state - task_state = self.state.get_task_state_for_task_raise(task) - task_state.add_candidates(task.candidates) - - -class AddCandidatesSession(PreviewSession): - """ - Preview session that adds a candidate to the ones already fetched. - - Can only run on a session state of a preview session that already has - candidates. - """ - - search: TaskIdMapping[Search | Literal["skip"]] - initial_task_states: dict[str, ProgressState] - - def __init__( - self, - state: SessionState, - config_overlay: dict | None = None, - search: TaskIdMappingArg[Search | Literal["skip"]] = None, - **kwargs, - ): - super().__init__(state, config_overlay, **kwargs) - - # None means skip search for this task - self.search = parse_task_id_mapping(search, "skip") - self.initial_task_states = {} - - for task in self.state.task_states: - self.initial_task_states[task.id] = deepcopy(task.progress) - if task.progress >= Progress.PREVIEW_COMPLETED: - # Reset task progress only for tasks that have search values - # other tasks are skipped - # This should only be relevant for searches on multiple tasks - # (i.e. folders) - s = self.search[task.id] - if s != "skip": - task.set_progress(Progress.LOOKING_UP_CANDIDATES - 1) - - def lookup_candidates(self, task: importer.ImportTask): - """Amend the found candidate to the already existing candidates (if any).""" - # see ref in lookup_candidates in beets/importer.py - - task_state = self.state.get_task_state_for_task_raise(task) - search = self.search[task_state.id] - - if search == "skip": - log.debug(f"Skipping search for {task_state.id=}") - return - - # Beets treats empty strings like real strings, which is not what we want here. - # TODO: revisit once PR merged upstream: https://github.com/beetbox/beets/pull/6117 - if ( - search["search_artist"] is not None - and search["search_artist"].strip() == "" - ): - search["search_artist"] = None - if search["search_album"] is not None and search["search_album"].strip() == "": - search["search_album"] = None - search["search_ids"] = list( - filter(lambda x: x.strip() != "", search["search_ids"]) - ) - - log.debug(f"Using {search=} for {task_state.id=}, {task_state.paths=}") - - try: - _, _, prop = autotag.tag_album( - task.items, - search_ids=search["search_ids"], - search_album=search["search_album"], - search_artist=search["search_artist"], - ) - except Exception as e: - # TODO: With beets 2.6.0 this should be revisited - # since beets should than be able to handle these exceptions - # gracefully upstream. - # https://github.com/beetbox/beets/pull/5965 - from beetsplug.musicbrainz import MusicBrainzAPIError - from beetsplug.spotify import APIError as SpotifyAPIError - - if isinstance(e, MusicBrainzAPIError): - raise NoCandidatesFoundException( - f"Failed to contact Musicbrainz API: {e.get_message()}", - persist_in_db=False, - ) - elif isinstance(e, SpotifyAPIError): - raise NoCandidatesFoundException( - f"Failed to contact Spotify API: {e}", - persist_in_db=False, - ) - else: - raise NoCandidatesFoundException( - f"Failed to contact online APIs.", - persist_in_db=False, - ) - - task_state.add_candidates(prop.candidates) - - # Update quality of best candidate, likely not needed for us, only beets cli. - task.rec = max(prop.recommendation, task.rec or autotag.Recommendation.none) - - if len(prop.candidates) == 0: - error_text = "Search found no candidates via " - if search["search_ids"]: - error_text += f"ids: {', '.join(search['search_ids'])}; " - if search["search_artist"]: - error_text += f"artist: {search['search_artist']}; " - if search["search_album"]: - error_text += f"album: {search['search_album']}; " - error_text += NoCandidatesFoundException.metadata_plugin_info() - raise NoCandidatesFoundException( - error_text, - persist_in_db=False, - ) - # Hack: Clear exception if we found new candidates - elif ( - self.state.exc is not None - and self.state.exc["type"] == "NoCandidatesFoundException" - ): - log.debug( - "Clearing previous NoCandidatesFoundException after finding candidates via search." - ) - self.state.exc = None - - def finalize(self, task: importer.ImportTask): - """Restore initial taks and session states.""" - - task_state = self.state.get_task_state_for_task_raise(task) - if task_state.id in self.initial_task_states: - task_state.set_progress(self.initial_task_states[task_state.id]) - else: - log.warning( - f"Task {task_state.id} not in initial task states. " - + "Cannot restore previous progress." - ) - - self.logger.debug(f"Finalized {self} {task}") - - -class ImportSession(BaseSession): - """ - Import session that assumes we already have a match-id. - - Needs to run from an already finished Preview Session. - """ - - candidate_ids: TaskIdMapping[CandidateChoice] - duplicate_actions: TaskIdMapping[DuplicateAction] - - def __init__( - self, - state: SessionState, - config_overlay: dict | None = None, - candidate_ids: TaskIdMappingArg[CandidateChoice] = None, - duplicate_actions: TaskIdMappingArg[DuplicateAction] = None, - ): - """Create new ImportSession. - - Parameters - ---------- - candidate_ids : optional - Either id of candidate(s) or the import choice. This is used to determine which - candidate to import. If a dict is given, the keys are the task ids and the - values are the candidate ids. You can also use the import choice enum - `ImportChoice.ASIS` or `ImportChoice.BEST` to indicate that you want to - import the candidate as-is or the best candidate. - FIXME: at the moment asis is broken - duplicate_actions : str - The action to take if duplicates are found. One of "skip", "keep", - "remove", "merge", "ask". If None, the default is read from - the user config and applied to all tasks. - """ - - config_overlay = {} if config_overlay is None else config_overlay - if config_overlay.get("import", {}).get("search_ids") is not None: - raise ValueError("search_ids set in config_overlay. This is not supported.") - - super().__init__(state, config_overlay) - - # Create a mapping for the duplicate action - # each task might have a different action. - # if none is given the default action is used from the config - default_action: DuplicateAction = self.get_config_value( - "import.duplicate_action", str - ) - self.duplicate_actions = parse_task_id_mapping( - duplicate_actions, default_action - ) - - # For candidates, None means to take best - self.candidate_ids = parse_task_id_mapping( - candidate_ids, CandidateChoiceFallback.BEST - ) - - async def run_async(self) -> SessionState: - # only allow import sessions to run on preview states (not other import states) - if self.state.progress == Progress.IMPORT_COMPLETED: - log.error( - f"Cannot run {self.__class__.__name__} from states that already " - + f"completed an import. (i.e. other imports) [{self.state.progress}]" - ) - e = UserError("Cannot redo imports. Try undo and/or retag!") - self.state.exc = to_serialized_exception(e) - raise e - elif self.state.progress > Progress.PREVIEW_COMPLETED: - log.warning( - f"Resetting state from {self.state.progress} to PREVIEW_COMPLETED for " - + f"import session {self.state.id}." - ) - for task in self.state.task_states: - task.set_progress(Progress.PREVIEW_COMPLETED) - - state = await super().run_async() - - # Trigger cli_exit event to have compatibility with beets plugin that - # execute after the import is completed. - plugins.send("cli_exit", lib=self.lib) - return state - - # ------------------------------ Stages ------------------------------ # - - @property - def stages(self): - stages = StageOrder() - - # FIXME: user_query calls task.choose_match, which calls session.choose_match. - # Better abstraction needed upstream. - stages.append(user_query(self)) - - # Early import stages - plugs = plugins.find_plugins() - for p in plugs: - for stage in p.get_early_import_stages(): - stages.append( - plugin_stage( - self, - stage, - ProgressState( - Progress.EARLY_IMPORTING, - plugin_name=stage.__name__, - ), - ), - name=f"early_plugin_stage_{p.__class__.__name__}_{stage.__name__}", - ) - - # Import stages - for p in plugs: - for stage in p.get_import_stages(): - stages.append( - plugin_stage( - self, - stage, - ProgressState( - Progress.IMPORTING, - plugin_name=stage.__name__, - ), - ), - name=f"plugin_stage_{p.__class__.__name__}_{stage.__name__}", - ) - - # finally, move files - stages.append(manipulate_files(self)) - stages.append(finalize(self)) - - return stages - - def finalize(self, task: importer.ImportTask): - """ - Reset previous match threshold exceptions. - - Needed because we might run a normal ImportSession manually after - the AutoImportSession. The AutoImportSession needs to keep an Exception in its - status to inform the frontend, which we need to clear here. - (No need to override `finalize` in AutoImportSession, despite it inherriting - from here, because it raises when below threshold). - """ - if ( - self.state.exc is not None - and self.state.exc["type"] == "NotImportedException" - and self.state.exc["message"].startswith("Match below threshold") - ): - log.debug( - "Clearing previous MatchThresholdException after successful import." - ) - self.state.exc = None - super().finalize(task) - - # --------------------------- Stage Definitions -------------------------- # - - def choose_match(self, task: importer.ImportTask): - self.logger.setLevel(logging.DEBUG) - self.logger.debug(f"choose_match {task}") - - task_state = self.state.get_task_state_for_task_raise(task) - - # Pick the candidate to import - candidate_id = self.candidate_ids[task_state.id] - - if isinstance(candidate_id, str): - candidate_state = task_state.get_candidate_state_by_id(candidate_id) - if candidate_state is None: - raise ValueError(f"Candidate with id {candidate_id} not found.") - elif candidate_id == CandidateChoiceFallback.BEST: - candidate_state = task_state.best_candidate_state - if candidate_state is None: - raise ValueError(f"No candidate found.") - elif candidate_id == CandidateChoiceFallback.ASIS: - candidate_state = task_state.asis_candidate - else: - raise NotImplementedError("ImportChoice.ASIS not implemented yet.") - - # update task_state to keep track of the choice in the database - task_state.chosen_candidate_state_id = candidate_state.id - log.debug( - f"Setting chosen candidate for task {task_state.id} to {candidate_state.id}" - ) - - # Let plugins display info - results = plugins.send("import_task_before_choice", session=self, task=task) - actions = [action for action in results if action] - - if len(actions) > 0: - # decide if we can just move past this and ignore the plugins - raise UserError( - f"Plugins returned actions, which is not supported for {self.__class__.__name__}" - ) - - # ASIS - if candidate_state.id == task_state.asis_candidate.id: - log.debug(f"Importing {task} as-is") - return importer.Action.ASIS - - return candidate_state.match - - def resolve_duplicate( - self, task: importer.ImportTask, found_duplicates: list[BeetsAlbum] - ): - log.debug( - f"Resolving duplicates for {task} with action {self.duplicate_actions}" - ) - - if len(found_duplicates) == 0: - log.debug(f"No duplicates found for") - return - - task_state = self.state.get_task_state_for_task_raise(task) - task_duplicate_action = self.duplicate_actions[task_state.id] - task_state.duplicate_action = task_duplicate_action - match task_duplicate_action: - case "skip": - task.set_choice(importer.Action.SKIP) - case "keep": - pass - case "remove": - task.should_remove_duplicates = True - case "merge": - task.should_merge_duplicates = True - case "ask": - # task.set_choice(importer.action.SKIP) - raise DuplicateException( - "You have set the duplicate action to 'ask' in your beets config." - ) - case _: - raise DuplicateException( - f"Unknown duplicate action: {self.duplicate_actions}" - ) - - -class BootlegImportSession(ImportSession): - """ - Import session to import without modifying metadata. - - No preview session required. - - Essentially `beet import --group-albums -A`, ideal for bootlegs and - just getting a folder into your library where you are sure the metadata is correct. - """ - - def __init__( - self, - state, - config_overlay: dict | None = None, - **kwargs, - ): - """Create new ImportAsIsSession. - - Parameters - ---------- - state: SessionState - Preconfigured state of the import session. - config_overlay : dict - Configuration to overlay on top of the default config. - "import.group_albums", "import.autotag" and "import.search_ids" are ignored. - **kwargs - See `ImportSession`. - """ - - config_overlay = {} if config_overlay is None else deepcopy(config_overlay) - import_overlay = config_overlay.get("import", {}) - - if "group_albums" in import_overlay and import_overlay["group_albums"] is False: - log.warning("Overwriting 'group_albums' in config_overlay.") - if "autotag" in import_overlay and import_overlay["autotag"] is True: - log.warning("Overwriting 'autotag' in config_overlay.") - if "search_ids" in import_overlay: - log.warning("Overwriting 'search_ids' in config_overlay.") - - import_overlay["group_albums"] = True - import_overlay["autotag"] = False - import_overlay["search_ids"] = None - - config_overlay["import"] = import_overlay - - super().__init__(state, config_overlay, **kwargs) - - # overwrite the default action for all tasks - self.candidate_ids = parse_task_id_mapping( - {"*": CandidateChoiceFallback.ASIS}, - CandidateChoiceFallback.ASIS, - ) - - @property - def stages(self): - stages = super().stages - stages.insert(before="user_query", stage=group_albums(self)) - return stages - - -class AutoImportSession(ImportSession): - """Generate a preview and import if the best match is good enough. - - Preview generation is skipped if the provided session state already has a preview. - - Wether the import is triggered depends on the specified `import_threshold`, or - the beets-config setting `match.strong_rec_thresh`. - The match quality is calculated via penalties, thus it ranges from 0 to 1, but a - perfect match is at 0. The same convention is used for thresholds. - - The default threshold is 0.04, so that a "96% match or better" will be imported. - - Raises a `NotImportedException` if the match quality is worse than the threshold, - stopping the pipeline. - """ - - import_threshold: float - - def __init__( - self, - state, - config_overlay: dict | None = None, - import_threshold: float | None = None, - **kwargs, - ): - """Create new AutoImportSession. - - Parameters - ---------- - state: SessionState - Preconfigured state of the import session. - config_overlay : optional, dict - Configuration to overlay on top of the default config. - import_threshold: optional, float - 0 to import only perfect matches, 1 to import everything. Default is 0.04. - **kwargs - See `ImportSession`. - """ - - super().__init__(state, config_overlay, **kwargs) - - if import_threshold is None: - self.import_threshold = self.get_config_value( - "match.strong_rec_thresh", float - ) - else: - self.import_threshold = import_threshold - - @property - def stages(self): - stages = super().stages - stages.insert(before="user_query", stage=match_threshold(self)) - return stages - - def match_threshold(self, task: importer.ImportTask): - """Check if the match quality is good enough to import. - - Returns true if candidates were found, and the match quality is better than - threshlold. - - Note: What stops the pipeline is that we set task.choice to importer.action.SKIP, - or raise an exception. - - Currently raising, as we do not have a dedicated progress for "not imported". - - FIXME: Instead of adding a whole new stage, with progress and a session function, - we could simply extend the choose_match function. It's defined for the normal - import session, and gets called early in user_query. - (see #78) - """ - try: - task_state = self.state.get_task_state_for_task(task) - distance = float(task_state.best_candidate_state.distance) # type: ignore - except (AttributeError, TypeError): - distance = 2.0 - - if len(task.candidates) == 0: - raise NoCandidatesFoundException() - - if distance > self.import_threshold: - log.debug( - f"Best candidate was worse than threshold {distance=} {self.import_threshold=}" - ) - d = (1 - distance) * 100 - t = (1 - self.import_threshold) * 100 - raise NotImportedException(f"Match below threshold ({d:.0f}% < {t:.0f}%)") - # beets would handle this via the task action: - task.set_choice(importer.action.SKIP) - else: - log.info( - f"Best candidate was better than threshold, importing to library. {distance=} {self.import_threshold=}" - ) - - -class UndoSession(BaseSession): - delete_files: bool - - def __init__( - self, - state: SessionState, - config_overlay: dict | None = None, - delete_files: bool = False, - **kwargs, - ): - super().__init__(state, config_overlay, **kwargs) - self.delete_files = delete_files - - async def run_async(self) -> SessionState: - """Undo an import. - - A bit of a hack to reuse the BaseSession as we do - not operate on tasks here but makes things easier in - calling this in the invoker. - - Note: this Session should only be run in the (single-thread) import queue, - because we want to limit beets-library and file interactions to be serial - to avoid conflicts. - """ - - if self.state.progress != Progress.IMPORT_COMPLETED: - log.error( - f"Cannot undo import from state {self.state.progress}. " - + "Only imports can be undone." - ) - e = UserError( - "Cannot undo if never imported! You need to import to undo first." - ) - self.state.exc = to_serialized_exception(e) - raise e - - for t_state in self.state.task_states: - t_state.set_progress(Progress.DELETING) - - # Revert the movement of items that beets does during manipulate_files() - # TODO: support Move operations (currently we only allow copy) - - # HACK: To allow an reimport after an undo, we need to use the old - # paths of the items. This is a bit hacky, but We did not find - # a better way to do this while maintaining the original beets logic. - # -> the old_paths attribute of a beets_task is set during task.manipulate_files() - # Unfortunately, here task.imported_items() does not work yet (.match not set) - - for t_state in self.state.task_states: - chosen_candidate = t_state.chosen_candidate_state - if chosen_candidate is None: - raise ValueError("No chosen candidate found for task.") - if t_state.task.old_paths is None: - raise ValueError("No old paths found for task.") - for idx, item in enumerate(chosen_candidate.match.mapping.keys()): # type: ignore - # yes, keys are the items and need to be updated! - # mapping maps objects to objects. - item.path = t_state.task.old_paths[idx] - - # Delete files - excs: list[Exception] = [] - pths: list[Path] = [] - for t_state in self.state.task_states: - try: - delete_from_beets(t_state.id, self.delete_files, self.lib) - except Exception as e: - items = self.lib.items(f"gui_import_id:{t_state.id}") - - excs.append(e) - pths.extend([Path(item.path.decode("utf-8")) for item in items]) - - if len(excs) > 0: - for exc in excs: - log.exception(exc) - - # FIXME: Multi/array exceptions need handling - self.state.exc = to_serialized_exception(excs[0]) - raise IntegrityException( - "Could not delete all items. Some items might be left in the library. " - + f"Problematic files were: {pths}" - ) - - # Update our state and progress - for t_state in self.state.task_states: - t_state.set_progress(Progress.DELETION_COMPLETED) - - # Send cli_exit event to have compatibility with beets plugin that - # execute after the import is completed. - plugins.send("cli_exit", lib=self.lib) - - return self.state - - @property - def stages(self): - # This tweaked session skips the pipeline - return StageOrder() - - -## Edge cases -# 1. Session doesn't exist anymore -> delete -# 2. Session exists but multiple tasks -> undo (session) -# 3. Session exists but file do not anymore in import folder -> undo, warnings - -# Flow: -# - remove from beets db -# - revert our session (if possible) - - -def delete_from_beets(task_id: str, delete_files: bool, lib: BeetsLibrary): - """Low-level, delete the items from the beets library.""" - - # We set a gui_import_id in the beets database this is equal to the session id - # see _apply_choice in stages.py - - items = lib.items(f"gui_import_id:{task_id}") - - if len(items) == 0: - raise ValueError("No items found that match this import session id.") - - with lib.transaction(): - for item in items: - item.remove(delete_files) +""" +Session classes for the import pipeline. + +Sessions often take particular arguments, such as a duplicate action. In the simplest and most common case, +each session has one task (i.e. one album) to deal with. Sometimes, however, one session may have multiple tasks, +such as when one folder contains files from two albums. + +To account for this, we use the TaskMapping type. +They contain an action to take for each task (mapping a task_id as string to the action), and the default value +must be None (which means that the session uses the default action for that task, loaded from user config). + +When no mapping is given, the default action is used for all tasks. + +``` +TaskIdMappingArg = defaultdict[str, T | None] | None +``` + +If you want to pass a value to all tasks, you can omit looking up the task ids and +instead use "*" as the key, which will apply the action to all tasks of the session. + +``` +action_for_all : TaskIdMappingArg[DuplicateAction] = {"*": "remove"} +``` +""" + +from __future__ import annotations + +import asyncio +import logging +from abc import ABC, abstractmethod +from collections import defaultdict +from collections.abc import Callable +from copy import deepcopy +from enum import Enum +from pathlib import Path +from typing import Any, Literal, TypedDict, TypeGuard, TypeVar + +import nest_asyncio +from beets import autotag, importer, plugins +from beets.ui import UserError, _open_library +from beets.util import bytestring_path +from deprecated import deprecated + +from beets_flask.config import get_config +from beets_flask.disk import is_archive_file +from beets_flask.importer.progress import Progress, ProgressState +from beets_flask.importer.types import ( + BeetsAlbum, + BeetsLibrary, + DuplicateAction, +) +from beets_flask.logger import log +from beets_flask.server.exceptions import ( + ApiException, + DuplicateException, + IntegrityException, + NoCandidatesFoundException, + NotImportedException, + to_serialized_exception, +) +from beets_flask.utility import capture_stdout_stderr + +from .pipeline import AsyncPipeline +from .stages import ( + StageOrder, + finalize, + group_albums, + identify_duplicates, + lookup_candidates, + manipulate_files, + match_threshold, + plugin_stage, + read_tasks, + user_query, +) +from .states import ProgressState, SessionState + +nest_asyncio.apply() + +# ---------------------------------------------------------------------------- # +# Types and helpers # +# ---------------------------------------------------------------------------- # + +T = TypeVar("T") + +TaskIdMapping = defaultdict[str, T] +TaskIdMappingArg = dict[str, T | None] | None + + +def parse_task_id_mapping(mapping: TaskIdMappingArg[T], default: T) -> TaskIdMapping[T]: + """ + Convert the flexible arguments to stricter TaskIdMapping that sessions use internally. + + Parameters + ---------- + mapping : TaskIdMappingArg + For each task_id (key) which action to take (value). + If None, the default action is used for all tasks. + If "*" is used as key, this action is used for all tasks, and only one key-value + pair is allowed. + default : T + Default value to use for all tasks that are not in the mapping, or "*". + + + Note + ---- + TaskIdMappings are defaultdicts, which keeps the lower level logic simpler. + TaskIdMappingsArgs are just dicts, which are serializable trhough api and redis + thread bounds. + """ + + m: TaskIdMapping[T] = defaultdict(lambda: default) + if mapping is not None: + if "*" in mapping.keys(): + if len(mapping) > 1: + raise ValueError( + "If you use '*' as key, you cannot use any other keys in the mapping." + ) + else: + return defaultdict(lambda: mapping["*"] or default) + + for k, v in mapping.items(): + if v is None: + continue + m[k] = v + + return m + + +class CandidateChoiceFallback(Enum): + """ + Type for the candidate choice. + + Candidate Choices are either a string (the candidate id) or a special case + (asis candidate, or the best one). + """ + + ASIS = 1 + BEST = 2 + + +CandidateChoice = str | CandidateChoiceFallback + + +class Search(TypedDict): + """Search for a candidate. + + This is used to search for a candidate in the preview session. + """ + + search_ids: list[str] + search_artist: str | None + search_album: str | None + + +def _is_search(d: Any) -> TypeGuard[Search]: + """Check if the given dict is a Search object.""" + return ( + d is not None + and isinstance(d, dict) + and "search_ids" in d + and isinstance(d["search_ids"], list) + ) + + +# ---------------------------------------------------------------------------- # +# Sessions # +# ---------------------------------------------------------------------------- # + + +class BaseSession(importer.ImportSession, ABC): + """Base class for our GUI-based ImportSessions. + + Operates on single Albums / files. + + Parameters + ---------- + path : list[str] + list of album folders to import + config_overlay : str or dict + path to a config file to overlay on top of the default config. + Note that if `dict`, the lazyconfig notation e.g. `{import.default_action: skip}` + wont work reliably. Better nest the dicts: `{import: {default_action: skip}}` + + Note: It's a design choice to require that you manually create and pass the + `SessionState` object. Usually the states go into the database, which needs explizit + handling beyond the session. + """ + + # attributes needed to create a beetsTag instance for our database + # are contained in the associated SessionState -> TaskState -> CandidateStates + state: SessionState + + pipeline: AsyncPipeline[importer.ImportTask, Any] | None = None + config_overlay: dict + + # FIXME: only for typehint until we update beets + lib: BeetsLibrary # type: ignore + + def __init__( + self, + state: SessionState, + config_overlay: dict | None = None, + ): + if not state.path.exists(): + raise FileNotFoundError(f"Path {state.path} does not exist.") + if state.path.is_file() and not is_archive_file(state.path): + raise ValueError( + f"Path {state.path} is not an archive file. " + + "Importing singletons is not supported yet." + ) + + # FIXME: This is a super bad convention of the original beets. + # We do not want to pollute a global config object every time a session runs. + # This is fine for the cli tool, where each run creates only one session + # but not for our long-running webserver. + config = get_config() + if isinstance(config_overlay, dict): + config.set_args(config_overlay) + + self.config_overlay = config_overlay or {} + self.state = state + + super().__init__( + lib=_open_library(config), + paths=[bytestring_path(state.path)], + query=None, + loghandler=None, + ) + # Hacky workaround to use our logging, to allow plugins to communicate + self.logger.handlers = log.handlers + log.debug(f"Created new {self.__class__.__name__} for {state.path}") + + @property + def path(self) -> Path: + return self.state.path + + @deprecated + def run_and_capture_output(self) -> tuple[str, str]: + """Run the import session and capture the output. + + Uses the original beets import session run method, + with lots of overhead. + Sets self.preivew to output and error messages occuring during run. + + Returns + ------- + tuple[str, str]: out, err + """ + self.logger.debug(f"{self.paths}") + out, err, _ = capture_stdout_stderr(self.run) + self.preview = out + "\n\n" + err if err else out + return out, err + + def get_config_value(self, key: str, type_func: Callable | None = None) -> Any: + """Get a config value from the overlay or default. + + Use dots to separate levels. + """ + + path = key.split(".") + + overlay = self.config_overlay + for p in path: + overlay = overlay.get(p, {}) + + # overlay takes priority + if not isinstance(overlay, dict): + return type_func(overlay) if type_func else overlay + + # get settings from user settings, this is not a dict, but confuse config + # the confuse config views do not throw key errors, and their .get() is not + # the same as dict.get(), but rather resolves the value. + default = get_config() + for p in path: + default = default[p] + default = default.get(type_func) if type_func else default.get() # type: ignore[call-overload] + return default + + # -------------------------- State handling helpers -------------------------- # + + def set_task_progress( + self, task: importer.ImportTask, progress: ProgressState | Progress | str + ): + """Set the progress for a task belonging to the session. + + If string is given it is set as the message of the current progress. + Note: currently we only implement status on the level of the whole import session, + but should eventually do this per selection (task). + """ + + task_state = self.state.get_task_state_for_task_raise(task) + + task_state.set_progress(progress) + + def get_task_progress(self, task: importer.ImportTask) -> ProgressState | None: + """Get the progress of the task, via this sessions state.""" + task_state = self.state.get_task_state_for_task(task) + return task_state.progress if task_state else None + + # -------------------------------- Stages -------------------------------- # + + @property + @abstractmethod + def stages(self) -> StageOrder: + """Set the stages for the session. + + In Subclasses, define the order of stages here. + """ + raise NotImplementedError("Implement in subclass") + + def resolve_duplicate(self, task: importer.ImportTask, found_duplicates): + """Overload default resolve duplicate and skip it. + + This basically skips this stage. + """ + self.logger.warning( + "Skipping duplicate resolution. " + + f"Your session should implement this! -> {self.__class__.__name__}" + ) + task.set_choice(importer.Action.SKIP) + + def choose_item(self, task: importer.ImportTask): + """Overload default choose item and skip it. + + This session should not reach this stage. + """ + self.logger.debug(f"skipping choose_item {task}") + return importer.Action.SKIP + + def should_resume(self, path): + """Overload default should_resume and skip it. + + Should normally be no problem if the config is set correctly, but just + in case. + """ + self.logger.debug(f"skipping should_resume {path}") + return False + + def identify_duplicates(self, task: importer.ImportTask): + """For all candidates, check if they have duplicates in the library. + + This stage should only be run for preview sessions, but we still have + some old code in stages.py/user_query(). + """ + raise NotImplementedError( + f"This session should not reach this stage. {self.__class__.__name__}" + ) + + def lookup_candidates(self, task: importer.ImportTask): + """Lookup candidates for the task. + + This stage should only be run for preview sessions, but we still have + some old code in stages.py/user_query(). + """ + raise NotImplementedError( + f"This session should not reach this stage. {self.__class__.__name__}" + ) + + def finalize(self, task: importer.ImportTask): + """Last stage called and customizable any session.""" + self.logger.debug(f"Finalized {self} {task}") + + # ---------------------------------- Run --------------------------------- # + + def run_sync(self) -> SessionState: + """Run the import session synchronously.""" + return asyncio.run(self.run_async()) + + async def run_async(self) -> SessionState: + """Run the import session asynchronously. + + Does not set tasks to completed at the end. + Take care of this in subclasses. + """ + # For now, until we improve the upstream beets config logic, + # adhere to importer.ImportSession convention and create a local copy + # of the config. + config = get_config() + self.set_config(config["import"]) + + # TODO: check some config values. that are not compatible with our code. + self.pipeline = AsyncPipeline(start_tasks=read_tasks(self)) + + for s in self.stages.values(): + self.pipeline.add_stage(s) + + log.info(f"Running {self.__class__.__name__} on state<{self.state.id=}>.") + log.debug(f"Running {len(self.pipeline.stages)} stages.") + + # reset exception state + # TODO: To clear or not to clear, + # exception hierarchy and own table for exceptions/warnings + # self.state.exc = None + plugins.send("import_begin", session=self) + try: + assert self.pipeline is not None + await self.pipeline.run_async() + except importer.ImportAbortError: + log.debug(f"Interactive import session aborted by user") + except ApiException as e: + if e.persist_in_db: + log.debug(f"Persisting exception {e} in session state") + self.state.exc = to_serialized_exception(e) + raise e + except Exception as e: + self.state.exc = to_serialized_exception(e) + raise e + + log.info(f"Completed {self.__class__.__name__} on state<{self.state.id=}>.") + return self.state + + +class PreviewSession(BaseSession): + """Preview what would be imported. Only fetches candidates.""" + + group_albums: bool | None + autotag: bool | None + + def __init__( + self, + state: SessionState, + config_overlay: dict | None = None, + group_albums: bool | None = None, + autotag: bool | None = None, + **kwargs, + ): + """ + Create new PreviewSession. + + Parameters + ---------- + group_albums : bool | None + Whether to create multple tasks, one for each album found in the metadata + of the files. Set to true if you have multiple albums in a single folder. + If None: get value from beets config. + autotag : bool | None + Whether to look up metadata online. If None: get value from beets config. + """ + + super().__init__(state, config_overlay, **kwargs) + self.group_albums = group_albums + self.autotag = autotag + + # -------------------------------- Stages -------------------------------- # + + @property + def stages(self) -> StageOrder: + stages = StageOrder() + + if self.get_config_value("import.singletons"): + # beets tweaks the album grouping settings via overlay for singletons. + raise NotImplementedError("Singletons not implemented yet.") + + if self.group_albums or ( + self.group_albums is None and self.get_config_value("import.group_albums") + ): + stages.append(group_albums(self)) + + if self.autotag or ( + self.autotag is None and self.get_config_value("import.autotag") + ): + stages.append(lookup_candidates(self)) + + stages.append(identify_duplicates(self)) + stages.append(finalize(self)) + + return stages + + # --------------------------- Stage Definitions -------------------------- # + + def identify_duplicates(self, task: importer.ImportTask): + """For all candidates, check if they have duplicates in the library.""" + task_state = self.state.get_task_state_for_task_raise(task) + + for idx, cs in enumerate( + task_state.candidate_states + [task_state.asis_candidate] + ): + # This is a mutable operation i.e. candidate state is modfied here! + duplicates = cs.identify_duplicates(self.lib) + + if len(duplicates) > 0: + log.debug(f"Found duplicates for {cs.id=}: {duplicates}") + + def lookup_candidates(self, task: importer.ImportTask): + """Lookup candidates for the task.""" + + search_ids = self.config["search_ids"].as_str_seq() # might be an empty list + log.debug( + f"Looking up candidates for {task.paths}, using search ids {search_ids}" + ) + + task.lookup_candidates(search_ids) + + if len(task.candidates) == 0: + raise NoCandidatesFoundException(persist_in_db=True) + + # Update our state + task_state = self.state.get_task_state_for_task_raise(task) + task_state.add_candidates(task.candidates) + + +class AddCandidatesSession(PreviewSession): + """ + Preview session that adds a candidate to the ones already fetched. + + Can only run on a session state of a preview session that already has + candidates. + """ + + search: TaskIdMapping[Search | Literal["skip"]] + initial_task_states: dict[str, ProgressState] + + def __init__( + self, + state: SessionState, + config_overlay: dict | None = None, + search: TaskIdMappingArg[Search | Literal["skip"]] = None, + **kwargs, + ): + super().__init__(state, config_overlay, **kwargs) + + # None means skip search for this task + self.search = parse_task_id_mapping(search, "skip") + self.initial_task_states = {} + + for task in self.state.task_states: + self.initial_task_states[task.id] = deepcopy(task.progress) + if task.progress >= Progress.PREVIEW_COMPLETED: + # Reset task progress only for tasks that have search values + # other tasks are skipped + # This should only be relevant for searches on multiple tasks + # (i.e. folders) + s = self.search[task.id] + if s != "skip": + task.set_progress(Progress.LOOKING_UP_CANDIDATES - 1) + + def lookup_candidates(self, task: importer.ImportTask): + """Amend the found candidate to the already existing candidates (if any).""" + # see ref in lookup_candidates in beets/importer.py + + task_state = self.state.get_task_state_for_task_raise(task) + search = self.search[task_state.id] + + if search == "skip": + log.debug(f"Skipping search for {task_state.id=}") + return + + # Beets treats empty strings like real strings, which is not what we want here. + # TODO: revisit once PR merged upstream: https://github.com/beetbox/beets/pull/6117 + if ( + search["search_artist"] is not None + and search["search_artist"].strip() == "" + ): + search["search_artist"] = None + if search["search_album"] is not None and search["search_album"].strip() == "": + search["search_album"] = None + search["search_ids"] = list( + filter(lambda x: x.strip() != "", search["search_ids"]) + ) + + log.debug(f"Using {search=} for {task_state.id=}, {task_state.paths=}") + + try: + _, _, prop = autotag.tag_album( + task.items, + search_ids=search["search_ids"], + search_album=search["search_album"], + search_artist=search["search_artist"], + ) + except Exception as e: + # TODO: With beets 2.6.0 this should be revisited + # since beets should than be able to handle these exceptions + # gracefully upstream. + # https://github.com/beetbox/beets/pull/5965 + from beetsplug.musicbrainz import MusicBrainzAPIError + from beetsplug.spotify import APIError as SpotifyAPIError + + if isinstance(e, MusicBrainzAPIError): + raise NoCandidatesFoundException( + f"Failed to contact Musicbrainz API: {e.get_message()}", + persist_in_db=False, + ) + elif isinstance(e, SpotifyAPIError): + raise NoCandidatesFoundException( + f"Failed to contact Spotify API: {e}", + persist_in_db=False, + ) + else: + raise NoCandidatesFoundException( + f"Failed to contact online APIs.", + persist_in_db=False, + ) + + task_state.add_candidates(prop.candidates) + + # Update quality of best candidate, likely not needed for us, only beets cli. + task.rec = max(prop.recommendation, task.rec or autotag.Recommendation.none) + + if len(prop.candidates) == 0: + error_text = "Search found no candidates via " + if search["search_ids"]: + error_text += f"ids: {', '.join(search['search_ids'])}; " + if search["search_artist"]: + error_text += f"artist: {search['search_artist']}; " + if search["search_album"]: + error_text += f"album: {search['search_album']}; " + error_text += NoCandidatesFoundException.metadata_plugin_info() + raise NoCandidatesFoundException( + error_text, + persist_in_db=False, + ) + # Hack: Clear exception if we found new candidates + elif ( + self.state.exc is not None + and self.state.exc["type"] == "NoCandidatesFoundException" + ): + log.debug( + "Clearing previous NoCandidatesFoundException after finding candidates via search." + ) + self.state.exc = None + + def finalize(self, task: importer.ImportTask): + """Restore initial taks and session states.""" + + task_state = self.state.get_task_state_for_task_raise(task) + if task_state.id in self.initial_task_states: + task_state.set_progress(self.initial_task_states[task_state.id]) + else: + log.warning( + f"Task {task_state.id} not in initial task states. " + + "Cannot restore previous progress." + ) + + self.logger.debug(f"Finalized {self} {task}") + + +class ImportSession(BaseSession): + """ + Import session that assumes we already have a match-id. + + Needs to run from an already finished Preview Session. + """ + + candidate_ids: TaskIdMapping[CandidateChoice] + duplicate_actions: TaskIdMapping[DuplicateAction] + + def __init__( + self, + state: SessionState, + config_overlay: dict | None = None, + candidate_ids: TaskIdMappingArg[CandidateChoice] = None, + duplicate_actions: TaskIdMappingArg[DuplicateAction] = None, + ): + """Create new ImportSession. + + Parameters + ---------- + candidate_ids : optional + Either id of candidate(s) or the import choice. This is used to determine which + candidate to import. If a dict is given, the keys are the task ids and the + values are the candidate ids. You can also use the import choice enum + `ImportChoice.ASIS` or `ImportChoice.BEST` to indicate that you want to + import the candidate as-is or the best candidate. + FIXME: at the moment asis is broken + duplicate_actions : str + The action to take if duplicates are found. One of "skip", "keep", + "remove", "merge", "ask". If None, the default is read from + the user config and applied to all tasks. + """ + + config_overlay = {} if config_overlay is None else config_overlay + if config_overlay.get("import", {}).get("search_ids") is not None: + raise ValueError("search_ids set in config_overlay. This is not supported.") + + super().__init__(state, config_overlay) + + # Create a mapping for the duplicate action + # each task might have a different action. + # if none is given the default action is used from the config + default_action: DuplicateAction = self.get_config_value( + "import.duplicate_action", str + ) + self.duplicate_actions = parse_task_id_mapping( + duplicate_actions, default_action + ) + + # For candidates, None means to take best + self.candidate_ids = parse_task_id_mapping( + candidate_ids, CandidateChoiceFallback.BEST + ) + + async def run_async(self) -> SessionState: + # only allow import sessions to run on preview states (not other import states) + if self.state.progress == Progress.IMPORT_COMPLETED: + log.error( + f"Cannot run {self.__class__.__name__} from states that already " + + f"completed an import. (i.e. other imports) [{self.state.progress}]" + ) + e = UserError("Cannot redo imports. Try undo and/or retag!") + self.state.exc = to_serialized_exception(e) + raise e + elif self.state.progress > Progress.PREVIEW_COMPLETED: + log.warning( + f"Resetting state from {self.state.progress} to PREVIEW_COMPLETED for " + + f"import session {self.state.id}." + ) + for task in self.state.task_states: + task.set_progress(Progress.PREVIEW_COMPLETED) + + state = await super().run_async() + + # Trigger cli_exit event to have compatibility with beets plugin that + # execute after the import is completed. + plugins.send("cli_exit", lib=self.lib) + return state + + # ------------------------------ Stages ------------------------------ # + + @property + def stages(self): + stages = StageOrder() + + # FIXME: user_query calls task.choose_match, which calls session.choose_match. + # Better abstraction needed upstream. + stages.append(user_query(self)) + + # Early import stages + plugs = plugins.find_plugins() + for p in plugs: + for stage in p.get_early_import_stages(): + stages.append( + plugin_stage( + self, + stage, + ProgressState( + Progress.EARLY_IMPORTING, + plugin_name=stage.__name__, + ), + ), + name=f"early_plugin_stage_{p.__class__.__name__}_{stage.__name__}", + ) + + # Import stages + for p in plugs: + for stage in p.get_import_stages(): + stages.append( + plugin_stage( + self, + stage, + ProgressState( + Progress.IMPORTING, + plugin_name=stage.__name__, + ), + ), + name=f"plugin_stage_{p.__class__.__name__}_{stage.__name__}", + ) + + # finally, move files + stages.append(manipulate_files(self)) + stages.append(finalize(self)) + + return stages + + def finalize(self, task: importer.ImportTask): + """ + Reset previous match threshold exceptions. + + Needed because we might run a normal ImportSession manually after + the AutoImportSession. The AutoImportSession needs to keep an Exception in its + status to inform the frontend, which we need to clear here. + (No need to override `finalize` in AutoImportSession, despite it inherriting + from here, because it raises when below threshold). + """ + if ( + self.state.exc is not None + and self.state.exc["type"] == "NotImportedException" + and self.state.exc["message"].startswith("Match below threshold") + ): + log.debug( + "Clearing previous MatchThresholdException after successful import." + ) + self.state.exc = None + super().finalize(task) + + # --------------------------- Stage Definitions -------------------------- # + + def choose_match(self, task: importer.ImportTask): + self.logger.setLevel(logging.DEBUG) + self.logger.debug(f"choose_match {task}") + + task_state = self.state.get_task_state_for_task_raise(task) + + # Pick the candidate to import + candidate_id = self.candidate_ids[task_state.id] + + if isinstance(candidate_id, str): + candidate_state = task_state.get_candidate_state_by_id(candidate_id) + if candidate_state is None: + raise ValueError(f"Candidate with id {candidate_id} not found.") + elif candidate_id == CandidateChoiceFallback.BEST: + candidate_state = task_state.best_candidate_state + if candidate_state is None: + raise ValueError(f"No candidate found.") + elif candidate_id == CandidateChoiceFallback.ASIS: + candidate_state = task_state.asis_candidate + else: + raise NotImplementedError("ImportChoice.ASIS not implemented yet.") + + # update task_state to keep track of the choice in the database + task_state.chosen_candidate_state_id = candidate_state.id + log.debug( + f"Setting chosen candidate for task {task_state.id} to {candidate_state.id}" + ) + + # Let plugins display info + results = plugins.send("import_task_before_choice", session=self, task=task) + actions = [action for action in results if action] + + if len(actions) > 0: + # decide if we can just move past this and ignore the plugins + raise UserError( + f"Plugins returned actions, which is not supported for {self.__class__.__name__}" + ) + + # ASIS + if candidate_state.id == task_state.asis_candidate.id: + log.debug(f"Importing {task} as-is") + return importer.Action.ASIS + + return candidate_state.match + + def resolve_duplicate( + self, task: importer.ImportTask, found_duplicates: list[BeetsAlbum] + ): + log.debug( + f"Resolving duplicates for {task} with action {self.duplicate_actions}" + ) + + if len(found_duplicates) == 0: + log.debug(f"No duplicates found for") + return + + task_state = self.state.get_task_state_for_task_raise(task) + task_duplicate_action = self.duplicate_actions[task_state.id] + task_state.duplicate_action = task_duplicate_action + match task_duplicate_action: + case "skip": + task.set_choice(importer.Action.SKIP) + case "keep": + pass + case "remove": + task.should_remove_duplicates = True + case "merge": + task.should_merge_duplicates = True + case "ask": + # task.set_choice(importer.action.SKIP) + raise DuplicateException( + "You have set the duplicate action to 'ask' in your beets config." + ) + case _: + raise DuplicateException( + f"Unknown duplicate action: {self.duplicate_actions}" + ) + + +class BootlegImportSession(ImportSession): + """ + Import session to import without modifying metadata. + + No preview session required. + + Essentially `beet import --group-albums -A`, ideal for bootlegs and + just getting a folder into your library where you are sure the metadata is correct. + """ + + def __init__( + self, + state, + config_overlay: dict | None = None, + **kwargs, + ): + """Create new ImportAsIsSession. + + Parameters + ---------- + state: SessionState + Preconfigured state of the import session. + config_overlay : dict + Configuration to overlay on top of the default config. + "import.group_albums", "import.autotag" and "import.search_ids" are ignored. + **kwargs + See `ImportSession`. + """ + + config_overlay = {} if config_overlay is None else deepcopy(config_overlay) + import_overlay = config_overlay.get("import", {}) + + if "group_albums" in import_overlay and import_overlay["group_albums"] is False: + log.warning("Overwriting 'group_albums' in config_overlay.") + if "autotag" in import_overlay and import_overlay["autotag"] is True: + log.warning("Overwriting 'autotag' in config_overlay.") + if "search_ids" in import_overlay: + log.warning("Overwriting 'search_ids' in config_overlay.") + + import_overlay["group_albums"] = True + import_overlay["autotag"] = False + import_overlay["search_ids"] = None + + config_overlay["import"] = import_overlay + + super().__init__(state, config_overlay, **kwargs) + + # overwrite the default action for all tasks + self.candidate_ids = parse_task_id_mapping( + {"*": CandidateChoiceFallback.ASIS}, + CandidateChoiceFallback.ASIS, + ) + + @property + def stages(self): + stages = super().stages + stages.insert(before="user_query", stage=group_albums(self)) + return stages + + +class AutoImportSession(ImportSession): + """Generate a preview and import if the best match is good enough. + + Preview generation is skipped if the provided session state already has a preview. + + Wether the import is triggered depends on the specified `import_threshold`, or + the beets-config setting `match.strong_rec_thresh`. + The match quality is calculated via penalties, thus it ranges from 0 to 1, but a + perfect match is at 0. The same convention is used for thresholds. + + The default threshold is 0.04, so that a "96% match or better" will be imported. + + Raises a `NotImportedException` if the match quality is worse than the threshold, + stopping the pipeline. + """ + + import_threshold: float + + def __init__( + self, + state, + config_overlay: dict | None = None, + import_threshold: float | None = None, + **kwargs, + ): + """Create new AutoImportSession. + + Parameters + ---------- + state: SessionState + Preconfigured state of the import session. + config_overlay : optional, dict + Configuration to overlay on top of the default config. + import_threshold: optional, float + 0 to import only perfect matches, 1 to import everything. Default is 0.04. + **kwargs + See `ImportSession`. + """ + + super().__init__(state, config_overlay, **kwargs) + + if import_threshold is None: + self.import_threshold = self.get_config_value( + "match.strong_rec_thresh", float + ) + else: + self.import_threshold = import_threshold + + @property + def stages(self): + stages = super().stages + stages.insert(before="user_query", stage=match_threshold(self)) + return stages + + def match_threshold(self, task: importer.ImportTask): + """Check if the match quality is good enough to import. + + Returns true if candidates were found, and the match quality is better than + threshlold. + + Note: What stops the pipeline is that we set task.choice to importer.action.SKIP, + or raise an exception. + + Currently raising, as we do not have a dedicated progress for "not imported". + + FIXME: Instead of adding a whole new stage, with progress and a session function, + we could simply extend the choose_match function. It's defined for the normal + import session, and gets called early in user_query. + (see #78) + """ + try: + task_state = self.state.get_task_state_for_task(task) + distance = float(task_state.best_candidate_state.distance) # type: ignore + except (AttributeError, TypeError): + distance = 2.0 + + if len(task.candidates) == 0: + raise NoCandidatesFoundException() + + if distance > self.import_threshold: + log.debug( + f"Best candidate was worse than threshold {distance=} {self.import_threshold=}" + ) + d = (1 - distance) * 100 + t = (1 - self.import_threshold) * 100 + raise NotImportedException(f"Match below threshold ({d:.0f}% < {t:.0f}%)") + # beets would handle this via the task action: + task.set_choice(importer.action.SKIP) + else: + log.info( + f"Best candidate was better than threshold, importing to library. {distance=} {self.import_threshold=}" + ) + + +class UndoSession(BaseSession): + delete_files: bool + + def __init__( + self, + state: SessionState, + config_overlay: dict | None = None, + delete_files: bool = False, + **kwargs, + ): + super().__init__(state, config_overlay, **kwargs) + self.delete_files = delete_files + + async def run_async(self) -> SessionState: + """Undo an import. + + A bit of a hack to reuse the BaseSession as we do + not operate on tasks here but makes things easier in + calling this in the invoker. + + Note: this Session should only be run in the (single-thread) import queue, + because we want to limit beets-library and file interactions to be serial + to avoid conflicts. + """ + + if self.state.progress != Progress.IMPORT_COMPLETED: + log.error( + f"Cannot undo import from state {self.state.progress}. " + + "Only imports can be undone." + ) + e = UserError( + "Cannot undo if never imported! You need to import to undo first." + ) + self.state.exc = to_serialized_exception(e) + raise e + + for t_state in self.state.task_states: + t_state.set_progress(Progress.DELETING) + + # Revert the movement of items that beets does during manipulate_files() + # TODO: support Move operations (currently we only allow copy) + + # HACK: To allow an reimport after an undo, we need to use the old + # paths of the items. This is a bit hacky, but We did not find + # a better way to do this while maintaining the original beets logic. + # -> the old_paths attribute of a beets_task is set during task.manipulate_files() + # Unfortunately, here task.imported_items() does not work yet (.match not set) + + for t_state in self.state.task_states: + chosen_candidate = t_state.chosen_candidate_state + if chosen_candidate is None: + raise ValueError("No chosen candidate found for task.") + if t_state.task.old_paths is None: + raise ValueError("No old paths found for task.") + for idx, item in enumerate(chosen_candidate.match.mapping.keys()): # type: ignore + # yes, keys are the items and need to be updated! + # mapping maps objects to objects. + item.path = t_state.task.old_paths[idx] + + # Delete files + excs: list[Exception] = [] + pths: list[Path] = [] + for t_state in self.state.task_states: + try: + delete_from_beets(t_state.id, self.delete_files, self.lib) + except Exception as e: + items = self.lib.items(f"gui_import_id:{t_state.id}") + + excs.append(e) + pths.extend([Path(item.path.decode("utf-8")) for item in items]) + + if len(excs) > 0: + for exc in excs: + log.exception(exc) + + # FIXME: Multi/array exceptions need handling + self.state.exc = to_serialized_exception(excs[0]) + raise IntegrityException( + "Could not delete all items. Some items might be left in the library. " + + f"Problematic files were: {pths}" + ) + + # Update our state and progress + for t_state in self.state.task_states: + t_state.set_progress(Progress.DELETION_COMPLETED) + + # Send cli_exit event to have compatibility with beets plugin that + # execute after the import is completed. + plugins.send("cli_exit", lib=self.lib) + + return self.state + + @property + def stages(self): + # This tweaked session skips the pipeline + return StageOrder() + + +## Edge cases +# 1. Session doesn't exist anymore -> delete +# 2. Session exists but multiple tasks -> undo (session) +# 3. Session exists but file do not anymore in import folder -> undo, warnings + +# Flow: +# - remove from beets db +# - revert our session (if possible) + + +def delete_from_beets(task_id: str, delete_files: bool, lib: BeetsLibrary): + """Low-level, delete the items from the beets library.""" + + # We set a gui_import_id in the beets database this is equal to the session id + # see _apply_choice in stages.py + + items = lib.items(f"gui_import_id:{task_id}") + + if len(items) == 0: + raise ValueError("No items found that match this import session id.") + + with lib.transaction(): + for item in items: + item.remove(delete_files) diff --git a/backend/beets_flask/importer/stages.py b/backend/beets_flask/importer/stages.py index ab499b2d..ac978a87 100644 --- a/backend/beets_flask/importer/stages.py +++ b/backend/beets_flask/importer/stages.py @@ -1,694 +1,694 @@ -"""Beets-flask extends every stage in the normal beets import session. - -This allows us to keep track of the import state and communicate it to the frontend. -""" - -from __future__ import annotations - -import itertools -from collections.abc import Callable, Generator -from datetime import datetime -from functools import wraps -from inspect import isgenerator -from typing import ( - TYPE_CHECKING, - Any, - Literal, - TypeVar, - TypeVarTuple, - cast, -) - -from beets import library, plugins -from beets.importer import ( - Action, - ImportTask, - SentinelImportTask, -) -from beets.importer.stages import ( - _extend_pipeline, - _freshen_items, -) -from beets.importer.tasks import ImportTaskFactory -from beets.util import MoveOperation, bytestring_path, displayable_path -from beets.util import pipeline as beets_pipeline - -from beets_flask import log -from beets_flask.server.exceptions import ( - NoCandidatesFoundException, - NotImportedException, -) - -from .progress import Progress, ProgressState -from .types import BeetsImportTask - -if TYPE_CHECKING: - from beets_flask.importer.session import ( - AutoImportSession, - BaseSession, - ImportSession, - ) - from beets_flask.importer.types import BeetsItem - - from .pipeline import Stage - - # Tell type-checking that subclasses of BaseSession are allowed - Session = TypeVar("Session", bound=BaseSession) - - -def skip_until( - progress: Progress, -) -> Callable[ - [Callable[[Session, Task, *Arg], Ret]], - Callable[[Session, Task, *Arg], Ret | Task], -]: - """Decorater to skip all tasks with progress lower than the desired one. - - Needed to resume up a session without re-doing everything thats already contained - in the state. - - Usage - ----- - ```python - @skip_until(Progress.READING_FILES) - def read_task(session: BaseSessionNew, task: ImportTask): - pass - ``` - """ - - def decorator( - func: Callable[[Session, Task, *Arg], Ret], - ) -> Callable[[Session, Task, *Arg], Ret | Task]: - @wraps(func) - def wrapper(session: Session, task: Task, *args: *Arg) -> Ret | Task: - # Skip automatically if the task is already progressed - # Note that >= and > below both have caveats: - # >= might give us a too optimistic state if the function call fails - # > might re-run the function if we resume, and we have not testet - # what happens when doing the same state-mutation twice - if not isinstance(task, ImportTask): - log.warning( - f"Skipping {progress} for {task} as it is not an ImportTask" - ) - return task - - # We set the progress before the stage runs, thus we need to compare - # with - and redo - the previous progress. - # Otherwise, we couldn't retry a stage once it crashed or was aborted. - # For jobs that did not fail, we have extra progress levels (dummies) that - # do not need any compute to redo. They are the most common point to resume - # from (PREVIEW_COMPLETED in particular). - prev_progress = session.state.upsert_task(task).progress - if prev_progress > progress: - log.debug( - f"Skipping {progress} for {task} because task progress {prev_progress=}" - ) - return task - - log.debug(f"Skipped until encountered {progress} for {task}") - return func(session, task, *args) - - return wrapper - - return decorator - - -def set_progress( - before: Progress, - after: Progress | None = None, - on_error: dict[type[Exception], Progress] = {}, -) -> Callable[ - [Callable[[Session, Task, *Arg], Ret]], - Callable[[Session, Task, *Arg], Ret | Task], -]: - """Decorater to set the progress of a task. - - Basically calls - `session.set_progress(task, progress)` - before the decorated function. - - Optionally, if an Exception is encountered while processing the task, - another progress can be set (to account for later pipeline-steps, which will not - take place). By default, we still raise. - - When combining, use after `skip_until()` decorator. - - Parameters - ---------- - before: Progress - Progress to set when (before) processing the task. - after: Progress, optional - Progress to set after processing the task. By default `before` is kept. - on_error: dict[type[Exception], Progress], optional - Mapping, which Progress to set for which Exception (instance check on the - Exception). By default, if an exception occurs, the `before` Progress is kept. - - Usage - ----- - ```python - @set_progress(Progress.READING_FILES) - def read_task(session: BaseSessionNew, task: ImportTask): - pass - - @set_progress( - Progress.LOOKING_UP_CANDIDATES, - on_Error={NoCandidatesFoundException: Progress.PREVIEW_COMPLETED} - ) - def lookup_candidates(session: BaseSessionNew, task: ImportTask): - pass - ``` - """ - - def decorator( - func: Callable[[Session, Task, *Arg], Ret], - ) -> Callable[[Session, Task, *Arg], Ret | Task]: - @wraps(func) - def wrapper(session: Session, task: Task, *args: *Arg) -> Ret | Task: - log.debug(f"Setting progress {before=} for {task}") - - session.set_task_progress(task, before) - try: - res = func(session, task, *args) - if after is not None: - log.debug(f"Setting progress {after=} for {task}") - session.set_task_progress(task, after) - return res - except Exception as e: - for e_to_handle, p in on_error.items(): - if isinstance(e, e_to_handle): - log.debug( - f"Setting progress to {p} after Exception for {task} {e}" - ) - session.set_task_progress(task, p) - raise - - return wrapper - - return decorator - - -class StageOrder(dict): - """An ordered dict of stages, mapping the stage name to its generator. - - Returned by each sessions `.stages` property. - """ - - def append( - self, - stage: Stage[ImportTask, Any], - name: str | None = None, - ): - """Append a stage to the Order.""" - - name = name or str(getattr(stage, "__name__", f"unknown_stage")) - if name in self.keys(): - raise ValueError(f"Stage with name {name} already exists.") - - self[name] = stage - - def insert( - self, - stage: Stage[ImportTask, Any], - name: str | None = None, - after: str | None = None, - before: str | None = None, - ): - """Insert a stage after or before another specific stage.""" - - if after is None and before is None or (after and before): - raise ValueError("Either `after` or `before` must be specified.") - - name = name or str(getattr(stage, "__name__", f"unknown_stage")) - if name in self.keys(): - raise ValueError(f"Stage with name {name} already exists.") - - # even for the OrderedDict there is no insert at index method, so just rebuild - keys = list(self.keys()) - values = list(self.values()) - - idx = keys.index(after or before) # type: ignore - if after: - idx += 1 - keys.insert(idx, name) - values.insert(idx, stage) - - self.clear() - self.update(zip(keys, values)) - - -# --------------------------------- Decorator -------------------------------- # - -Arg = TypeVarTuple("Arg") # args und kwargs -Ret = TypeVar("Ret") # return type -Task = TypeVar( - "Task", - bound=BeetsImportTask, -) # task - - -def stage( - func: Callable[[*Arg, Task], Ret | None], -): - """Decorate a function to become a simple stage. - - Yields a task and waits until the next task is sent to it. - - >>> @stage - ... def add(n, i): - ... return i + n - >>> pipe = Pipeline([ - ... iter([1, 2, 3]), - ... add(2), - ... ]) - >>> list(pipe.pull()) - [3, 4, 5] - """ - - # use the wraps decorator to lift function name etc, we use this in StageOrder class - @wraps(func) - def coro( - *args: *Arg, - ) -> Generator[Ret | Task | None, Task, None]: - # in some edge-cases we get no task. thus, we have to include the generic R - task: Task | Ret | Generator[Task] | None = None - while True: - if isgenerator(task): - for t in task: - task = yield t - else: - task = yield cast( - Task | None | Ret, task - ) # wait for send to arrive. the first next() always returns None - # yield task, call func which gives new task, yield new task in next() - task = cast(Task, task) # Slightly hacky, but we know task is a Task here - task = func(*(args + (task,))) - - return coro - - -def mutator_stage( - func: Callable[[*Arg, Task], Ret], name: str | None = None -) -> Callable[[*Arg], Generator[Ret | Task | None, Task, None]]: - """Decorate a function that manipulates items in a coroutine to become a simple stage. - - Yields a task and waits until the next task is sent to it. - - >>> @mutator_stage - ... def setkey(key, item): - ... item[key] = True - >>> pipe = Pipeline([ - ... iter([{'x': False}, {'a': False}]), - ... setkey('x'), - ... ]) - >>> list(pipe.pull()) - [{'x': True}, {'a': False, 'x': True}] - """ - - @wraps(func) - def coro( - *args: *Arg, - ) -> Generator[Ret | Task | None, Task, None]: - task = None - while True: - task = yield task # wait for send to arrive. the first next() always returns None - # perform function on task, and in next() send the same, modified task - # funcs prob. modify task in place? - func(*(args + (task,))) - - return coro - - -# --------------------------------- Producer --------------------------------- # - - -def read_tasks( - session: BaseSession, -): - """Read the files from the paths and generate tasks. - - If the session already has tasks yield them. - - Adapted closely from beets, but we do not need/support resuming and skipping - """ - - log.debug(f"Reading files") - - # Our Skip-check usually uses Progress, but here we do not have progress yet - # We want to catch the case when we resume a session, - # where we manually add old tasks from disk. - if len(session.state.tasks) > 0: - for task in session.state.tasks: - yield task - return - - for toppath in session.paths: - task_factory = ImportTaskFactory(toppath, session) - for task in task_factory.tasks(): - if isinstance(task, SentinelImportTask): - log.debug(f"Skipping {displayable_path(toppath)}") - continue - - task_state = session.state.upsert_task(task) - log.debug(f"Reading files from {displayable_path(toppath)}") - task_state.set_progress(Progress.READING_FILES) - yield task - - if not task_factory.imported: - log.warning(f"No files imported from {displayable_path(toppath)}") - - -# ----------------------------- Transform Stages ----------------------------- # - - -def __group(item: library.Item) -> tuple[str, str]: - return (item.albumartist or item.artist, item.album) - - -@stage -@skip_until(Progress.GROUPING_ALBUMS) -@set_progress(Progress.GROUPING_ALBUMS) -def group_albums( - session: BaseSession, - task: ImportTask, -) -> beets_pipeline.MultiMessage: - """ - Groups items of the task into albums using their metadata. - - The groups are identified using artist and album fields. - - Creates a new Task for each found album, and removes the old task (and task_state) - from the session. - """ - - # Split old task into multiple, and only keep new ones! - session.state.remove_task(task) - - tasks: list[ImportTask] = [] - sorted_items = sorted(task.items, key=__group) - for _, _items in itertools.groupby(sorted_items, __group): - items = list(_items) # Consume the iterator to a list - task = ImportTask(task.toppath, [i.path for i in items], items) - tasks += task.handle_created(session) - - # handle the beets-flask specific states - for task in tasks: - session.state.upsert_task(task) - - # FIXME: Not really sure we need this tbh, see also task.skip - # tasks.append(SentinelImportTask(task.toppath, task.paths)) - - return beets_pipeline.MultiMessage(tasks) - - -@mutator_stage -@skip_until(Progress.LOOKING_UP_CANDIDATES) -@set_progress( - Progress.LOOKING_UP_CANDIDATES, - on_error={NoCandidatesFoundException: Progress.PREVIEW_COMPLETED}, -) -def lookup_candidates( - session: BaseSession, - task: ImportTask, -): - """Performing the initial MusicBrainz lookup for an album. - - We tweaks this from upstream beets to not - call `task.lookup_candidates()` but instead `session.lookup_candidates(task)`, - with some extra logic - - This is more consistent, as it allows the logic - to be modified by each kind of session. - - Calls `task.lookup_candidates()`, - which sets attributes of the task: - - cur_artist # metadata in file - - cur_album - - candidates - - rec - """ - if task.skip: - # FIXME This gets duplicated a lot. We need a better - # abstraction. - return - - # FIXME: what happens with our new skip logic and plugins? - plugins.send("import_task_start", session=session, task=task) - - session.lookup_candidates(task) - - -@mutator_stage -@skip_until(Progress.IDENTIFYING_DUPLICATES) -@set_progress(before=Progress.IDENTIFYING_DUPLICATES, after=Progress.PREVIEW_COMPLETED) -def identify_duplicates( - session: BaseSession, - task: ImportTask, -): - """Stage to identify which candidates would be duplicates if imported.""" - if task.skip: - return - - session.identify_duplicates(task) - - -@stage -@skip_until(Progress.WAITING_FOR_USER_SELECTION) -@set_progress(Progress.WAITING_FOR_USER_SELECTION) -def user_query( - session: ImportSession, - task: ImportTask, -) -> ImportTask | beets_pipeline.MultiMessage | Literal["__PIPELINE_BUBBLE__"]: - """A coroutine for interfacing with the user about the tagging process. - - The coroutine accepts an ImportTask objects. It uses the - session's `choose_match` method to determine the `action` for - this task. Depending on the action additional stages are executed - and the processed task is yielded. - - It emits the ``import_task_choice`` event for plugins. Plugins have - access to the choice via the ``task.choice_flag`` property and may - choose to change it. - """ - - if task.skip: - return task - - if session.already_merged(task.paths): - # For some reason the type hinter does not like this without cast - return cast(Literal["__PIPELINE_BUBBLE__"], beets_pipeline.BUBBLE) - - # Ask the user for a choice. - # This calls session.choose_match(task)... but whyyy? - task.choose_match(session) - plugins.send("import_task_choice", session=session, task=task) - - # TODO: Import as singletons i.e. task.choice_flag is action.TRACKS - - # As albums: group items by albums and create task for each album - if task.choice_flag is Action.ALBUMS: - log.warning("This should never run via beets-flask!") - return _extend_pipeline( - [task], - group_albums(session), - lookup_candidates(session), - user_query(session), - ) - - # Note: this checks the global config for the default action. - found_duplicates = task.find_duplicates(session.lib) - # Different than vanilla beets, we do not use a stage for duplicate resolution, - # as it only calls the sessions method, anyway. - session.resolve_duplicate(task, found_duplicates) - - if task.should_merge_duplicates: - # Create a new task for tagging the current items - # and duplicates together - duplicate_items = task.duplicate_items(session.lib) - - # Duplicates would be reimported so make them look "fresh" - _freshen_items(duplicate_items) - duplicate_paths = [item.path for item in duplicate_items] - - # Record merged paths in the session so they are not reimported - session.mark_merged(duplicate_paths) - - merged_task = ImportTask( - None, task.paths + duplicate_paths, task.items + duplicate_items - ) - return _extend_pipeline( - [merged_task], lookup_candidates(session), user_query(session) - ) - - _apply_choice(session, task) - - return task - - -# Dynamicly set_progress for plugin name -# e.g. DetailedProgress(Progress.EARLY_IMPORT, plugin_name="my_plugin") -@mutator_stage -def plugin_stage( - session: BaseSession, - func: Callable[[BaseSession, ImportTask], None], - progress: ProgressState, - task: ImportTask, -): - # TODO: Skip if already progressed - session.set_task_progress(task, progress) - if task.skip: - return - - log.debug(f"Running plugin stage {func.__name__}") - func(session, task) - log.debug(f"Finished plugin stage {func.__name__}") - - task.reload() - log.debug(f"Reload plugin stage done {func.__name__}") - - -@mutator_stage -@skip_until(Progress.MATCH_THRESHOLD) -@set_progress( - Progress.MATCH_THRESHOLD, - on_error={NotImportedException: Progress.PREVIEW_COMPLETED}, -) -def match_threshold( - session: AutoImportSession, - task: ImportTask, -): - """Stage to determine if a match is good enough to be auto-imported.""" - if task.skip: - log.debug(f"Skipping task: {session=}, {task=}") - return task - - session.match_threshold(task) - - -# --------------------------------- Consumer --------------------------------- # - - -@stage -@skip_until(Progress.MANIPULATING_FILES) -@set_progress(before=Progress.MANIPULATING_FILES, after=Progress.IMPORT_COMPLETED) -def manipulate_files( - session: BaseSession, - task: ImportTask, -) -> ImportTask: - """A coroutine (pipeline stage) that performs necessary file. - - manipulations *after* items have been added to the library and - finalizes each task. - """ - if not task.skip: - if task.should_remove_duplicates: - task.remove_duplicates(session.lib) - - if session.config["copy"]: - operation = MoveOperation.COPY - # elif session.config["move"]: - # operation = MoveOperation.MOVE - # elif session.config["link"]: - # operation = MoveOperation.LINK - # elif session.config["hardlink"]: - # operation = MoveOperation.HARDLINK - # elif session.config["reflink"] == "auto": - # operation = MoveOperation.REFLINK_AUTO - # elif session.config["reflink"]: - # operation = MoveOperation.REFLINK - # else: - # operation = None - else: - log.warning( - "Beets-flask does not yet support other import modes than 'copy'. " - + "Please consider updating your config." - ) - operation = MoveOperation.COPY - - task.manipulate_files(session, operation, write=session.config["write"]) - - # Progress, cleanup, and event. - task.finalize(session) - - # Cleanup of duplicates. After successful import, we do not want the candidates - # to have duplicate-info anymore. The reason is that if we undo the import, - # there wont be any duplicates in the library. Also, we dont want to show - # the duplicate-badge for things that were imported. - # TODO: outlook: add new badges for what has been done with duplicates? - task_state = session.state.get_task_state_for_task_raise(task) - chosen_candidate = task_state.chosen_candidate_state - if chosen_candidate: - chosen_candidate.duplicate_ids = [] - # We remove duplicate info from all candidates. - # Although this causes the frontend to possibly not show the duplicate - # actions, that is fine: Upon re-import, we raise a duplicate action error, - # which is shown to the user, and then the duplicate actions will appear. - - if task_state.toppath is not None and len(session.state.task_states) == 1: - # If we have only one task and no toppath this indicates that we are - # operating on a temp directory we override the toppath - # to the session's folder path. - task.toppath = bytestring_path(session.state.folder_path) - - return task - - -@stage -def finalize( - session: BaseSession, - task: ImportTask, -) -> ImportTask: - """Give our custom sessions a way to do something at the very end.""" - session.finalize(task) - return task - - -def _apply_choice(session: ImportSession, task: ImportTask): - # tweaked version of beets apply_choices. - # we do not want to rely on the global config object - # (in particular, the set_fields, because we always want to set gui_import_id). - # see the original: - # from beets.importer import apply_choice - - if task.skip: - return - - # Change metadata. - if task.apply: - task.apply_metadata() - plugins.send("import_task_apply", session=session, task=task) - - # We need to reinsert items into the db if they were removed earlier - item: BeetsItem - for item in task.imported_items(): - item._db = session.lib - item.album_id = None - item.id = None - - task.add(session.lib) - task.set_fields(session.lib) - - # copy of core logic from set_fields() - items: list[BeetsItem] = task.imported_items() - - timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") - task_state = session.state.get_task_state_for_task_raise(task) - with session.lib.transaction(): - for item in items: - item.set_parse("gui_import_id", task_state.id) - item.set_parse("gui_import_date", timestamp) - item.store() - task.album.set_parse("gui_import_id", task_state.id) - task.album.set_parse("gui_import_date", timestamp) - task.album.store() - - -__all__ = [ - "read_tasks", - "group_albums", - "lookup_candidates", - "identify_duplicates", - "user_query", - "plugin_stage", - "manipulate_files", - "mark_tasks_completed", -] +"""Beets-flask extends every stage in the normal beets import session. + +This allows us to keep track of the import state and communicate it to the frontend. +""" + +from __future__ import annotations + +import itertools +from collections.abc import Callable, Generator +from datetime import datetime +from functools import wraps +from inspect import isgenerator +from typing import ( + TYPE_CHECKING, + Any, + Literal, + TypeVar, + TypeVarTuple, + cast, +) + +from beets import library, plugins +from beets.importer import ( + Action, + ImportTask, + SentinelImportTask, +) +from beets.importer.stages import ( + _extend_pipeline, + _freshen_items, +) +from beets.importer.tasks import ImportTaskFactory +from beets.util import MoveOperation, bytestring_path, displayable_path +from beets.util import pipeline as beets_pipeline + +from beets_flask import log +from beets_flask.server.exceptions import ( + NoCandidatesFoundException, + NotImportedException, +) + +from .progress import Progress, ProgressState +from .types import BeetsImportTask + +if TYPE_CHECKING: + from beets_flask.importer.session import ( + AutoImportSession, + BaseSession, + ImportSession, + ) + from beets_flask.importer.types import BeetsItem + + from .pipeline import Stage + + # Tell type-checking that subclasses of BaseSession are allowed + Session = TypeVar("Session", bound=BaseSession) + + +def skip_until( + progress: Progress, +) -> Callable[ + [Callable[[Session, Task, *Arg], Ret]], + Callable[[Session, Task, *Arg], Ret | Task], +]: + """Decorater to skip all tasks with progress lower than the desired one. + + Needed to resume up a session without re-doing everything thats already contained + in the state. + + Usage + ----- + ```python + @skip_until(Progress.READING_FILES) + def read_task(session: BaseSessionNew, task: ImportTask): + pass + ``` + """ + + def decorator( + func: Callable[[Session, Task, *Arg], Ret], + ) -> Callable[[Session, Task, *Arg], Ret | Task]: + @wraps(func) + def wrapper(session: Session, task: Task, *args: *Arg) -> Ret | Task: + # Skip automatically if the task is already progressed + # Note that >= and > below both have caveats: + # >= might give us a too optimistic state if the function call fails + # > might re-run the function if we resume, and we have not testet + # what happens when doing the same state-mutation twice + if not isinstance(task, ImportTask): + log.warning( + f"Skipping {progress} for {task} as it is not an ImportTask" + ) + return task + + # We set the progress before the stage runs, thus we need to compare + # with - and redo - the previous progress. + # Otherwise, we couldn't retry a stage once it crashed or was aborted. + # For jobs that did not fail, we have extra progress levels (dummies) that + # do not need any compute to redo. They are the most common point to resume + # from (PREVIEW_COMPLETED in particular). + prev_progress = session.state.upsert_task(task).progress + if prev_progress > progress: + log.debug( + f"Skipping {progress} for {task} because task progress {prev_progress=}" + ) + return task + + log.debug(f"Skipped until encountered {progress} for {task}") + return func(session, task, *args) + + return wrapper + + return decorator + + +def set_progress( + before: Progress, + after: Progress | None = None, + on_error: dict[type[Exception], Progress] = {}, +) -> Callable[ + [Callable[[Session, Task, *Arg], Ret]], + Callable[[Session, Task, *Arg], Ret | Task], +]: + """Decorater to set the progress of a task. + + Basically calls + `session.set_progress(task, progress)` + before the decorated function. + + Optionally, if an Exception is encountered while processing the task, + another progress can be set (to account for later pipeline-steps, which will not + take place). By default, we still raise. + + When combining, use after `skip_until()` decorator. + + Parameters + ---------- + before: Progress + Progress to set when (before) processing the task. + after: Progress, optional + Progress to set after processing the task. By default `before` is kept. + on_error: dict[type[Exception], Progress], optional + Mapping, which Progress to set for which Exception (instance check on the + Exception). By default, if an exception occurs, the `before` Progress is kept. + + Usage + ----- + ```python + @set_progress(Progress.READING_FILES) + def read_task(session: BaseSessionNew, task: ImportTask): + pass + + @set_progress( + Progress.LOOKING_UP_CANDIDATES, + on_Error={NoCandidatesFoundException: Progress.PREVIEW_COMPLETED} + ) + def lookup_candidates(session: BaseSessionNew, task: ImportTask): + pass + ``` + """ + + def decorator( + func: Callable[[Session, Task, *Arg], Ret], + ) -> Callable[[Session, Task, *Arg], Ret | Task]: + @wraps(func) + def wrapper(session: Session, task: Task, *args: *Arg) -> Ret | Task: + log.debug(f"Setting progress {before=} for {task}") + + session.set_task_progress(task, before) + try: + res = func(session, task, *args) + if after is not None: + log.debug(f"Setting progress {after=} for {task}") + session.set_task_progress(task, after) + return res + except Exception as e: + for e_to_handle, p in on_error.items(): + if isinstance(e, e_to_handle): + log.debug( + f"Setting progress to {p} after Exception for {task} {e}" + ) + session.set_task_progress(task, p) + raise + + return wrapper + + return decorator + + +class StageOrder(dict): + """An ordered dict of stages, mapping the stage name to its generator. + + Returned by each sessions `.stages` property. + """ + + def append( + self, + stage: Stage[ImportTask, Any], + name: str | None = None, + ): + """Append a stage to the Order.""" + + name = name or str(getattr(stage, "__name__", f"unknown_stage")) + if name in self.keys(): + raise ValueError(f"Stage with name {name} already exists.") + + self[name] = stage + + def insert( + self, + stage: Stage[ImportTask, Any], + name: str | None = None, + after: str | None = None, + before: str | None = None, + ): + """Insert a stage after or before another specific stage.""" + + if after is None and before is None or (after and before): + raise ValueError("Either `after` or `before` must be specified.") + + name = name or str(getattr(stage, "__name__", f"unknown_stage")) + if name in self.keys(): + raise ValueError(f"Stage with name {name} already exists.") + + # even for the OrderedDict there is no insert at index method, so just rebuild + keys = list(self.keys()) + values = list(self.values()) + + idx = keys.index(after or before) # type: ignore + if after: + idx += 1 + keys.insert(idx, name) + values.insert(idx, stage) + + self.clear() + self.update(zip(keys, values)) + + +# --------------------------------- Decorator -------------------------------- # + +Arg = TypeVarTuple("Arg") # args und kwargs +Ret = TypeVar("Ret") # return type +Task = TypeVar( + "Task", + bound=BeetsImportTask, +) # task + + +def stage( + func: Callable[[*Arg, Task], Ret | None], +): + """Decorate a function to become a simple stage. + + Yields a task and waits until the next task is sent to it. + + >>> @stage + ... def add(n, i): + ... return i + n + >>> pipe = Pipeline([ + ... iter([1, 2, 3]), + ... add(2), + ... ]) + >>> list(pipe.pull()) + [3, 4, 5] + """ + + # use the wraps decorator to lift function name etc, we use this in StageOrder class + @wraps(func) + def coro( + *args: *Arg, + ) -> Generator[Ret | Task | None, Task, None]: + # in some edge-cases we get no task. thus, we have to include the generic R + task: Task | Ret | Generator[Task] | None = None + while True: + if isgenerator(task): + for t in task: + task = yield t + else: + task = yield cast( + Task | None | Ret, task + ) # wait for send to arrive. the first next() always returns None + # yield task, call func which gives new task, yield new task in next() + task = cast(Task, task) # Slightly hacky, but we know task is a Task here + task = func(*(args + (task,))) + + return coro + + +def mutator_stage( + func: Callable[[*Arg, Task], Ret], name: str | None = None +) -> Callable[[*Arg], Generator[Ret | Task | None, Task, None]]: + """Decorate a function that manipulates items in a coroutine to become a simple stage. + + Yields a task and waits until the next task is sent to it. + + >>> @mutator_stage + ... def setkey(key, item): + ... item[key] = True + >>> pipe = Pipeline([ + ... iter([{'x': False}, {'a': False}]), + ... setkey('x'), + ... ]) + >>> list(pipe.pull()) + [{'x': True}, {'a': False, 'x': True}] + """ + + @wraps(func) + def coro( + *args: *Arg, + ) -> Generator[Ret | Task | None, Task, None]: + task = None + while True: + task = yield task # wait for send to arrive. the first next() always returns None + # perform function on task, and in next() send the same, modified task + # funcs prob. modify task in place? + func(*(args + (task,))) + + return coro + + +# --------------------------------- Producer --------------------------------- # + + +def read_tasks( + session: BaseSession, +): + """Read the files from the paths and generate tasks. + + If the session already has tasks yield them. + + Adapted closely from beets, but we do not need/support resuming and skipping + """ + + log.debug(f"Reading files") + + # Our Skip-check usually uses Progress, but here we do not have progress yet + # We want to catch the case when we resume a session, + # where we manually add old tasks from disk. + if len(session.state.tasks) > 0: + for task in session.state.tasks: + yield task + return + + for toppath in session.paths: + task_factory = ImportTaskFactory(toppath, session) + for task in task_factory.tasks(): + if isinstance(task, SentinelImportTask): + log.debug(f"Skipping {displayable_path(toppath)}") + continue + + task_state = session.state.upsert_task(task) + log.debug(f"Reading files from {displayable_path(toppath)}") + task_state.set_progress(Progress.READING_FILES) + yield task + + if not task_factory.imported: + log.warning(f"No files imported from {displayable_path(toppath)}") + + +# ----------------------------- Transform Stages ----------------------------- # + + +def __group(item: library.Item) -> tuple[str, str]: + return (item.albumartist or item.artist, item.album) + + +@stage +@skip_until(Progress.GROUPING_ALBUMS) +@set_progress(Progress.GROUPING_ALBUMS) +def group_albums( + session: BaseSession, + task: ImportTask, +) -> beets_pipeline.MultiMessage: + """ + Groups items of the task into albums using their metadata. + + The groups are identified using artist and album fields. + + Creates a new Task for each found album, and removes the old task (and task_state) + from the session. + """ + + # Split old task into multiple, and only keep new ones! + session.state.remove_task(task) + + tasks: list[ImportTask] = [] + sorted_items = sorted(task.items, key=__group) + for _, _items in itertools.groupby(sorted_items, __group): + items = list(_items) # Consume the iterator to a list + task = ImportTask(task.toppath, [i.path for i in items], items) + tasks += task.handle_created(session) + + # handle the beets-flask specific states + for task in tasks: + session.state.upsert_task(task) + + # FIXME: Not really sure we need this tbh, see also task.skip + # tasks.append(SentinelImportTask(task.toppath, task.paths)) + + return beets_pipeline.MultiMessage(tasks) + + +@mutator_stage +@skip_until(Progress.LOOKING_UP_CANDIDATES) +@set_progress( + Progress.LOOKING_UP_CANDIDATES, + on_error={NoCandidatesFoundException: Progress.PREVIEW_COMPLETED}, +) +def lookup_candidates( + session: BaseSession, + task: ImportTask, +): + """Performing the initial MusicBrainz lookup for an album. + + We tweaks this from upstream beets to not + call `task.lookup_candidates()` but instead `session.lookup_candidates(task)`, + with some extra logic + + This is more consistent, as it allows the logic + to be modified by each kind of session. + + Calls `task.lookup_candidates()`, + which sets attributes of the task: + - cur_artist # metadata in file + - cur_album + - candidates + - rec + """ + if task.skip: + # FIXME This gets duplicated a lot. We need a better + # abstraction. + return + + # FIXME: what happens with our new skip logic and plugins? + plugins.send("import_task_start", session=session, task=task) + + session.lookup_candidates(task) + + +@mutator_stage +@skip_until(Progress.IDENTIFYING_DUPLICATES) +@set_progress(before=Progress.IDENTIFYING_DUPLICATES, after=Progress.PREVIEW_COMPLETED) +def identify_duplicates( + session: BaseSession, + task: ImportTask, +): + """Stage to identify which candidates would be duplicates if imported.""" + if task.skip: + return + + session.identify_duplicates(task) + + +@stage +@skip_until(Progress.WAITING_FOR_USER_SELECTION) +@set_progress(Progress.WAITING_FOR_USER_SELECTION) +def user_query( + session: ImportSession, + task: ImportTask, +) -> ImportTask | beets_pipeline.MultiMessage | Literal["__PIPELINE_BUBBLE__"]: + """A coroutine for interfacing with the user about the tagging process. + + The coroutine accepts an ImportTask objects. It uses the + session's `choose_match` method to determine the `action` for + this task. Depending on the action additional stages are executed + and the processed task is yielded. + + It emits the ``import_task_choice`` event for plugins. Plugins have + access to the choice via the ``task.choice_flag`` property and may + choose to change it. + """ + + if task.skip: + return task + + if session.already_merged(task.paths): + # For some reason the type hinter does not like this without cast + return cast(Literal["__PIPELINE_BUBBLE__"], beets_pipeline.BUBBLE) + + # Ask the user for a choice. + # This calls session.choose_match(task)... but whyyy? + task.choose_match(session) + plugins.send("import_task_choice", session=session, task=task) + + # TODO: Import as singletons i.e. task.choice_flag is action.TRACKS + + # As albums: group items by albums and create task for each album + if task.choice_flag is Action.ALBUMS: + log.warning("This should never run via beets-flask!") + return _extend_pipeline( + [task], + group_albums(session), + lookup_candidates(session), + user_query(session), + ) + + # Note: this checks the global config for the default action. + found_duplicates = task.find_duplicates(session.lib) + # Different than vanilla beets, we do not use a stage for duplicate resolution, + # as it only calls the sessions method, anyway. + session.resolve_duplicate(task, found_duplicates) + + if task.should_merge_duplicates: + # Create a new task for tagging the current items + # and duplicates together + duplicate_items = task.duplicate_items(session.lib) + + # Duplicates would be reimported so make them look "fresh" + _freshen_items(duplicate_items) + duplicate_paths = [item.path for item in duplicate_items] + + # Record merged paths in the session so they are not reimported + session.mark_merged(duplicate_paths) + + merged_task = ImportTask( + None, task.paths + duplicate_paths, task.items + duplicate_items + ) + return _extend_pipeline( + [merged_task], lookup_candidates(session), user_query(session) + ) + + _apply_choice(session, task) + + return task + + +# Dynamicly set_progress for plugin name +# e.g. DetailedProgress(Progress.EARLY_IMPORT, plugin_name="my_plugin") +@mutator_stage +def plugin_stage( + session: BaseSession, + func: Callable[[BaseSession, ImportTask], None], + progress: ProgressState, + task: ImportTask, +): + # TODO: Skip if already progressed + session.set_task_progress(task, progress) + if task.skip: + return + + log.debug(f"Running plugin stage {func.__name__}") + func(session, task) + log.debug(f"Finished plugin stage {func.__name__}") + + task.reload() + log.debug(f"Reload plugin stage done {func.__name__}") + + +@mutator_stage +@skip_until(Progress.MATCH_THRESHOLD) +@set_progress( + Progress.MATCH_THRESHOLD, + on_error={NotImportedException: Progress.PREVIEW_COMPLETED}, +) +def match_threshold( + session: AutoImportSession, + task: ImportTask, +): + """Stage to determine if a match is good enough to be auto-imported.""" + if task.skip: + log.debug(f"Skipping task: {session=}, {task=}") + return task + + session.match_threshold(task) + + +# --------------------------------- Consumer --------------------------------- # + + +@stage +@skip_until(Progress.MANIPULATING_FILES) +@set_progress(before=Progress.MANIPULATING_FILES, after=Progress.IMPORT_COMPLETED) +def manipulate_files( + session: BaseSession, + task: ImportTask, +) -> ImportTask: + """A coroutine (pipeline stage) that performs necessary file. + + manipulations *after* items have been added to the library and + finalizes each task. + """ + if not task.skip: + if task.should_remove_duplicates: + task.remove_duplicates(session.lib) + + if session.config["copy"]: + operation = MoveOperation.COPY + # elif session.config["move"]: + # operation = MoveOperation.MOVE + # elif session.config["link"]: + # operation = MoveOperation.LINK + # elif session.config["hardlink"]: + # operation = MoveOperation.HARDLINK + # elif session.config["reflink"] == "auto": + # operation = MoveOperation.REFLINK_AUTO + # elif session.config["reflink"]: + # operation = MoveOperation.REFLINK + # else: + # operation = None + else: + log.warning( + "Beets-flask does not yet support other import modes than 'copy'. " + + "Please consider updating your config." + ) + operation = MoveOperation.COPY + + task.manipulate_files(session, operation, write=session.config["write"]) + + # Progress, cleanup, and event. + task.finalize(session) + + # Cleanup of duplicates. After successful import, we do not want the candidates + # to have duplicate-info anymore. The reason is that if we undo the import, + # there wont be any duplicates in the library. Also, we dont want to show + # the duplicate-badge for things that were imported. + # TODO: outlook: add new badges for what has been done with duplicates? + task_state = session.state.get_task_state_for_task_raise(task) + chosen_candidate = task_state.chosen_candidate_state + if chosen_candidate: + chosen_candidate.duplicate_ids = [] + # We remove duplicate info from all candidates. + # Although this causes the frontend to possibly not show the duplicate + # actions, that is fine: Upon re-import, we raise a duplicate action error, + # which is shown to the user, and then the duplicate actions will appear. + + if task_state.toppath is not None and len(session.state.task_states) == 1: + # If we have only one task and no toppath this indicates that we are + # operating on a temp directory we override the toppath + # to the session's folder path. + task.toppath = bytestring_path(session.state.folder_path) + + return task + + +@stage +def finalize( + session: BaseSession, + task: ImportTask, +) -> ImportTask: + """Give our custom sessions a way to do something at the very end.""" + session.finalize(task) + return task + + +def _apply_choice(session: ImportSession, task: ImportTask): + # tweaked version of beets apply_choices. + # we do not want to rely on the global config object + # (in particular, the set_fields, because we always want to set gui_import_id). + # see the original: + # from beets.importer import apply_choice + + if task.skip: + return + + # Change metadata. + if task.apply: + task.apply_metadata() + plugins.send("import_task_apply", session=session, task=task) + + # We need to reinsert items into the db if they were removed earlier + item: BeetsItem + for item in task.imported_items(): + item._db = session.lib + item.album_id = None + item.id = None + + task.add(session.lib) + task.set_fields(session.lib) + + # copy of core logic from set_fields() + items: list[BeetsItem] = task.imported_items() + + timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") + task_state = session.state.get_task_state_for_task_raise(task) + with session.lib.transaction(): + for item in items: + item.set_parse("gui_import_id", task_state.id) + item.set_parse("gui_import_date", timestamp) + item.store() + task.album.set_parse("gui_import_id", task_state.id) + task.album.set_parse("gui_import_date", timestamp) + task.album.store() + + +__all__ = [ + "read_tasks", + "group_albums", + "lookup_candidates", + "identify_duplicates", + "user_query", + "plugin_stage", + "manipulate_files", + "mark_tasks_completed", +] diff --git a/backend/beets_flask/importer/states.py b/backend/beets_flask/importer/states.py index 1c1d48ce..0c9bba34 100644 --- a/backend/beets_flask/importer/states.py +++ b/backend/beets_flask/importer/states.py @@ -1,892 +1,892 @@ -"""State classes represent the current state of an import session.""" - -from __future__ import annotations - -from abc import ABC -from collections.abc import Sequence -from dataclasses import dataclass -from datetime import datetime -from pathlib import Path -from typing import Literal, NotRequired, TypedDict, cast -from uuid import uuid4 as uuid - -import beets.ui.commands as uicommands -from beets import importer -from beets.ui import _open_library -from beets.util import bytestring_path, get_most_common_tags -from deprecated import deprecated - -from beets_flask.config import get_config -from beets_flask.disk import Archive, Folder -from beets_flask.importer.progress import ( - Progress, - ProgressState, - SerializedProgressState, -) -from beets_flask.importer.types import DuplicateAction -from beets_flask.server.exceptions import SerializedException -from beets_flask.utility import capture_stdout_stderr - -from .types import ( - AlbumInfo, - BeetsAlbum, - BeetsAlbumInfo, - BeetsAlbumMatch, - BeetsDistance, - BeetsImportTask, - BeetsItem, - BeetsLibrary, - BeetsTrackInfo, - BeetsTrackMatch, - ItemInfo, - TrackInfo, -) - - -class BaseState(ABC): - """Base class for all states. - - Some shared functionality, but mostly common attributes. - """ - - id: str - created_at: datetime - updated_at: datetime - - def __init__(self) -> None: - self.id = str(uuid()) - self.created_at = datetime.now() - self.updated_at = datetime.now() - - def serialize(self) -> SerializedBaseState: - """Serialize the state to a dictionary.""" - return { - "id": self.id, - "created_at": self.created_at, - "updated_at": self.updated_at, - } - - -@dataclass(init=False) -class SessionState(BaseState): - """Highest level state of an import session. - - Contains task (selection) states for each task. - """ - - id: str - _task_states: list[TaskState] - folder_path: Path - folder_hash: str - - # session-level buttons. continue from choose_match when not None - user_response: Literal["abort"] | Literal["apply"] | None = None - - # If a session run fails we store the exc here - # should be set to none whenever the session is started - exc: SerializedException | None = None - - def __init__(self, folder: Folder | Archive | Path) -> None: - super().__init__() - - # Alternate constructor is part of the SessionStateInDb class - self._task_states = [] - if isinstance(folder, str): - folder = Path(folder) - if isinstance(folder, Path): - if folder.is_dir(): - # If the path is a file, we assume it is an archive - folder = Folder.from_path(folder) - else: - folder = Archive.from_path(folder) - - # Why not just a folder object as member? - # -> We do not always want to compute the children (or save them to db) - self.folder_path = folder.path - self.folder_hash = folder.hash - - def __repr__(self) -> str: - return ( - f"SessionState:\n" - + f" * id={self.id}\n" - + f" * folder_path={self.folder_path}\n" - + f" * folder_hash={self.folder_hash}\n" - + f" * task_states={[ts.id for ts in self.task_states]}\n" - + f" * progress={self.progress}" - ) - - @property - @deprecated("Use the folder attribute instead!") - def path(self) -> Path: - return self.folder_path - - @property - def task_states(self): - return self._task_states - - @property - def task_state_ids(self): - return [s.id for s in self.task_states] - - @property - def tasks(self): - return [s.task for s in self.task_states] - - @property - def progress(self): - """The session progress is the loweset progress of all tasks.""" - if len(self.task_states) == 0: - return ProgressState(Progress.NOT_STARTED) - - return min([s.progress for s in self.task_states]) - - def get_task_state_for_task( - self, - task: BeetsImportTask, - ) -> TaskState | None: - """Get the task state for a given task. - - Returns None if not found. - """ - state: TaskState | None = None - for s in self.task_states: - if s.task == task: - state = s - break - return state - - def get_task_state_for_task_raise( - self, - task: BeetsImportTask, - ) -> TaskState: - """Get the task state for a given task. - - Raises ValueError if not found. - """ - state = self.get_task_state_for_task(task) - if state is None: - raise ValueError(f"Task {task} not found in session.") - return state - - def get_task_state_by_id(self, id: str) -> TaskState | None: - state: TaskState | None = None - for s in self.task_states: - if s.id == id: - state = s - break - return state - - def upsert_task( - self, - task: BeetsImportTask, - ) -> TaskState: - """Upsert selection state. - - If it does not exist yet it is created or updated - if entry exists. - """ - state = self.get_task_state_for_task(task) - - if state is None: - state = TaskState(task) - self._task_states.append(state) - - return state - - def remove_task(self, task: BeetsImportTask) -> None: - """Remove a task from the session state. - - If the task does not exist, nothing happens. - """ - state = self.get_task_state_for_task(task) - if state is not None: - self._task_states.remove(state) - - def remove_task_by_id(self, id: str) -> None: - """Remove a task from the session state by id. - - If the task does not exist, nothing happens. - """ - state = self.get_task_state_by_id(id) - if state is not None: - self._task_states.remove(state) - - def serialize(self) -> SerializedSessionState: - """JSON representation to match the frontend types.""" - - r = SerializedSessionState( - **super().serialize(), - folder_path=str(self.folder_path), - folder_hash=str(self.folder_hash), - tasks=[s.serialize() for s in self.task_states], - status=self.progress.serialize(), - ) - - if self.exc is not None: - r["exc"] = self.exc - - return r - - -@dataclass(init=False) -class TaskState(BaseState): - """State representation of a beets ImportTask. - - In the frontend, a selection of the available candidates in the task may be needed - from the user. Exposes some (typed) attributes of the task (e.g. toppath, paths, items) - Has a list of associated CandidateStates, that represent `matches` in beets. - """ - - progress: ProgressState - task: BeetsImportTask - candidate_states: list[CandidateState] - chosen_candidate_state_id: str | None = None - - # the completed state blocks the choose_match function - # of interactive sessions via our await_completion method - completed: bool = False - - # User choices and user input in interactive Session - # None if no choice has been made yet - # (or the frontend has not marked the default selection) - duplicate_action: DuplicateAction | None = None - - def __init__( - self, - task: BeetsImportTask, - ) -> None: - super().__init__() - # we might run into inconsistencies here, if candidates of the task - # change. but I do not know when or why they would. - self.task = task - self.candidate_states = [CandidateState(c, self) for c in self.task.candidates] - self.progress = ProgressState() - - def __repr__(self) -> str: - return ( - f"TaskState:\n" - + f" * id={self.id}\n" - + f" * candidate_states={[ts.id for ts in self.candidate_states]}\n" - + f" * chosen_candidate_state_id={self.chosen_candidate_state_id}\n" - + f" * progress={self.progress}\n" - + f" * completed={self.completed}\n" - + f" * toppath={self.toppath}\n" - ) - - @property - def candidates( - self, - ) -> Sequence[BeetsAlbumMatch | BeetsTrackMatch]: - """Task candidates, i.e. possible matches to choose from.""" - return self.task.candidates - - @property - def asis_candidate_id(self) -> str: - """Id of the asis candidate.""" - return "asis-" + str(self.id) - - @property - def asis_candidate(self) -> CandidateState: - """Get the asis candidate state.""" - return CandidateState.asis_candidate(self) - - def add_candidates( - self, - candidates: Sequence[BeetsAlbumMatch | BeetsTrackMatch], - insert_at: int = 0, - ) -> list[CandidateState]: - """Add new candidates to the selection state.""" - if len(self.task.candidates) == 0 or len(self.candidate_states) == 0: - insert_at = 0 - - # task.candidates is a sequence and thus immutable - _ = list(self.task.candidates) - _[insert_at:insert_at] = candidates - self.task.candidates = _ - - new_states = [CandidateState(c, self) for c in candidates] - self.candidate_states[insert_at:insert_at] = new_states - return new_states - - def get_candidate_state_by_id(self, id: str) -> CandidateState | None: - """Get candidate state by id.""" - for c in self.candidate_states + [self.asis_candidate]: - if c.id == id: - return c - return None - - @property - def chosen_candidate_state(self) -> CandidateState | None: - if self.chosen_candidate_state_id is None: - return None - return self.get_candidate_state_by_id(self.chosen_candidate_state_id) - - @property - def toppath(self) -> Path | None: - """Highest-level (common) folder holding music files.""" - if self.task.toppath is not None and isinstance(self.task.toppath, bytes): - return Path(self.task.toppath.decode("utf-8")) - return None - - @property - def paths(self) -> list[Path]: - """Lowest-level folders holding music files.""" - return [Path(p.decode("utf-8")) for p in self.task.paths] - - @property - def item_paths_before_import(self) -> list[Path]: - """Explicit paths to all media files that would be imported.""" - if self.toppath is None: - return [] - if self.toppath.is_file(): - return [self.toppath] - - items: list[bytes] = [] - for _, i in importer.tasks.albums_in_dir(bytestring_path(self.toppath)): - # the generator returns a nested list of the outer diretories - # and file paths. thus, extend and then cast - items.extend(i) - - return [Path(i.decode("utf-8")) for i in items] - - @property - def items(self) -> list[BeetsItem]: - """Items (representing music files on disk) of the associated task.""" - return [item for item in self.task.items] - - @property - def items_minimal(self) -> list[ItemInfo]: - """Items of the associated task as MinimalItemAndTrackInfo.""" - return [ItemInfo.from_beets(i) for i in self.task.items] - - @property - def best_candidate_state(self) -> CandidateState | None: - """Returns the best candidate of this task (never asis).""" - best = None - for candidate in self.candidate_states: - if best is None or candidate.distance < best.distance.distance: - best = candidate - return best - - @property - def choice_flag(self) -> importer.tasks.Action | None: - return self.task.choice_flag - - @choice_flag.setter - def choice_flag(self, value: importer.tasks.Action | None): - self.task.choice_flag = value - - @property - def current_metadata(self) -> Metadata: - """Current metadata of the task. - - This is the metadata of the music files on disk. - (In a beets context, cur_artist and cur_album) - """ - likelies, _ = get_most_common_tags(self.items) - return Metadata(**{k: str(v) for k, v in likelies.items()}) # type: ignore[typeddict-item] - - # ---------------------------------------------------------------------------- # - - def serialize(self) -> SerializedTaskState: - """JSON representation to match the frontend types.""" - return SerializedTaskState( - **super().serialize(), - items=self.items_minimal, - candidates=[c.serialize() for c in self.candidate_states], - asis_candidate=self.asis_candidate.serialize(), - current_metadata=self.current_metadata, - # TODO: maybe we can merge current_metadata (which is cur_artist/album in - # old beets) into the asis_candidate - chosen_candidate_id=self.chosen_candidate_state_id, - duplicate_action=self.duplicate_action, - completed=self.completed, - toppath=str(self.toppath), - paths=[str(p) for p in self.paths], - ) - - def set_progress(self, progress: ProgressState | Progress | str) -> None: - """Set the progress of the task. - - If string is given it is set as the message of the current progress. - """ - if isinstance(progress, ProgressState): - self.progress = progress - elif isinstance(progress, Progress): - self.progress = ProgressState(progress) - elif isinstance(progress, str): - # just convenience for debugging should not be used in production - self.progress.message = progress - else: - raise ValueError(f"Unknown progress type: {progress}") - - -@dataclass(init=False) -class CandidateState(BaseState): - """ - State representation of a single candidate (match) for an import task. - - Can represent an album (self.type == "album") or a track (self.type == "track"). - Keeps a reference to the associated SelectionState, so we can access the beets task. - Exposes some attributes of the task and match - - Note: currently only tested for album matches. - """ - - id: str - duplicate_ids: list[str] # Beets ids of duplicates in the library (album) - match: BeetsAlbumMatch | BeetsTrackMatch - # Reference upwards - task_state: TaskState - - _mapping: dict[int, int] # index mapping from items to tracks - - def __init__( - self, - match: BeetsAlbumMatch | BeetsTrackMatch, - task_state: TaskState, - mapping: dict[int, int] | None = None, - ) -> None: - super().__init__() - self.match = match - self.duplicate_ids = [] # checked and set by session - self.task_state = task_state - - # current_mapping is dynamic and looks at the match to generate integer / index mapping - # this can cause problems, when loading a previously imported candidate from the db - # as, in this case, the mapping is wrong and _index_mapping will fail. - # we take care of this by manually overwriting when constructing from the db. - self._mapping = mapping or self.current_mapping - - def __repr__(self) -> str: - return ( - f"CandidateState:\n" - + f" * id={self.id}\n" - + f" * match={self.match.info.album}\n" - + f" * task_state_id={self.task_state.id}\n" - + f" * distance={self.distance}\n" - + f" * penalties={self.penalties}\n" - + f" * {len(self.items)=}\n" - + f" * {len(self.tracks)=}\n" - + f" * mapping={self.mapping}\n" - ) - - @property - def type(self) -> Literal["album", "track"]: - if isinstance(self.match, BeetsAlbumMatch): - return "album" - elif isinstance(self.match, BeetsTrackMatch): - return "track" - else: - raise ValueError("Unknown type") - - @property - def diff_preview(self) -> str: - """Diff preview of the match to the current meta data.""" - out, err, _ = capture_stdout_stderr( - uicommands.show_change, - self.task_state.task.cur_artist, - self.task_state.task.cur_album, - self.match, - ) - res = out.lstrip("\n") - if len(err) > 0: - res += f"\n\nError: {err}" - return res - - @classmethod - def asis_candidate(cls, task_state: TaskState) -> CandidateState: - """ - Alternate constructor for an asis import option. - - We mock the album match to display - current meta data in the frontend. - This is pretty much duct-tape. - """ - items: list[BeetsItem] = task_state.task.items - - # FIXME: we do this lookup twice, once here and once in current_metadata - if len(items) > 0: - info, _ = get_most_common_tags(items) - else: - info = {} - info["data_source"] = "asis" - info["data_url"] = f"file://{task_state.toppath}" - - def _generate_kwargs(item): - kwargs = {} - # before import the keys are in _dirty, - # after import in _fields - for key in list(item._dirty) + list(item._fields): - val = getattr(item, key) - if val is not None and val != "": - kwargs[key] = val - # tracks use index, items use track, and beets diff preview crashes without index - kwargs["index"] = item.track or 0 - return kwargs - - tracks = [BeetsTrackInfo(**_generate_kwargs(i)) for i in items] - - match = BeetsAlbumMatch( - distance=BeetsDistance(), - info=BeetsAlbumInfo( - tracks=tracks, - **info, - ), - extra_items=[], - extra_tracks=[], - mapping={i: tracks[idx] for idx, i in enumerate(items)}, - ) - candidate = cls(match=match, task_state=task_state) - candidate.id = task_state.asis_candidate_id - # As the asis candidate state is not maintained we not to - # recheck if it is a duplicate - candidate.identify_duplicates() - - return candidate - - # --------------------- Helper to lift / unnset from match to -------------------- # - @property - def cur_artist(self) -> str: - """Current artist, usually the meta data of the music files. - - Named to be consistent with beets. - """ - return str(self.task_state.task.cur_artist) - - @property - def cur_album(self) -> str: - """Current album, usually the meta data of the music files. - - Named to be consistent with beets. - """ - return str(self.task_state.task.cur_album) - - @property - def artist(self) -> str | None: - """Artist of the match.""" - return self.match.info.artist - - @property - def album(self) -> str | None: - """Album of the match.""" - return self.match.info.album - - @property - def items(self) -> list[BeetsItem]: - """In beets, items refers to the music files on disk. - - Tracks correspond to an online match, (or the library?) - """ - return self.task_state.task.items - - @property - def tracks(self) -> list[BeetsTrackInfo]: - """Tracks of the match (usually tracks in online match).""" - if isinstance(self.match, BeetsAlbumMatch): - return self.match.info.tracks - return [self.match.info] - - @property - def distance(self) -> BeetsDistance: - """Distance of the match to the current meta data. - - Metadata may be from the task i.e. album or track. - """ - return self.match.distance - - @property - def penalties(self) -> list[str]: - penalties = list(self.match.distance.keys()) - - # renaming for consistency! - # Beets has a somewhat unintuitive naming: - # "items" are things on disk - # "tracks" are meta-data i.e. online. - # but penalties: - # "unmatched_tracks" have online not on disk - # "missing_tracks" have on disk, not online - - # beets object | beets penalty | beets_flask - # -------------|------------------|------------ - # extra_items | unmatched_tracks | extra_items - # extra_tracks | missing_tracks | extra_tracks - - penalties = [ - p.replace("unmatched_tracks", "extra_items").replace( - "missing_tracks", "extra_tracks" - ) - for p in penalties - ] - - return list(penalties) - - @property - def num_tracks(self) -> int: - """Number of tracks in the match (usually tracks in online match).""" - if isinstance(self.match, BeetsAlbumMatch): - return len(self.match.info.tracks) - return 1 - - @property - def num_items(self) -> int: - """Number of items in the task (usually files on disk).""" - return len(self.items) - - @property - def url(self) -> str | None: - """URL of the match.""" - if isinstance(self.match, BeetsAlbumMatch): - try: - return self.match.info.data_url - except AttributeError: # not set in the match - return None - - return None - - @property - def is_asis(self) -> bool: - """Returns True if this is an "as is" candidate.""" - return self.id.startswith("asis-") - - @property - def mapping(self) -> dict[int, int]: - return self._mapping - - @property - def current_mapping(self) -> dict[int, int]: - """Get the current mapping from items to tracks, calculated from the match.""" - if isinstance(self.match, BeetsAlbumMatch): - return _index_mapping( - self.match.mapping, - self.items, - self.tracks, - ) - - raise ValueError("Current mapping only available for album matches.") - - # ------------------------------------ utility ----------------------------------- # - - def identify_duplicates(self, lib: BeetsLibrary | None = None) -> list[BeetsAlbum]: - """Find duplicates. - - Copy of beets' `task.find_duplicates` but works on any candidates' match. - - # FIXME: Tracks are not checked for duplicates. Tbh noone cares about tracks anyways - """ - if lib is None: - lib = _open_library(get_config()) - - info = self.match.info.copy() - info["albumartist"] = info["artist"] - - if info["artist"] is None: - # As-is import with no artist. Skip check. - return [] - - # Construct a query to find duplicates with this metadata. We - # use a temporary Album object to generate any computed fields. - tmp_album = BeetsAlbum(lib, **info) - keys: list[str] = cast( - list[str], - get_config()["import"]["duplicate_keys"]["album"].as_str_seq() or [], - ) - dup_query = tmp_album.duplicates_query(keys) - - # Re-Importing: Don't count albums with the same files as duplicates. - task_paths = {i.path for i in self.task_state.task.items if i} - - duplicates: list[BeetsAlbum] = [] - for album in lib.albums(dup_query): - # Check whether the album paths are all present in the task - # i.e. album is being completely re-imported by the task, - # in which case it is not a duplicate (will be replaced). - album_paths = {i.path for i in album.items()} - if not (album_paths <= task_paths): - duplicates.append(album) - - # Write duplicates information! - self.duplicate_ids = [d.id for d in duplicates] - - return duplicates - - @property - def has_duplicates_in_library(self) -> bool: - """Returns False, either if no duplicates found, or you have not checked yet. - - Call `identify_duplicates` first to ensure this works. - """ - return len(self.duplicate_ids) > 0 - - def serialize(self) -> SerializedCandidateState: - """JSON representation to match the frontend types.""" - # we lift the match.info to reduce nesting in the frontend. - info: TrackInfo | AlbumInfo - tracks: list[TrackInfo] - mapping: dict[int, int] = {} - - if isinstance(self.match.info, BeetsTrackInfo): - # This hardly ever happens, we might support this more in the future - info = TrackInfo.from_beets(self.match.info) - tracks = [TrackInfo.from_beets(self.match.info)] - elif isinstance(self.match.info, BeetsAlbumInfo): - info = AlbumInfo.from_beets(self.match.info) - - # Map beets types to our types, allows serialization magic - tracks = [TrackInfo.from_beets(track) for track in self.match.info.tracks] - - # mapping = _index_mapping( - # self.match.mapping, # type: ignore - # self.items, - # self.tracks, - # ) - mapping = self.mapping - - else: - raise ValueError(f"Unknown type of matchinfo {type(self.match.info)}") - - res = SerializedCandidateState( - **super().serialize(), - penalties=self.penalties, - duplicate_ids=self.duplicate_ids, - type=self.type, - distance=self.distance.distance, - info=info, - tracks=tracks, - mapping=mapping, - ) - - return res - - -def _index_mapping( - mapping: dict[BeetsItem, BeetsTrackInfo], - items: list[BeetsItem], - tracks: list[BeetsTrackInfo], -) -> dict[int, int]: - """Helper to create an index mapping from items to tracks. - - the mapping of a beets albummatch uses objects, but we don not want - to send them over redundantly. convert to an index mapping, - where first index is in self.items, and second is in self.match.info.tracks - - This is used to serialize the mapping of a candidate state. - """ - - # log.debug(f"items: {[i.title for i in items]}") - - idxs = [] - tdxs = [] - for item, track in mapping.items(): - # log.debug(f"track: {track.index} {track.track_alt} {track.title}") - # log.debug(f"item: {item.track} {item.title}") - - # compare items via paths, and tracks via full dicts - # we used to compare via track_id, but this might be None. - found_idx = found_tdx = None - for idx, _ in enumerate(items): - if item.path == items[idx].path: - found_idx = idx - break - for tdx, _ in enumerate(tracks): - if track == tracks[tdx]: - found_tdx = tdx - break - idxs.append(found_idx) - tdxs.append(found_tdx) - - if None in idxs or None in tdxs: - # breakpoint() - raise ValueError( - f"Index mapping failed: {idxs=} {tdxs=} {len(items)=} {len(tracks)=}" - ) - - # ignore type for mypy, we have checked that its not None! - res: dict[int, int] = {idx: tdx for idx, tdx in zip(idxs, tdxs)} # type: ignore[misc] - # log.debug(f"Index mapping: {res}") - - return res - - -# ---------------------------- Serialization types --------------------------- # -# Used for getting typehints in the frontend. I.e. we generate the types from -# these typed dicts! See the generate_types.py script for more information. - - -class Metadata(TypedDict): - """Returned from current_metadata(). - - FIXME: I think this should not be defined here! - """ - - artist: str | None - album: str | None - albumartist: str | None - year: str | None - disctotal: str | None - mb_albumid: str | None - label: str | None - barcode: str | None - catalognum: str | None - country: str | None - media: str | None - albumdisambig: str | None - - -class SerializedBaseState(TypedDict): - """Serialized base state. - - This is used to serialize the base state to a dictionary. - """ - - id: str - created_at: datetime - updated_at: datetime - - -class SerializedSessionState(SerializedBaseState): - folder_path: str - folder_hash: str - tasks: list[SerializedTaskState] - status: SerializedProgressState - - exc: NotRequired[SerializedException | None] - - -class SerializedTaskState(SerializedBaseState): - items: list[ItemInfo] - current_metadata: Metadata - - # Fetched data - candidates: list[SerializedCandidateState] - asis_candidate: SerializedCandidateState - - duplicate_action: str | None - chosen_candidate_id: str | None - completed: bool - toppath: str | None - paths: list[str] - - -class SerializedCandidateState(SerializedBaseState): - duplicate_ids: list[str] - type: str - - penalties: list[str] - distance: float - - info: TrackInfo | ItemInfo | AlbumInfo - - # Mapping from items to tracks index based - mapping: dict[int, int] - tracks: list[TrackInfo] - - -__all__ = [ - "SessionState", - "TaskState", - "CandidateState", - "SerializedSessionState", - "SerializedTaskState", - "SerializedCandidateState", -] +"""State classes represent the current state of an import session.""" + +from __future__ import annotations + +from abc import ABC +from collections.abc import Sequence +from dataclasses import dataclass +from datetime import datetime +from pathlib import Path +from typing import Literal, NotRequired, TypedDict, cast +from uuid import uuid4 as uuid + +import beets.ui.commands as uicommands +from beets import importer +from beets.ui import _open_library +from beets.util import bytestring_path, get_most_common_tags +from deprecated import deprecated + +from beets_flask.config import get_config +from beets_flask.disk import Archive, Folder +from beets_flask.importer.progress import ( + Progress, + ProgressState, + SerializedProgressState, +) +from beets_flask.importer.types import DuplicateAction +from beets_flask.server.exceptions import SerializedException +from beets_flask.utility import capture_stdout_stderr + +from .types import ( + AlbumInfo, + BeetsAlbum, + BeetsAlbumInfo, + BeetsAlbumMatch, + BeetsDistance, + BeetsImportTask, + BeetsItem, + BeetsLibrary, + BeetsTrackInfo, + BeetsTrackMatch, + ItemInfo, + TrackInfo, +) + + +class BaseState(ABC): + """Base class for all states. + + Some shared functionality, but mostly common attributes. + """ + + id: str + created_at: datetime + updated_at: datetime + + def __init__(self) -> None: + self.id = str(uuid()) + self.created_at = datetime.now() + self.updated_at = datetime.now() + + def serialize(self) -> SerializedBaseState: + """Serialize the state to a dictionary.""" + return { + "id": self.id, + "created_at": self.created_at, + "updated_at": self.updated_at, + } + + +@dataclass(init=False) +class SessionState(BaseState): + """Highest level state of an import session. + + Contains task (selection) states for each task. + """ + + id: str + _task_states: list[TaskState] + folder_path: Path + folder_hash: str + + # session-level buttons. continue from choose_match when not None + user_response: Literal["abort"] | Literal["apply"] | None = None + + # If a session run fails we store the exc here + # should be set to none whenever the session is started + exc: SerializedException | None = None + + def __init__(self, folder: Folder | Archive | Path) -> None: + super().__init__() + + # Alternate constructor is part of the SessionStateInDb class + self._task_states = [] + if isinstance(folder, str): + folder = Path(folder) + if isinstance(folder, Path): + if folder.is_dir(): + # If the path is a file, we assume it is an archive + folder = Folder.from_path(folder) + else: + folder = Archive.from_path(folder) + + # Why not just a folder object as member? + # -> We do not always want to compute the children (or save them to db) + self.folder_path = folder.path + self.folder_hash = folder.hash + + def __repr__(self) -> str: + return ( + f"SessionState:\n" + + f" * id={self.id}\n" + + f" * folder_path={self.folder_path}\n" + + f" * folder_hash={self.folder_hash}\n" + + f" * task_states={[ts.id for ts in self.task_states]}\n" + + f" * progress={self.progress}" + ) + + @property + @deprecated("Use the folder attribute instead!") + def path(self) -> Path: + return self.folder_path + + @property + def task_states(self): + return self._task_states + + @property + def task_state_ids(self): + return [s.id for s in self.task_states] + + @property + def tasks(self): + return [s.task for s in self.task_states] + + @property + def progress(self): + """The session progress is the loweset progress of all tasks.""" + if len(self.task_states) == 0: + return ProgressState(Progress.NOT_STARTED) + + return min([s.progress for s in self.task_states]) + + def get_task_state_for_task( + self, + task: BeetsImportTask, + ) -> TaskState | None: + """Get the task state for a given task. + + Returns None if not found. + """ + state: TaskState | None = None + for s in self.task_states: + if s.task == task: + state = s + break + return state + + def get_task_state_for_task_raise( + self, + task: BeetsImportTask, + ) -> TaskState: + """Get the task state for a given task. + + Raises ValueError if not found. + """ + state = self.get_task_state_for_task(task) + if state is None: + raise ValueError(f"Task {task} not found in session.") + return state + + def get_task_state_by_id(self, id: str) -> TaskState | None: + state: TaskState | None = None + for s in self.task_states: + if s.id == id: + state = s + break + return state + + def upsert_task( + self, + task: BeetsImportTask, + ) -> TaskState: + """Upsert selection state. + + If it does not exist yet it is created or updated + if entry exists. + """ + state = self.get_task_state_for_task(task) + + if state is None: + state = TaskState(task) + self._task_states.append(state) + + return state + + def remove_task(self, task: BeetsImportTask) -> None: + """Remove a task from the session state. + + If the task does not exist, nothing happens. + """ + state = self.get_task_state_for_task(task) + if state is not None: + self._task_states.remove(state) + + def remove_task_by_id(self, id: str) -> None: + """Remove a task from the session state by id. + + If the task does not exist, nothing happens. + """ + state = self.get_task_state_by_id(id) + if state is not None: + self._task_states.remove(state) + + def serialize(self) -> SerializedSessionState: + """JSON representation to match the frontend types.""" + + r = SerializedSessionState( + **super().serialize(), + folder_path=str(self.folder_path), + folder_hash=str(self.folder_hash), + tasks=[s.serialize() for s in self.task_states], + status=self.progress.serialize(), + ) + + if self.exc is not None: + r["exc"] = self.exc + + return r + + +@dataclass(init=False) +class TaskState(BaseState): + """State representation of a beets ImportTask. + + In the frontend, a selection of the available candidates in the task may be needed + from the user. Exposes some (typed) attributes of the task (e.g. toppath, paths, items) + Has a list of associated CandidateStates, that represent `matches` in beets. + """ + + progress: ProgressState + task: BeetsImportTask + candidate_states: list[CandidateState] + chosen_candidate_state_id: str | None = None + + # the completed state blocks the choose_match function + # of interactive sessions via our await_completion method + completed: bool = False + + # User choices and user input in interactive Session + # None if no choice has been made yet + # (or the frontend has not marked the default selection) + duplicate_action: DuplicateAction | None = None + + def __init__( + self, + task: BeetsImportTask, + ) -> None: + super().__init__() + # we might run into inconsistencies here, if candidates of the task + # change. but I do not know when or why they would. + self.task = task + self.candidate_states = [CandidateState(c, self) for c in self.task.candidates] + self.progress = ProgressState() + + def __repr__(self) -> str: + return ( + f"TaskState:\n" + + f" * id={self.id}\n" + + f" * candidate_states={[ts.id for ts in self.candidate_states]}\n" + + f" * chosen_candidate_state_id={self.chosen_candidate_state_id}\n" + + f" * progress={self.progress}\n" + + f" * completed={self.completed}\n" + + f" * toppath={self.toppath}\n" + ) + + @property + def candidates( + self, + ) -> Sequence[BeetsAlbumMatch | BeetsTrackMatch]: + """Task candidates, i.e. possible matches to choose from.""" + return self.task.candidates + + @property + def asis_candidate_id(self) -> str: + """Id of the asis candidate.""" + return "asis-" + str(self.id) + + @property + def asis_candidate(self) -> CandidateState: + """Get the asis candidate state.""" + return CandidateState.asis_candidate(self) + + def add_candidates( + self, + candidates: Sequence[BeetsAlbumMatch | BeetsTrackMatch], + insert_at: int = 0, + ) -> list[CandidateState]: + """Add new candidates to the selection state.""" + if len(self.task.candidates) == 0 or len(self.candidate_states) == 0: + insert_at = 0 + + # task.candidates is a sequence and thus immutable + _ = list(self.task.candidates) + _[insert_at:insert_at] = candidates + self.task.candidates = _ + + new_states = [CandidateState(c, self) for c in candidates] + self.candidate_states[insert_at:insert_at] = new_states + return new_states + + def get_candidate_state_by_id(self, id: str) -> CandidateState | None: + """Get candidate state by id.""" + for c in self.candidate_states + [self.asis_candidate]: + if c.id == id: + return c + return None + + @property + def chosen_candidate_state(self) -> CandidateState | None: + if self.chosen_candidate_state_id is None: + return None + return self.get_candidate_state_by_id(self.chosen_candidate_state_id) + + @property + def toppath(self) -> Path | None: + """Highest-level (common) folder holding music files.""" + if self.task.toppath is not None and isinstance(self.task.toppath, bytes): + return Path(self.task.toppath.decode("utf-8")) + return None + + @property + def paths(self) -> list[Path]: + """Lowest-level folders holding music files.""" + return [Path(p.decode("utf-8")) for p in self.task.paths] + + @property + def item_paths_before_import(self) -> list[Path]: + """Explicit paths to all media files that would be imported.""" + if self.toppath is None: + return [] + if self.toppath.is_file(): + return [self.toppath] + + items: list[bytes] = [] + for _, i in importer.tasks.albums_in_dir(bytestring_path(self.toppath)): + # the generator returns a nested list of the outer diretories + # and file paths. thus, extend and then cast + items.extend(i) + + return [Path(i.decode("utf-8")) for i in items] + + @property + def items(self) -> list[BeetsItem]: + """Items (representing music files on disk) of the associated task.""" + return [item for item in self.task.items] + + @property + def items_minimal(self) -> list[ItemInfo]: + """Items of the associated task as MinimalItemAndTrackInfo.""" + return [ItemInfo.from_beets(i) for i in self.task.items] + + @property + def best_candidate_state(self) -> CandidateState | None: + """Returns the best candidate of this task (never asis).""" + best = None + for candidate in self.candidate_states: + if best is None or candidate.distance < best.distance.distance: + best = candidate + return best + + @property + def choice_flag(self) -> importer.tasks.Action | None: + return self.task.choice_flag + + @choice_flag.setter + def choice_flag(self, value: importer.tasks.Action | None): + self.task.choice_flag = value + + @property + def current_metadata(self) -> Metadata: + """Current metadata of the task. + + This is the metadata of the music files on disk. + (In a beets context, cur_artist and cur_album) + """ + likelies, _ = get_most_common_tags(self.items) + return Metadata(**{k: str(v) for k, v in likelies.items()}) # type: ignore[typeddict-item] + + # ---------------------------------------------------------------------------- # + + def serialize(self) -> SerializedTaskState: + """JSON representation to match the frontend types.""" + return SerializedTaskState( + **super().serialize(), + items=self.items_minimal, + candidates=[c.serialize() for c in self.candidate_states], + asis_candidate=self.asis_candidate.serialize(), + current_metadata=self.current_metadata, + # TODO: maybe we can merge current_metadata (which is cur_artist/album in + # old beets) into the asis_candidate + chosen_candidate_id=self.chosen_candidate_state_id, + duplicate_action=self.duplicate_action, + completed=self.completed, + toppath=str(self.toppath), + paths=[str(p) for p in self.paths], + ) + + def set_progress(self, progress: ProgressState | Progress | str) -> None: + """Set the progress of the task. + + If string is given it is set as the message of the current progress. + """ + if isinstance(progress, ProgressState): + self.progress = progress + elif isinstance(progress, Progress): + self.progress = ProgressState(progress) + elif isinstance(progress, str): + # just convenience for debugging should not be used in production + self.progress.message = progress + else: + raise ValueError(f"Unknown progress type: {progress}") + + +@dataclass(init=False) +class CandidateState(BaseState): + """ + State representation of a single candidate (match) for an import task. + + Can represent an album (self.type == "album") or a track (self.type == "track"). + Keeps a reference to the associated SelectionState, so we can access the beets task. + Exposes some attributes of the task and match + + Note: currently only tested for album matches. + """ + + id: str + duplicate_ids: list[str] # Beets ids of duplicates in the library (album) + match: BeetsAlbumMatch | BeetsTrackMatch + # Reference upwards + task_state: TaskState + + _mapping: dict[int, int] # index mapping from items to tracks + + def __init__( + self, + match: BeetsAlbumMatch | BeetsTrackMatch, + task_state: TaskState, + mapping: dict[int, int] | None = None, + ) -> None: + super().__init__() + self.match = match + self.duplicate_ids = [] # checked and set by session + self.task_state = task_state + + # current_mapping is dynamic and looks at the match to generate integer / index mapping + # this can cause problems, when loading a previously imported candidate from the db + # as, in this case, the mapping is wrong and _index_mapping will fail. + # we take care of this by manually overwriting when constructing from the db. + self._mapping = mapping or self.current_mapping + + def __repr__(self) -> str: + return ( + f"CandidateState:\n" + + f" * id={self.id}\n" + + f" * match={self.match.info.album}\n" + + f" * task_state_id={self.task_state.id}\n" + + f" * distance={self.distance}\n" + + f" * penalties={self.penalties}\n" + + f" * {len(self.items)=}\n" + + f" * {len(self.tracks)=}\n" + + f" * mapping={self.mapping}\n" + ) + + @property + def type(self) -> Literal["album", "track"]: + if isinstance(self.match, BeetsAlbumMatch): + return "album" + elif isinstance(self.match, BeetsTrackMatch): + return "track" + else: + raise ValueError("Unknown type") + + @property + def diff_preview(self) -> str: + """Diff preview of the match to the current meta data.""" + out, err, _ = capture_stdout_stderr( + uicommands.show_change, + self.task_state.task.cur_artist, + self.task_state.task.cur_album, + self.match, + ) + res = out.lstrip("\n") + if len(err) > 0: + res += f"\n\nError: {err}" + return res + + @classmethod + def asis_candidate(cls, task_state: TaskState) -> CandidateState: + """ + Alternate constructor for an asis import option. + + We mock the album match to display + current meta data in the frontend. + This is pretty much duct-tape. + """ + items: list[BeetsItem] = task_state.task.items + + # FIXME: we do this lookup twice, once here and once in current_metadata + if len(items) > 0: + info, _ = get_most_common_tags(items) + else: + info = {} + info["data_source"] = "asis" + info["data_url"] = f"file://{task_state.toppath}" + + def _generate_kwargs(item): + kwargs = {} + # before import the keys are in _dirty, + # after import in _fields + for key in list(item._dirty) + list(item._fields): + val = getattr(item, key) + if val is not None and val != "": + kwargs[key] = val + # tracks use index, items use track, and beets diff preview crashes without index + kwargs["index"] = item.track or 0 + return kwargs + + tracks = [BeetsTrackInfo(**_generate_kwargs(i)) for i in items] + + match = BeetsAlbumMatch( + distance=BeetsDistance(), + info=BeetsAlbumInfo( + tracks=tracks, + **info, + ), + extra_items=[], + extra_tracks=[], + mapping={i: tracks[idx] for idx, i in enumerate(items)}, + ) + candidate = cls(match=match, task_state=task_state) + candidate.id = task_state.asis_candidate_id + # As the asis candidate state is not maintained we not to + # recheck if it is a duplicate + candidate.identify_duplicates() + + return candidate + + # --------------------- Helper to lift / unnset from match to -------------------- # + @property + def cur_artist(self) -> str: + """Current artist, usually the meta data of the music files. + + Named to be consistent with beets. + """ + return str(self.task_state.task.cur_artist) + + @property + def cur_album(self) -> str: + """Current album, usually the meta data of the music files. + + Named to be consistent with beets. + """ + return str(self.task_state.task.cur_album) + + @property + def artist(self) -> str | None: + """Artist of the match.""" + return self.match.info.artist + + @property + def album(self) -> str | None: + """Album of the match.""" + return self.match.info.album + + @property + def items(self) -> list[BeetsItem]: + """In beets, items refers to the music files on disk. + + Tracks correspond to an online match, (or the library?) + """ + return self.task_state.task.items + + @property + def tracks(self) -> list[BeetsTrackInfo]: + """Tracks of the match (usually tracks in online match).""" + if isinstance(self.match, BeetsAlbumMatch): + return self.match.info.tracks + return [self.match.info] + + @property + def distance(self) -> BeetsDistance: + """Distance of the match to the current meta data. + + Metadata may be from the task i.e. album or track. + """ + return self.match.distance + + @property + def penalties(self) -> list[str]: + penalties = list(self.match.distance.keys()) + + # renaming for consistency! + # Beets has a somewhat unintuitive naming: + # "items" are things on disk + # "tracks" are meta-data i.e. online. + # but penalties: + # "unmatched_tracks" have online not on disk + # "missing_tracks" have on disk, not online + + # beets object | beets penalty | beets_flask + # -------------|------------------|------------ + # extra_items | unmatched_tracks | extra_items + # extra_tracks | missing_tracks | extra_tracks + + penalties = [ + p.replace("unmatched_tracks", "extra_items").replace( + "missing_tracks", "extra_tracks" + ) + for p in penalties + ] + + return list(penalties) + + @property + def num_tracks(self) -> int: + """Number of tracks in the match (usually tracks in online match).""" + if isinstance(self.match, BeetsAlbumMatch): + return len(self.match.info.tracks) + return 1 + + @property + def num_items(self) -> int: + """Number of items in the task (usually files on disk).""" + return len(self.items) + + @property + def url(self) -> str | None: + """URL of the match.""" + if isinstance(self.match, BeetsAlbumMatch): + try: + return self.match.info.data_url + except AttributeError: # not set in the match + return None + + return None + + @property + def is_asis(self) -> bool: + """Returns True if this is an "as is" candidate.""" + return self.id.startswith("asis-") + + @property + def mapping(self) -> dict[int, int]: + return self._mapping + + @property + def current_mapping(self) -> dict[int, int]: + """Get the current mapping from items to tracks, calculated from the match.""" + if isinstance(self.match, BeetsAlbumMatch): + return _index_mapping( + self.match.mapping, + self.items, + self.tracks, + ) + + raise ValueError("Current mapping only available for album matches.") + + # ------------------------------------ utility ----------------------------------- # + + def identify_duplicates(self, lib: BeetsLibrary | None = None) -> list[BeetsAlbum]: + """Find duplicates. + + Copy of beets' `task.find_duplicates` but works on any candidates' match. + + # FIXME: Tracks are not checked for duplicates. Tbh noone cares about tracks anyways + """ + if lib is None: + lib = _open_library(get_config()) + + info = self.match.info.copy() + info["albumartist"] = info["artist"] + + if info["artist"] is None: + # As-is import with no artist. Skip check. + return [] + + # Construct a query to find duplicates with this metadata. We + # use a temporary Album object to generate any computed fields. + tmp_album = BeetsAlbum(lib, **info) + keys: list[str] = cast( + list[str], + get_config()["import"]["duplicate_keys"]["album"].as_str_seq() or [], + ) + dup_query = tmp_album.duplicates_query(keys) + + # Re-Importing: Don't count albums with the same files as duplicates. + task_paths = {i.path for i in self.task_state.task.items if i} + + duplicates: list[BeetsAlbum] = [] + for album in lib.albums(dup_query): + # Check whether the album paths are all present in the task + # i.e. album is being completely re-imported by the task, + # in which case it is not a duplicate (will be replaced). + album_paths = {i.path for i in album.items()} + if not (album_paths <= task_paths): + duplicates.append(album) + + # Write duplicates information! + self.duplicate_ids = [d.id for d in duplicates] + + return duplicates + + @property + def has_duplicates_in_library(self) -> bool: + """Returns False, either if no duplicates found, or you have not checked yet. + + Call `identify_duplicates` first to ensure this works. + """ + return len(self.duplicate_ids) > 0 + + def serialize(self) -> SerializedCandidateState: + """JSON representation to match the frontend types.""" + # we lift the match.info to reduce nesting in the frontend. + info: TrackInfo | AlbumInfo + tracks: list[TrackInfo] + mapping: dict[int, int] = {} + + if isinstance(self.match.info, BeetsTrackInfo): + # This hardly ever happens, we might support this more in the future + info = TrackInfo.from_beets(self.match.info) + tracks = [TrackInfo.from_beets(self.match.info)] + elif isinstance(self.match.info, BeetsAlbumInfo): + info = AlbumInfo.from_beets(self.match.info) + + # Map beets types to our types, allows serialization magic + tracks = [TrackInfo.from_beets(track) for track in self.match.info.tracks] + + # mapping = _index_mapping( + # self.match.mapping, # type: ignore + # self.items, + # self.tracks, + # ) + mapping = self.mapping + + else: + raise ValueError(f"Unknown type of matchinfo {type(self.match.info)}") + + res = SerializedCandidateState( + **super().serialize(), + penalties=self.penalties, + duplicate_ids=self.duplicate_ids, + type=self.type, + distance=self.distance.distance, + info=info, + tracks=tracks, + mapping=mapping, + ) + + return res + + +def _index_mapping( + mapping: dict[BeetsItem, BeetsTrackInfo], + items: list[BeetsItem], + tracks: list[BeetsTrackInfo], +) -> dict[int, int]: + """Helper to create an index mapping from items to tracks. + + the mapping of a beets albummatch uses objects, but we don not want + to send them over redundantly. convert to an index mapping, + where first index is in self.items, and second is in self.match.info.tracks + + This is used to serialize the mapping of a candidate state. + """ + + # log.debug(f"items: {[i.title for i in items]}") + + idxs = [] + tdxs = [] + for item, track in mapping.items(): + # log.debug(f"track: {track.index} {track.track_alt} {track.title}") + # log.debug(f"item: {item.track} {item.title}") + + # compare items via paths, and tracks via full dicts + # we used to compare via track_id, but this might be None. + found_idx = found_tdx = None + for idx, _ in enumerate(items): + if item.path == items[idx].path: + found_idx = idx + break + for tdx, _ in enumerate(tracks): + if track == tracks[tdx]: + found_tdx = tdx + break + idxs.append(found_idx) + tdxs.append(found_tdx) + + if None in idxs or None in tdxs: + # breakpoint() + raise ValueError( + f"Index mapping failed: {idxs=} {tdxs=} {len(items)=} {len(tracks)=}" + ) + + # ignore type for mypy, we have checked that its not None! + res: dict[int, int] = {idx: tdx for idx, tdx in zip(idxs, tdxs)} # type: ignore[misc] + # log.debug(f"Index mapping: {res}") + + return res + + +# ---------------------------- Serialization types --------------------------- # +# Used for getting typehints in the frontend. I.e. we generate the types from +# these typed dicts! See the generate_types.py script for more information. + + +class Metadata(TypedDict): + """Returned from current_metadata(). + + FIXME: I think this should not be defined here! + """ + + artist: str | None + album: str | None + albumartist: str | None + year: str | None + disctotal: str | None + mb_albumid: str | None + label: str | None + barcode: str | None + catalognum: str | None + country: str | None + media: str | None + albumdisambig: str | None + + +class SerializedBaseState(TypedDict): + """Serialized base state. + + This is used to serialize the base state to a dictionary. + """ + + id: str + created_at: datetime + updated_at: datetime + + +class SerializedSessionState(SerializedBaseState): + folder_path: str + folder_hash: str + tasks: list[SerializedTaskState] + status: SerializedProgressState + + exc: NotRequired[SerializedException | None] + + +class SerializedTaskState(SerializedBaseState): + items: list[ItemInfo] + current_metadata: Metadata + + # Fetched data + candidates: list[SerializedCandidateState] + asis_candidate: SerializedCandidateState + + duplicate_action: str | None + chosen_candidate_id: str | None + completed: bool + toppath: str | None + paths: list[str] + + +class SerializedCandidateState(SerializedBaseState): + duplicate_ids: list[str] + type: str + + penalties: list[str] + distance: float + + info: TrackInfo | ItemInfo | AlbumInfo + + # Mapping from items to tracks index based + mapping: dict[int, int] + tracks: list[TrackInfo] + + +__all__ = [ + "SessionState", + "TaskState", + "CandidateState", + "SerializedSessionState", + "SerializedTaskState", + "SerializedCandidateState", +] diff --git a/backend/beets_flask/importer/types.py b/backend/beets_flask/importer/types.py index be11f51f..9dece8ce 100644 --- a/backend/beets_flask/importer/types.py +++ b/backend/beets_flask/importer/types.py @@ -1,248 +1,248 @@ -"""Typed versions of data classes that beets uses. - -Also includes and our own derivatives. -""" - -from __future__ import annotations - -from abc import ABC -from collections.abc import Callable -from dataclasses import dataclass -from typing import ( - Any, - Literal, - NamedTuple, - cast, -) - -from beets import autotag -from beets.autotag.distance import Distance as BeetsDistance -from beets.autotag.hooks import AlbumInfo as BeetsAlbumInfo -from beets.autotag.hooks import AlbumMatch as BeetsAlbumMatch -from beets.autotag.hooks import TrackInfo as BeetsTrackInfo -from beets.autotag.hooks import TrackMatch as BeetsTrackMatch -from beets.importer import ImportTask as BeetsImportTask -from beets.library import Album as BeetsAlbum -from beets.library import Item as BeetsItem -from beets.library import Library as BeetsLibrary - -__all__ = [ - # Our stuff - "MusicInfo", - "TrackInfo", - "ItemInfo", - "AlbumInfo", - "DuplicateAction", - # Beets stuff - "BeetsAlbum", - "BeetsAlbumInfo", - "BeetsAlbumMatch", - "BeetsItem", - "BeetsTrackInfo", - "BeetsTrackMatch", - "BeetsLibrary", - "BeetsDistance", - "BeetsImportTask", -] - -# to be consistent with beets, here we do not use an enum. -# (beets uses strings for duplicate actions) -DuplicateAction = Literal["skip", "keep", "remove", "merge", "ask"] - - -class PromptChoice(NamedTuple): - short: str - long: str - callback: None | Callable - - def serialize(self): - return { - "short": self.short, - "long": self.long, - "callback": self.callback.__name__ if self.callback else "None", - } - - -@dataclass -class MusicInfo(ABC): - """Shared info for tracks, items and albums. - - Items (music files on disk), tracks (trackinfo), and album info are somewhat similar. - They share many fields --- especially once music has been imported. - In beets there is no shared baseclass from which the three inherit, but such a common - base class helps in the frontend. - - This is a minimal version of this, where fields exclsuive to one type are None for the - others. (and inconsistent fields could get renamed?) - - @PS: Shouldn't this be an abstract class? - """ - - type: Literal["item", "track", "album"] - - artist: str | None - album: str | None - data_url: str | None - data_source: str | None - year: int | None - genre: str | None - media: str | None - - @classmethod - def _from_instance( - cls, - info: autotag.TrackInfo | autotag.Item | autotag.AlbumInfo, - remap: dict[str, str] = dict(), - ): - """Convert from beets TrackInfo, Item or AlbumInfo to our MusicInfo. - - You should only call this in from_beets() methods of the derived classes. - """ - kwargs = class_attributes_to_kwargs(cls, info, remap=remap) - if isinstance(info, autotag.TrackInfo): - kwargs["type"] = "track" - return TrackInfo(**kwargs) - elif isinstance(info, BeetsItem): - return ItemInfo(**kwargs) - elif isinstance(info, autotag.AlbumInfo): - kwargs["type"] = "album" - return AlbumInfo(**kwargs) - - raise ValueError(f"Unknown type of info: {info}") - - def __repr__(self) -> str: - res = f"{self.__class__.__name__}" - res += f"{self.type=} " - res += f"{self.artist=} " - res += f"{self.album=}" - return res - - -def class_attributes_to_kwargs( - cls, - obj, - keys=None, - remap: dict[str, str] = dict(), -) -> dict[str, Any]: - """Convert the attributes of an object to a dictionary of keyword arguments. - - May be used for any class. If `keys` is provided, only those keys are used. - - Remap allows to map the keys of the object to different keys in the dictionary. - E.g. if the object has an attribute 'track' and you want to use 'index' in the - dictionary, you can use `remap={"track": "index"}`. - - Paramters - --------- - cls: class - The class of the object to convert _to_ e.g. TrackInfo - obj: object - The object to convert _from_ i.e. Datatype from beets - - """ - if keys is None: - keys = cls.__dataclass_fields__.keys() - # {'index', ...} - kwargs = dict() - for k in keys: - kwargs[k] = getattr(obj, k, None) - for k in remap.keys(): - kwargs[remap[k]] = getattr(obj, k, None) - return kwargs - - -@dataclass -class AlbumInfo(MusicInfo): - """Mre specific version of MusicInfo for albums. - - Attributes are an indicator of what might be available, and can be None. - """ - - # disambiguation - mediums: int | None # number of discs - country: str | None - label: str | None - catalognum: str | None - albumdisambig: str | None - - # Note: dont add 'tracks' here, our candidate states lift them already from album matches - @classmethod - def from_beets(cls, info: autotag.AlbumInfo): - """Helper to convert from beets AlbumInfo to our AlbumInfo.""" - return cast( - AlbumInfo, - cls._from_instance(info), - ) - - -@dataclass -class TrackInfo(MusicInfo): - """More specific version of MusicInfo for tracks. - - Attributes are an indicator of what might be available, and can be None. - """ - - title: str | None - length: float | None - isrc: str | None - - # Allows to compute the mapping in the frontend - index: int | None # 1-based - medium_index: int | None - medium: int | None - - @classmethod - def from_beets(cls, info: autotag.TrackInfo): - """Helper to convert from beets TrackInfo to our TrackInfo.""" - return cast( - TrackInfo, - cls._from_instance(info), - ) - - -@dataclass -class ItemInfo(MusicInfo): - """More specific version of MusicInfo for items. - - Corresponds to a music file or library item with file on disk. - - Attributes are an indicator of what might be available, and can be None. - """ - - title: str | None - length: float | None - isrc: str | None - - index: int | None # 1-based - - path: str | None - bitrate: int | None - format: str | None - - @property - def track(self) -> int | None: - """Track number of the item. - - In beets vanilla types are somewhat inconsistent, which makes frontend - code hard to understand. - items.track for files on disk (1-based index for track number) - track.index for meta data from candidates (1-based index) - we consistently used `.index` - """ - return self.index - - @classmethod - def from_beets(cls, info: autotag.Item): - """Helper to convert from beets Item to our ItemInfo.""" - return cast( - ItemInfo, - cls._from_instance( - info, - # beets' vanilla types are somewhat inconsistent, which makes frontend - # code hard to understand. - # items.track for files on disk (1-based index for track number) - # track.index for meta data from candidates (1-based index) - # we consistently used `.index` - remap={"track": "index"}, - ), - ) +"""Typed versions of data classes that beets uses. + +Also includes and our own derivatives. +""" + +from __future__ import annotations + +from abc import ABC +from collections.abc import Callable +from dataclasses import dataclass +from typing import ( + Any, + Literal, + NamedTuple, + cast, +) + +from beets import autotag +from beets.autotag.distance import Distance as BeetsDistance +from beets.autotag.hooks import AlbumInfo as BeetsAlbumInfo +from beets.autotag.hooks import AlbumMatch as BeetsAlbumMatch +from beets.autotag.hooks import TrackInfo as BeetsTrackInfo +from beets.autotag.hooks import TrackMatch as BeetsTrackMatch +from beets.importer import ImportTask as BeetsImportTask +from beets.library import Album as BeetsAlbum +from beets.library import Item as BeetsItem +from beets.library import Library as BeetsLibrary + +__all__ = [ + # Our stuff + "MusicInfo", + "TrackInfo", + "ItemInfo", + "AlbumInfo", + "DuplicateAction", + # Beets stuff + "BeetsAlbum", + "BeetsAlbumInfo", + "BeetsAlbumMatch", + "BeetsItem", + "BeetsTrackInfo", + "BeetsTrackMatch", + "BeetsLibrary", + "BeetsDistance", + "BeetsImportTask", +] + +# to be consistent with beets, here we do not use an enum. +# (beets uses strings for duplicate actions) +DuplicateAction = Literal["skip", "keep", "remove", "merge", "ask"] + + +class PromptChoice(NamedTuple): + short: str + long: str + callback: None | Callable + + def serialize(self): + return { + "short": self.short, + "long": self.long, + "callback": self.callback.__name__ if self.callback else "None", + } + + +@dataclass +class MusicInfo(ABC): + """Shared info for tracks, items and albums. + + Items (music files on disk), tracks (trackinfo), and album info are somewhat similar. + They share many fields --- especially once music has been imported. + In beets there is no shared baseclass from which the three inherit, but such a common + base class helps in the frontend. + + This is a minimal version of this, where fields exclsuive to one type are None for the + others. (and inconsistent fields could get renamed?) + + @PS: Shouldn't this be an abstract class? + """ + + type: Literal["item", "track", "album"] + + artist: str | None + album: str | None + data_url: str | None + data_source: str | None + year: int | None + genre: str | None + media: str | None + + @classmethod + def _from_instance( + cls, + info: autotag.TrackInfo | autotag.Item | autotag.AlbumInfo, + remap: dict[str, str] = dict(), + ): + """Convert from beets TrackInfo, Item or AlbumInfo to our MusicInfo. + + You should only call this in from_beets() methods of the derived classes. + """ + kwargs = class_attributes_to_kwargs(cls, info, remap=remap) + if isinstance(info, autotag.TrackInfo): + kwargs["type"] = "track" + return TrackInfo(**kwargs) + elif isinstance(info, BeetsItem): + return ItemInfo(**kwargs) + elif isinstance(info, autotag.AlbumInfo): + kwargs["type"] = "album" + return AlbumInfo(**kwargs) + + raise ValueError(f"Unknown type of info: {info}") + + def __repr__(self) -> str: + res = f"{self.__class__.__name__}" + res += f"{self.type=} " + res += f"{self.artist=} " + res += f"{self.album=}" + return res + + +def class_attributes_to_kwargs( + cls, + obj, + keys=None, + remap: dict[str, str] = dict(), +) -> dict[str, Any]: + """Convert the attributes of an object to a dictionary of keyword arguments. + + May be used for any class. If `keys` is provided, only those keys are used. + + Remap allows to map the keys of the object to different keys in the dictionary. + E.g. if the object has an attribute 'track' and you want to use 'index' in the + dictionary, you can use `remap={"track": "index"}`. + + Paramters + --------- + cls: class + The class of the object to convert _to_ e.g. TrackInfo + obj: object + The object to convert _from_ i.e. Datatype from beets + + """ + if keys is None: + keys = cls.__dataclass_fields__.keys() + # {'index', ...} + kwargs = dict() + for k in keys: + kwargs[k] = getattr(obj, k, None) + for k in remap.keys(): + kwargs[remap[k]] = getattr(obj, k, None) + return kwargs + + +@dataclass +class AlbumInfo(MusicInfo): + """Mre specific version of MusicInfo for albums. + + Attributes are an indicator of what might be available, and can be None. + """ + + # disambiguation + mediums: int | None # number of discs + country: str | None + label: str | None + catalognum: str | None + albumdisambig: str | None + + # Note: dont add 'tracks' here, our candidate states lift them already from album matches + @classmethod + def from_beets(cls, info: autotag.AlbumInfo): + """Helper to convert from beets AlbumInfo to our AlbumInfo.""" + return cast( + AlbumInfo, + cls._from_instance(info), + ) + + +@dataclass +class TrackInfo(MusicInfo): + """More specific version of MusicInfo for tracks. + + Attributes are an indicator of what might be available, and can be None. + """ + + title: str | None + length: float | None + isrc: str | None + + # Allows to compute the mapping in the frontend + index: int | None # 1-based + medium_index: int | None + medium: int | None + + @classmethod + def from_beets(cls, info: autotag.TrackInfo): + """Helper to convert from beets TrackInfo to our TrackInfo.""" + return cast( + TrackInfo, + cls._from_instance(info), + ) + + +@dataclass +class ItemInfo(MusicInfo): + """More specific version of MusicInfo for items. + + Corresponds to a music file or library item with file on disk. + + Attributes are an indicator of what might be available, and can be None. + """ + + title: str | None + length: float | None + isrc: str | None + + index: int | None # 1-based + + path: str | None + bitrate: int | None + format: str | None + + @property + def track(self) -> int | None: + """Track number of the item. + + In beets vanilla types are somewhat inconsistent, which makes frontend + code hard to understand. + items.track for files on disk (1-based index for track number) + track.index for meta data from candidates (1-based index) + we consistently used `.index` + """ + return self.index + + @classmethod + def from_beets(cls, info: autotag.Item): + """Helper to convert from beets Item to our ItemInfo.""" + return cast( + ItemInfo, + cls._from_instance( + info, + # beets' vanilla types are somewhat inconsistent, which makes frontend + # code hard to understand. + # items.track for files on disk (1-based index for track number) + # track.index for meta data from candidates (1-based index) + # we consistently used `.index` + remap={"track": "index"}, + ), + ) diff --git a/backend/beets_flask/invoker/__init__.py b/backend/beets_flask/invoker/__init__.py index 8fd246c7..5af3f324 100644 --- a/backend/beets_flask/invoker/__init__.py +++ b/backend/beets_flask/invoker/__init__.py @@ -1,7 +1,7 @@ -from .enqueue import EnqueueKind, enqueue, enqueue_delete_items - -__all__ = [ - "enqueue", - "enqueue_delete_items", - "EnqueueKind", -] +from .enqueue import EnqueueKind, enqueue, enqueue_delete_items + +__all__ = [ + "enqueue", + "enqueue_delete_items", + "EnqueueKind", +] diff --git a/backend/beets_flask/invoker/enqueue.py b/backend/beets_flask/invoker/enqueue.py index 531e13e3..57b49332 100644 --- a/backend/beets_flask/invoker/enqueue.py +++ b/backend/beets_flask/invoker/enqueue.py @@ -1,660 +1,674 @@ -from __future__ import annotations - -import asyncio -from collections.abc import Awaitable, Callable -from enum import Enum -from typing import ( - TYPE_CHECKING, - Any, - Literal, - ParamSpec, - TypeVar, -) - -from beets.ui import _open_library -from sqlalchemy import func, select -from sqlalchemy.orm import Session - -from beets_flask.config import get_config -from beets_flask.database import db_session_factory -from beets_flask.database.models.states import ( - FolderInDb, - SessionState, - SessionStateInDb, -) -from beets_flask.importer.progress import FolderStatus -from beets_flask.importer.session import ( - AddCandidatesSession, - AutoImportSession, - BootlegImportSession, - CandidateChoice, - ImportSession, - PreviewSession, - Search, - TaskIdMappingArg, - UndoSession, - delete_from_beets, -) -from beets_flask.importer.types import DuplicateAction -from beets_flask.logger import log -from beets_flask.redis import import_queue, preview_queue -from beets_flask.server.exceptions import ( - InvalidUsageException, - exception_as_return_value, -) -from beets_flask.server.websocket.status import ( - JobStatusUpdate, - emit_folder_status, - send_status_update, -) - -from .job import ExtraJobMeta, _set_job_meta - -if TYPE_CHECKING: - from rq.job import Job - from rq.queue import Queue - - -def emit_update_on_job_change(job, connection, result, *args, **kwargs): - """ - Callback for rq enqueue functions to emit a job status update via websocket. - - See https://python-rq.org/docs/#success-callback - """ - log.debug(f"job update for socket {job=} {connection=} {result=} {args=} {kwargs=}") - - def _is_serialized_exception(d: Any): - # I wish we could to instance checks on our SerializedException TypedDict - if not isinstance(result, dict): - return False - if "type" in d and "message" in d.keys(): - # the other keys are optional - return True - return False - - try: - asyncio.run( - send_status_update( - JobStatusUpdate( - message="Job status update", - num_jobs=1, - job_metas=[job.get_meta()], - exc=result if _is_serialized_exception(result) else None, - ) - ) - ) - except Exception as e: - log.error(f"Failed to emit job update: {e}", exc_info=True) - - -P = ParamSpec("P") # Parameters -R = TypeVar("R") # Return - - -def _enqueue( - queue: Queue, - f: Callable[P, Any | Awaitable[Any]], - *args: P.args, - **kwargs: P.kwargs, -) -> Job: - """Enqueue a job in redis. - - Helper that sets some shared behavior and allows - to for proper type hinting. - """ - - job = queue.enqueue( - f, - *args, - **kwargs, - on_success=emit_update_on_job_change, - ) - return job - - -class EnqueueKind(Enum): - """Enum for the different kinds of sessions we can enqueue.""" - - PREVIEW = "preview" - PREVIEW_ADD_CANDIDATES = "preview_add_candidates" - IMPORT_CANDIDATE = "import_candidate" - IMPORT_AUTO = "import_auto" - IMPORT_UNDO = "import_undo" - IMPORT_BOOTLEG = "import_bootleg" - # Bootlegs are essentially asis, but does not mean to just import the asis candidate, - # it has its own session that also groups albums, and skips previews. - - _AUTO_IMPORT = "_auto_import" - _AUTO_PREVIEW = "_auto_preview" - - @classmethod - def from_str(cls, kind: str) -> EnqueueKind: - """Convert a string to an EnqueueKind enum. - - Parameters - ---------- - kind : str - The string to convert. - """ - try: - return cls[kind.upper()] - except KeyError: - raise ValueError(f"Unknown kind {kind}") - - -@emit_folder_status(before=FolderStatus.PENDING) -async def enqueue( - hash: str, - path: str, - kind: EnqueueKind, - extra_meta: ExtraJobMeta | None = None, - **kwargs, -) -> Job: - """Delegate a preview or import to a redis worker, depending on its kind. - - Parameters - ---------- - hash : str - The hash of the folder to enqueue. - path : str - The path of the folder to enqueue. - kind : EnqueueKind - The kind of the folder to enqueue. - extra_meta: ExtraJobMeta, optional - Extra meta data to pass to the job. E.g. use this to assign a reference - for the frontend to the job, so we can track it via the websocket. - kwargs : dict - Additional arguments to pass to the worker functions. Depend on the kind, - use with care. - """ - if extra_meta is None: - extra_meta = ExtraJobMeta() - - match kind: - case EnqueueKind.PREVIEW: - job = enqueue_preview(hash, path, extra_meta, **kwargs) - case EnqueueKind.PREVIEW_ADD_CANDIDATES: - job = enqueue_preview_add_candidates(hash, path, extra_meta, **kwargs) - case EnqueueKind.IMPORT_AUTO: - job = enqueue_import_auto(hash, path, extra_meta, **kwargs) - case EnqueueKind.IMPORT_CANDIDATE: - job = enqueue_import_candidate(hash, path, extra_meta, **kwargs) - case EnqueueKind.IMPORT_BOOTLEG: - job = enqueue_import_bootleg(hash, path, extra_meta, **kwargs) - case EnqueueKind.IMPORT_UNDO: - job = enqueue_import_undo(hash, path, extra_meta, **kwargs) - case _: - raise InvalidUsageException(f"Unknown kind {kind}") - - log.debug(f"Enqueued {job.id=} {job.meta=}") - - return job - - -# --------------------------- Enqueue entry points --------------------------- # -# Mainly input validation and submitting to the redis queue - - -def enqueue_preview(hash: str, path: str, extra_meta: ExtraJobMeta, **kwargs) -> Job: - group_albums: bool | None = kwargs.pop("group_albums", None) - autotag: bool | None = kwargs.pop("autotag", None) - - if len(kwargs.keys()) > 0: - raise InvalidUsageException("EnqueueKind.PREVIEW does not accept any kwargs.") - job = _enqueue(preview_queue, run_preview, hash, path, group_albums, autotag) - _set_job_meta(job, hash, path, EnqueueKind.PREVIEW, extra_meta) - return job - - -def enqueue_preview_add_candidates( - hash: str, path: str, extra_meta: ExtraJobMeta, **kwargs -) -> Job: - # May contain search_ids, search_artist, search_album - # As always to allow task mapping - - search: TaskIdMappingArg[Search | Literal["skip"]] = kwargs.pop("search", None) - if len(kwargs.keys()) > 0: - raise InvalidUsageException( - "EnqueueKind.PREVIEW_ADD_CANDIDATES only accepts the following kwargs: " - + "search" - ) - - if search is None: - raise InvalidUsageException( - "EnqueueKind.PREVIEW_ADD_CANDIDATES requires a search kwarg." - ) - - # kwargs are mixed between our own function and redis enqueue -.- - # if we accidentally define a redis kwarg for our function, it will be ignored. - # https://python-rq.org/docs/#enqueueing-jobs - job = _enqueue( - preview_queue, - run_preview_add_candidates, - hash, - path, - search=search, - ) - _set_job_meta(job, hash, path, EnqueueKind.PREVIEW_ADD_CANDIDATES, extra_meta) - return job - - -def enqueue_import_candidate( - hash: str, path: str, extra_meta: ExtraJobMeta, **kwargs -) -> Job: - """ - Imports a candidate that has been fetched in a preview session. - - Kwargs - ------ - candidate_id : CandidateChoice | dict[str, CandidateChoice] | None - A valid candidate id for a candidate that has been fetched in a preview - session. - None stands for best, is resolved in the session. - additionally, if a dict is provided, it maps from task_id to candidate, and dupicate action, respectively. - duplicate_actions - See candidate_id. - TODO: Also allowed: "asis" (no exact match needed, there is only one - asis-candidate). - """ - - candidate_ids: TaskIdMappingArg[CandidateChoice] = kwargs.pop("candidate_ids", None) - duplicate_actions: TaskIdMappingArg[DuplicateAction] = kwargs.pop( - "duplicate_actions", None - ) - - if len(kwargs.keys()) > 0: - raise InvalidUsageException( - "EnqueueKind.IMPORT only accepts the following kwargs: " - + "candidate_ids, duplicate_actions." - ) - - # TODO: Validation: lookup candidates exits - - # For convenience: if the user calls this but no preview was generated before, - # use the auto-import instead (which also fetches previews). - try: - # TODO: along with validation: - # we need a special flag as task_id that stands for "do this for all tasks" - # used along with candidate_ids length == 1. - # then, only run the fallback auto-import for the args coming from gui import button - - # If the user did not specify a candidate_id, we assume they want the best - # candidate. - with db_session_factory() as db_session: - _get_live_state_by_folder(hash, path, db_session) - # raises if no state found - except: - log.info( - f"No previous session state fround for {hash=} {path=} " - + "switching to auto-import" - ) - return enqueue_import_auto(hash, path, extra_meta) - - job = _enqueue( - import_queue, - run_import_candidate, - hash, - path, - candidate_ids=candidate_ids, - duplicate_actions=duplicate_actions, - ) - _set_job_meta(job, hash, path, EnqueueKind.IMPORT_CANDIDATE, extra_meta) - return job - - -def enqueue_import_auto(hash: str, path: str, extra_meta: ExtraJobMeta, **kwargs): - """ - Enqueue an automatic import. - - Auto jobs first generate a preview (if needed) and then run an import, which always - imports the best candidate - but only if the preview is good enough (as specified - in the users beets config) - - This is a two step process, and previews run in another queue (thread) than imports. - - See AutoImportSession for more details. - """ - - group_albums: bool | None = kwargs.pop("group_albums", None) - autotag: bool | None = kwargs.pop("autotag", None) - import_threshold: float | None = kwargs.pop("import_threshold", None) - duplicate_actions: TaskIdMappingArg[DuplicateAction] = kwargs.pop( - "duplicate_actions", None - ) - - if len(kwargs.keys()) > 0: - raise InvalidUsageException( - "EnqueueKind.IMPORT_AUTO only accepts the following kwargs: " - + "group_albums, autotag, import_threshold, duplicate_actions." - ) - - # We only assign the on_success callback (likely coming - # via a kwarg) to the second job! - job1 = preview_queue.enqueue( - run_preview, hash, path, group_albums=group_albums, autotag=autotag, **kwargs - ) - _set_job_meta(job1, hash, path, EnqueueKind._AUTO_PREVIEW, extra_meta) - job2 = _enqueue( - import_queue, - run_import_auto, - hash, - path, - import_threshold=import_threshold, - duplicate_actions=duplicate_actions, - **kwargs, - # rq has no proper typing therefore our kwargs are not type checked properly - depends_on=job1, # type: ignore - ) - _set_job_meta(job2, hash, path, EnqueueKind._AUTO_IMPORT, extra_meta) - - return job2 - - -def enqueue_import_bootleg(hash: str, path: str, extra_meta: ExtraJobMeta, **kwargs): - job = _enqueue(import_queue, run_import_bootleg, hash, path, **kwargs) - _set_job_meta(job, hash, path, EnqueueKind.IMPORT_BOOTLEG, extra_meta) - return job - - -def enqueue_import_undo(hash: str, path: str, extra_meta: ExtraJobMeta, **kwargs): - delete_files: bool = kwargs.pop("delete_files", True) - - if len(kwargs.keys()) > 0: - raise InvalidUsageException( - "EnqueueKind.IMPORT_UNDO only accepts the following kwargs: " - + "delete_files." - ) - - job = _enqueue( - import_queue, - run_import_undo, - hash, - path, - delete_files=delete_files, - ) - _set_job_meta(job, hash, path, EnqueueKind.IMPORT_UNDO, extra_meta) - return job - - -def enqueue_delete_items(task_ids: list[str]) -> Job: - """Enqueue to delete items from the beets library. - - A bit of a special case as this does not use the normal - hash and path based enqueueing. - """ - job = _enqueue( - import_queue, - delete_items, - task_ids, - True, - # rq has no proper typing therefore our kwargs are not type checked properly - at_front=True, # type: ignore - ) - return job - - -# -------------------- Functions that run in redis workers ------------------- # -# TODO: We might want to move these to their own file, for a bit better separation of -# concerns. - - -# redis preview queue -@exception_as_return_value -@emit_folder_status(before=FolderStatus.PREVIEWING, after=FolderStatus.PREVIEWED) -async def run_preview( - hash: str, - path: str, - group_albums: bool | None, - autotag: bool | None, -): - """Fetch candidates for a folder using beets. - - Will refetch candidates if this is rerun even if candidates exist - in the db. - - Current convention is we have one session for one folder *has*, but - We might have multiple sessions for the same folder **path**. - Previews will **reset** any previous session state in the database, if they - exist for the same folder hash. - - Parameters - ---------- - hash : str - The hash of the folder for which to run the preview. - path : str - The path of the folder for which to run the preview. - group_albums : bool | None - Whether to create multple tasks, one for each album found in the metadata - of the files. Set to true if you have multiple albums in a single folder. - If None: get value from beets config. - autotag : bool | None - Whether to look up metadata online. If None: get value from beets config. - """ - - log.info(f"Preview task on {hash=} {path=}") - - with db_session_factory() as db_session: - f_on_disk = FolderInDb.get_current_on_disk(hash, path) - if hash != f_on_disk.hash: - log.warning( - f"Folder content has changed since the job was scheduled for {path}. " - + f"Using new content ({f_on_disk.hash}) instead of {hash}" - ) - - # here, in preview, we always want to start from a fresh state - # an existing state would skip the candidate lookup. - # otherwise, the retag action would not work, as preview starting from - s_state_live = SessionState(f_on_disk) - p_session = PreviewSession( - s_state_live, group_albums=group_albums, autotag=autotag - ) - - try: - await p_session.run_async() - finally: - # Get max revision for this folder hash - stmt = select(func.max(SessionStateInDb.folder_revision)).where( - SessionStateInDb.folder_hash == hash, - ) - max_rev = db_session.execute(stmt).scalar_one_or_none() - new_rev = 0 if max_rev is None else max_rev + 1 - s_state_indb = SessionStateInDb.from_live_state(p_session.state) - s_state_indb.folder_revision = new_rev - - db_session.merge(s_state_indb) - db_session.commit() - - log.info(f"Preview done. {hash=} {path=}") - return - - -# redis preview queue -@exception_as_return_value -@emit_folder_status(before=FolderStatus.PREVIEWING, after=FolderStatus.PREVIEWED) -async def run_preview_add_candidates( - hash: str, path: str, search: TaskIdMappingArg[Search | Literal["skip"]] -): - """Adds a candidate to an session which is already in the status tagged. - - This only works if all session tasks are tagged. I.e. preview completed. - - Parameters - ---------- - search : dict[str, Search] - A dictionary of task ids to search dicts. No value or none skips the search - for this task. - """ - log.info(f"Add preview candidates task on {hash=}") - - with db_session_factory() as db_session: - s_state_live = _get_live_state_by_folder(hash, path, db_session) - - a_session = AddCandidatesSession( - s_state_live, - search=search, - ) - try: - await a_session.run_async() - finally: - s_state_indb = SessionStateInDb.from_live_state(a_session.state) - db_session.merge(instance=s_state_indb) - db_session.commit() - - log.info(f"Add candidates done. {hash=} {path=}") - - -# redis import queue -@exception_as_return_value -@emit_folder_status(before=FolderStatus.IMPORTING, after=FolderStatus.IMPORTED) -async def run_import_candidate( - hash: str, - path: str, - candidate_ids: TaskIdMappingArg[CandidateChoice], - duplicate_actions: TaskIdMappingArg[DuplicateAction], -): - """Imports a candidate that has been fetched in a preview session. - - Parameters - ---------- - candidate_id : optional - If candidate_id is none the best candidate is used. - duplicate_action : optional - If duplicate_action is none, the default action from the config is used. - """ - log.info(f"Import task on {hash=} {path=}") - - with db_session_factory() as db_session: - s_state_live = _get_live_state_by_folder(hash, path, db_session) - - i_session = ImportSession( - s_state_live, - candidate_ids=candidate_ids, - duplicate_actions=duplicate_actions, - ) - - try: - await i_session.run_async() - finally: - s_state_indb = SessionStateInDb.from_live_state(i_session.state) - db_session.merge(instance=s_state_indb) - db_session.commit() - - log.info(f"Import candidate done. {hash=} {path=}") - - -# redis import queue -@exception_as_return_value -@emit_folder_status(before=FolderStatus.IMPORTING, after=FolderStatus.IMPORTED) -async def run_import_auto( - hash: str, - path: str, - import_threshold: float | None, - duplicate_actions: TaskIdMappingArg[DuplicateAction], -): - log.info(f"Auto Import task on {hash=} {path=}") - - with db_session_factory() as db_session: - s_state_live = _get_live_state_by_folder(hash, path, db_session) - i_session = AutoImportSession( - s_state_live, - import_threshold=import_threshold, - duplicate_actions=duplicate_actions, - ) - - try: - await i_session.run_async() - finally: - s_state_indb = SessionStateInDb.from_live_state(i_session.state) - db_session.merge(instance=s_state_indb) - db_session.commit() - - log.info(f"Auto Import done. {hash=} {path=}") - - -# redis import queue -@exception_as_return_value -@emit_folder_status(before=FolderStatus.IMPORTING, after=FolderStatus.IMPORTED) -async def run_import_bootleg(hash: str, path: str): - log.info(f"Bootleg Import task on {hash=} {path=}") - - with db_session_factory() as db_session: - # TODO: add duplicate action - # TODO: sort out how to generate previews for asis candidates - s_state_live = _get_live_state_by_folder( - hash, path, create_if_not_exists=True, db_session=db_session - ) - i_session = BootlegImportSession(s_state_live) - - try: - await i_session.run_async() - finally: - s_state_indb = SessionStateInDb.from_live_state(i_session.state) - db_session.merge(instance=s_state_indb) - db_session.commit() - - log.info(f"Bootleg Import done. {hash=} {path=}") - - -@exception_as_return_value -@emit_folder_status(before=FolderStatus.DELETING, after=FolderStatus.DELETED) -async def run_import_undo(hash: str, path: str, delete_files: bool): - log.info(f"Import Undo task on {hash=} {path=}") - - with db_session_factory() as db_session: - s_state_live = _get_live_state_by_folder(hash, path, db_session) - i_session = UndoSession(s_state_live, delete_files=delete_files) - - try: - await i_session.run_async() - finally: - s_state_indb = SessionStateInDb.from_live_state(i_session.state) - db_session.merge(instance=s_state_indb) - db_session.commit() - - log.info(f"Import Undo done. {hash=} {path=}") - - -def _get_live_state_by_folder( - hash: str, path: str, db_session: Session, create_if_not_exists=False -) -> SessionState: - f_on_disk = FolderInDb.get_current_on_disk(hash, path) - if hash != f_on_disk.hash: - log.warning( - f"Folder content has changed since the job was scheduled for {path}. " - + f"Using new content ({f_on_disk.hash}) instead of {hash}" - ) - - s_state_indb = SessionStateInDb.get_by_hash_and_path( - # we warn about hash change, and want the import to still run - # but on the old hash. - hash=hash, - path=path, - db_session=db_session, - ) - - if s_state_indb is None and create_if_not_exists: - s_state_live = SessionState(f_on_disk) - return s_state_live - - if s_state_indb is None: - # TODO: rq error handling - raise InvalidUsageException( - f"No session state found for {path=} {hash=} " - + f"fresh_hash_on_disk={f_on_disk}, this should not happen." - ) - - log.debug(f"Using existing session state for {path=}") - s_state_live = s_state_indb.to_live_state() - - # we need this expunge, otherwise we cannot overwrite session states: - # If object id is in session we cant add a new object to the session with the - # same id this will raise (see below session.merge) - db_session.expunge_all() - - return s_state_live - - -def delete_items(task_ids: list[str], delete_files: bool = True): - lib = _open_library(get_config()) - for task_id in task_ids: - delete_from_beets(task_id, delete_files=delete_files, lib=lib) +from __future__ import annotations + +import asyncio +from collections.abc import Awaitable, Callable +from enum import Enum +from pathlib import Path +from typing import ( + TYPE_CHECKING, + Any, + Literal, + ParamSpec, + TypeVar, +) + +from beets.ui import _open_library +from sqlalchemy import func, select +from sqlalchemy.orm import Session + +from beets_flask.config import get_config +from beets_flask.database import db_session_factory +from beets_flask.database.models.states import ( + FolderInDb, + SessionState, + SessionStateInDb, +) +from beets_flask.disk import compute_and_store_dir_stats +from beets_flask.importer.progress import FolderStatus +from beets_flask.importer.session import ( + AddCandidatesSession, + AutoImportSession, + BootlegImportSession, + CandidateChoice, + ImportSession, + PreviewSession, + Search, + TaskIdMappingArg, + UndoSession, + delete_from_beets, +) +from beets_flask.importer.types import DuplicateAction +from beets_flask.logger import log +from beets_flask.redis import import_queue, preview_queue +from beets_flask.server.exceptions import ( + InvalidUsageException, + exception_as_return_value, +) +from beets_flask.server.websocket.status import ( + JobStatusUpdate, + emit_folder_status, + send_status_update, +) + +from .job import ExtraJobMeta, _set_job_meta + +if TYPE_CHECKING: + from rq.job import Job + from rq.queue import Queue + + +def emit_update_on_job_change(job, connection, result, *args, **kwargs): + """ + Callback for rq enqueue functions to emit a job status update via websocket. + + See https://python-rq.org/docs/#success-callback + """ + log.debug(f"job update for socket {job=} {connection=} {result=} {args=} {kwargs=}") + + def _is_serialized_exception(d: Any): + # I wish we could to instance checks on our SerializedException TypedDict + if not isinstance(result, dict): + return False + if "type" in d and "message" in d.keys(): + # the other keys are optional + return True + return False + + try: + asyncio.run( + send_status_update( + JobStatusUpdate( + message="Job status update", + num_jobs=1, + job_metas=[job.get_meta()], + exc=result if _is_serialized_exception(result) else None, + ) + ) + ) + except Exception as e: + log.error(f"Failed to emit job update: {e}", exc_info=True) + + +P = ParamSpec("P") # Parameters +R = TypeVar("R") # Return + + +def _enqueue( + queue: Queue, + f: Callable[P, Any | Awaitable[Any]], + *args: P.args, + **kwargs: P.kwargs, +) -> Job: + """Enqueue a job in redis. + + Helper that sets some shared behavior and allows + to for proper type hinting. + """ + + job = queue.enqueue( + f, + *args, + **kwargs, + on_success=emit_update_on_job_change, + ) + return job + + +class EnqueueKind(Enum): + """Enum for the different kinds of sessions we can enqueue.""" + + PREVIEW = "preview" + PREVIEW_ADD_CANDIDATES = "preview_add_candidates" + IMPORT_CANDIDATE = "import_candidate" + IMPORT_AUTO = "import_auto" + IMPORT_UNDO = "import_undo" + IMPORT_BOOTLEG = "import_bootleg" + # Bootlegs are essentially asis, but does not mean to just import the asis candidate, + # it has its own session that also groups albums, and skips previews. + + _AUTO_IMPORT = "_auto_import" + _AUTO_PREVIEW = "_auto_preview" + + @classmethod + def from_str(cls, kind: str) -> EnqueueKind: + """Convert a string to an EnqueueKind enum. + + Parameters + ---------- + kind : str + The string to convert. + """ + try: + return cls[kind.upper()] + except KeyError: + raise ValueError(f"Unknown kind {kind}") + + +@emit_folder_status(before=FolderStatus.PENDING) +async def enqueue( + hash: str, + path: str, + kind: EnqueueKind, + extra_meta: ExtraJobMeta | None = None, + **kwargs, +) -> Job: + """Delegate a preview or import to a redis worker, depending on its kind. + + Parameters + ---------- + hash : str + The hash of the folder to enqueue. + path : str + The path of the folder to enqueue. + kind : EnqueueKind + The kind of the folder to enqueue. + extra_meta: ExtraJobMeta, optional + Extra meta data to pass to the job. E.g. use this to assign a reference + for the frontend to the job, so we can track it via the websocket. + kwargs : dict + Additional arguments to pass to the worker functions. Depend on the kind, + use with care. + """ + if extra_meta is None: + extra_meta = ExtraJobMeta() + + match kind: + case EnqueueKind.PREVIEW: + job = enqueue_preview(hash, path, extra_meta, **kwargs) + case EnqueueKind.PREVIEW_ADD_CANDIDATES: + job = enqueue_preview_add_candidates(hash, path, extra_meta, **kwargs) + case EnqueueKind.IMPORT_AUTO: + job = enqueue_import_auto(hash, path, extra_meta, **kwargs) + case EnqueueKind.IMPORT_CANDIDATE: + job = enqueue_import_candidate(hash, path, extra_meta, **kwargs) + case EnqueueKind.IMPORT_BOOTLEG: + job = enqueue_import_bootleg(hash, path, extra_meta, **kwargs) + case EnqueueKind.IMPORT_UNDO: + job = enqueue_import_undo(hash, path, extra_meta, **kwargs) + case _: + raise InvalidUsageException(f"Unknown kind {kind}") + + log.debug(f"Enqueued {job.id=} {job.meta=}") + + return job + + +# --------------------------- Enqueue entry points --------------------------- # +# Mainly input validation and submitting to the redis queue + + +def enqueue_preview(hash: str, path: str, extra_meta: ExtraJobMeta, **kwargs) -> Job: + group_albums: bool | None = kwargs.pop("group_albums", None) + autotag: bool | None = kwargs.pop("autotag", None) + + if len(kwargs.keys()) > 0: + raise InvalidUsageException("EnqueueKind.PREVIEW does not accept any kwargs.") + job = _enqueue(preview_queue, run_preview, hash, path, group_albums, autotag) + _set_job_meta(job, hash, path, EnqueueKind.PREVIEW, extra_meta) + return job + + +def enqueue_preview_add_candidates( + hash: str, path: str, extra_meta: ExtraJobMeta, **kwargs +) -> Job: + # May contain search_ids, search_artist, search_album + # As always to allow task mapping + + search: TaskIdMappingArg[Search | Literal["skip"]] = kwargs.pop("search", None) + if len(kwargs.keys()) > 0: + raise InvalidUsageException( + "EnqueueKind.PREVIEW_ADD_CANDIDATES only accepts the following kwargs: " + + "search" + ) + + if search is None: + raise InvalidUsageException( + "EnqueueKind.PREVIEW_ADD_CANDIDATES requires a search kwarg." + ) + + # kwargs are mixed between our own function and redis enqueue -.- + # if we accidentally define a redis kwarg for our function, it will be ignored. + # https://python-rq.org/docs/#enqueueing-jobs + job = _enqueue( + preview_queue, + run_preview_add_candidates, + hash, + path, + search=search, + ) + _set_job_meta(job, hash, path, EnqueueKind.PREVIEW_ADD_CANDIDATES, extra_meta) + return job + + +def enqueue_import_candidate( + hash: str, path: str, extra_meta: ExtraJobMeta, **kwargs +) -> Job: + """ + Imports a candidate that has been fetched in a preview session. + + Kwargs + ------ + candidate_id : CandidateChoice | dict[str, CandidateChoice] | None + A valid candidate id for a candidate that has been fetched in a preview + session. + None stands for best, is resolved in the session. + additionally, if a dict is provided, it maps from task_id to candidate, and dupicate action, respectively. + duplicate_actions + See candidate_id. + TODO: Also allowed: "asis" (no exact match needed, there is only one + asis-candidate). + """ + + candidate_ids: TaskIdMappingArg[CandidateChoice] = kwargs.pop("candidate_ids", None) + duplicate_actions: TaskIdMappingArg[DuplicateAction] = kwargs.pop( + "duplicate_actions", None + ) + + if len(kwargs.keys()) > 0: + raise InvalidUsageException( + "EnqueueKind.IMPORT only accepts the following kwargs: " + + "candidate_ids, duplicate_actions." + ) + + # TODO: Validation: lookup candidates exits + + # For convenience: if the user calls this but no preview was generated before, + # use the auto-import instead (which also fetches previews). + try: + # TODO: along with validation: + # we need a special flag as task_id that stands for "do this for all tasks" + # used along with candidate_ids length == 1. + # then, only run the fallback auto-import for the args coming from gui import button + + # If the user did not specify a candidate_id, we assume they want the best + # candidate. + with db_session_factory() as db_session: + _get_live_state_by_folder(hash, path, db_session) + # raises if no state found + except: + log.info( + f"No previous session state fround for {hash=} {path=} " + + "switching to auto-import" + ) + return enqueue_import_auto(hash, path, extra_meta) + + job = _enqueue( + import_queue, + run_import_candidate, + hash, + path, + candidate_ids=candidate_ids, + duplicate_actions=duplicate_actions, + ) + _set_job_meta(job, hash, path, EnqueueKind.IMPORT_CANDIDATE, extra_meta) + return job + + +def enqueue_import_auto(hash: str, path: str, extra_meta: ExtraJobMeta, **kwargs): + """ + Enqueue an automatic import. + + Auto jobs first generate a preview (if needed) and then run an import, which always + imports the best candidate - but only if the preview is good enough (as specified + in the users beets config) + + This is a two step process, and previews run in another queue (thread) than imports. + + See AutoImportSession for more details. + """ + + group_albums: bool | None = kwargs.pop("group_albums", None) + autotag: bool | None = kwargs.pop("autotag", None) + import_threshold: float | None = kwargs.pop("import_threshold", None) + duplicate_actions: TaskIdMappingArg[DuplicateAction] = kwargs.pop( + "duplicate_actions", None + ) + + if len(kwargs.keys()) > 0: + raise InvalidUsageException( + "EnqueueKind.IMPORT_AUTO only accepts the following kwargs: " + + "group_albums, autotag, import_threshold, duplicate_actions." + ) + + # We only assign the on_success callback (likely coming + # via a kwarg) to the second job! + job1 = preview_queue.enqueue( + run_preview, hash, path, group_albums=group_albums, autotag=autotag, **kwargs + ) + _set_job_meta(job1, hash, path, EnqueueKind._AUTO_PREVIEW, extra_meta) + job2 = _enqueue( + import_queue, + run_import_auto, + hash, + path, + import_threshold=import_threshold, + duplicate_actions=duplicate_actions, + **kwargs, + # rq has no proper typing therefore our kwargs are not type checked properly + depends_on=job1, # type: ignore + ) + _set_job_meta(job2, hash, path, EnqueueKind._AUTO_IMPORT, extra_meta) + + return job2 + + +def enqueue_import_bootleg(hash: str, path: str, extra_meta: ExtraJobMeta, **kwargs): + job = _enqueue(import_queue, run_import_bootleg, hash, path, **kwargs) + _set_job_meta(job, hash, path, EnqueueKind.IMPORT_BOOTLEG, extra_meta) + return job + + +def enqueue_import_undo(hash: str, path: str, extra_meta: ExtraJobMeta, **kwargs): + delete_files: bool = kwargs.pop("delete_files", True) + + if len(kwargs.keys()) > 0: + raise InvalidUsageException( + "EnqueueKind.IMPORT_UNDO only accepts the following kwargs: " + + "delete_files." + ) + + job = _enqueue( + import_queue, + run_import_undo, + hash, + path, + delete_files=delete_files, + ) + _set_job_meta(job, hash, path, EnqueueKind.IMPORT_UNDO, extra_meta) + return job + + +def enqueue_delete_items(task_ids: list[str]) -> Job: + """Enqueue to delete items from the beets library. + + A bit of a special case as this does not use the normal + hash and path based enqueueing. + """ + job = _enqueue( + import_queue, + delete_items, + task_ids, + True, + # rq has no proper typing therefore our kwargs are not type checked properly + at_front=True, # type: ignore + ) + return job + + +# -------------------- Functions that run in redis workers ------------------- # +# TODO: We might want to move these to their own file, for a bit better separation of +# concerns. + + +# redis preview queue +@exception_as_return_value +@emit_folder_status(before=FolderStatus.PREVIEWING, after=FolderStatus.PREVIEWED) +async def run_preview( + hash: str, + path: str, + group_albums: bool | None, + autotag: bool | None, +): + """Fetch candidates for a folder using beets. + + Will refetch candidates if this is rerun even if candidates exist + in the db. + + Current convention is we have one session for one folder *has*, but + We might have multiple sessions for the same folder **path**. + Previews will **reset** any previous session state in the database, if they + exist for the same folder hash. + + Parameters + ---------- + hash : str + The hash of the folder for which to run the preview. + path : str + The path of the folder for which to run the preview. + group_albums : bool | None + Whether to create multple tasks, one for each album found in the metadata + of the files. Set to true if you have multiple albums in a single folder. + If None: get value from beets config. + autotag : bool | None + Whether to look up metadata online. If None: get value from beets config. + """ + + log.info(f"Preview task on {hash=} {path=}") + + with db_session_factory() as db_session: + f_on_disk = FolderInDb.get_current_on_disk(hash, path) + if hash != f_on_disk.hash: + log.warning( + f"Folder content has changed since the job was scheduled for {path}. " + + f"Using new content ({f_on_disk.hash}) instead of {hash}" + ) + + # here, in preview, we always want to start from a fresh state + # an existing state would skip the candidate lookup. + # otherwise, the retag action would not work, as preview starting from + s_state_live = SessionState(f_on_disk) + p_session = PreviewSession( + s_state_live, group_albums=group_albums, autotag=autotag + ) + + try: + await p_session.run_async() + finally: + # Get max revision for this folder hash + stmt = select(func.max(SessionStateInDb.folder_revision)).where( + SessionStateInDb.folder_hash == hash, + ) + max_rev = db_session.execute(stmt).scalar_one_or_none() + new_rev = 0 if max_rev is None else max_rev + 1 + s_state_indb = SessionStateInDb.from_live_state(p_session.state) + s_state_indb.folder_revision = new_rev + + db_session.merge(s_state_indb) + db_session.commit() + + log.info(f"Preview done. {hash=} {path=}") + return + + +# redis preview queue +@exception_as_return_value +@emit_folder_status(before=FolderStatus.PREVIEWING, after=FolderStatus.PREVIEWED) +async def run_preview_add_candidates( + hash: str, path: str, search: TaskIdMappingArg[Search | Literal["skip"]] +): + """Adds a candidate to an session which is already in the status tagged. + + This only works if all session tasks are tagged. I.e. preview completed. + + Parameters + ---------- + search : dict[str, Search] + A dictionary of task ids to search dicts. No value or none skips the search + for this task. + """ + log.info(f"Add preview candidates task on {hash=}") + + with db_session_factory() as db_session: + s_state_live = _get_live_state_by_folder(hash, path, db_session) + + a_session = AddCandidatesSession( + s_state_live, + search=search, + ) + try: + await a_session.run_async() + finally: + s_state_indb = SessionStateInDb.from_live_state(a_session.state) + db_session.merge(instance=s_state_indb) + db_session.commit() + + log.info(f"Add candidates done. {hash=} {path=}") + + +# redis import queue +@exception_as_return_value +@emit_folder_status(before=FolderStatus.IMPORTING, after=FolderStatus.IMPORTED) +async def run_import_candidate( + hash: str, + path: str, + candidate_ids: TaskIdMappingArg[CandidateChoice], + duplicate_actions: TaskIdMappingArg[DuplicateAction], +): + """Imports a candidate that has been fetched in a preview session. + + Parameters + ---------- + candidate_id : optional + If candidate_id is none the best candidate is used. + duplicate_action : optional + If duplicate_action is none, the default action from the config is used. + """ + log.info(f"Import task on {hash=} {path=}") + + with db_session_factory() as db_session: + s_state_live = _get_live_state_by_folder(hash, path, db_session) + + i_session = ImportSession( + s_state_live, + candidate_ids=candidate_ids, + duplicate_actions=duplicate_actions, + ) + + try: + await i_session.run_async() + finally: + s_state_indb = SessionStateInDb.from_live_state(i_session.state) + db_session.merge(instance=s_state_indb) + db_session.commit() + + # Refresh cached library size so the home page reflects the new import. + lib_path = Path(get_config()["directory"].get(str)) + await compute_and_store_dir_stats(lib_path) + + log.info(f"Import candidate done. {hash=} {path=}") + + +# redis import queue +@exception_as_return_value +@emit_folder_status(before=FolderStatus.IMPORTING, after=FolderStatus.IMPORTED) +async def run_import_auto( + hash: str, + path: str, + import_threshold: float | None, + duplicate_actions: TaskIdMappingArg[DuplicateAction], +): + log.info(f"Auto Import task on {hash=} {path=}") + + with db_session_factory() as db_session: + s_state_live = _get_live_state_by_folder(hash, path, db_session) + i_session = AutoImportSession( + s_state_live, + import_threshold=import_threshold, + duplicate_actions=duplicate_actions, + ) + + try: + await i_session.run_async() + finally: + s_state_indb = SessionStateInDb.from_live_state(i_session.state) + db_session.merge(instance=s_state_indb) + db_session.commit() + + # Refresh cached library size so the home page reflects the new import. + lib_path = Path(get_config()["directory"].get(str)) + await compute_and_store_dir_stats(lib_path) + + log.info(f"Auto Import done. {hash=} {path=}") + + +# redis import queue +@exception_as_return_value +@emit_folder_status(before=FolderStatus.IMPORTING, after=FolderStatus.IMPORTED) +async def run_import_bootleg(hash: str, path: str): + log.info(f"Bootleg Import task on {hash=} {path=}") + + with db_session_factory() as db_session: + # TODO: add duplicate action + # TODO: sort out how to generate previews for asis candidates + s_state_live = _get_live_state_by_folder( + hash, path, create_if_not_exists=True, db_session=db_session + ) + i_session = BootlegImportSession(s_state_live) + + try: + await i_session.run_async() + finally: + s_state_indb = SessionStateInDb.from_live_state(i_session.state) + db_session.merge(instance=s_state_indb) + db_session.commit() + + # Refresh cached library size so the home page reflects the new import. + lib_path = Path(get_config()["directory"].get(str)) + await compute_and_store_dir_stats(lib_path) + + log.info(f"Bootleg Import done. {hash=} {path=}") + + +@exception_as_return_value +@emit_folder_status(before=FolderStatus.DELETING, after=FolderStatus.DELETED) +async def run_import_undo(hash: str, path: str, delete_files: bool): + log.info(f"Import Undo task on {hash=} {path=}") + + with db_session_factory() as db_session: + s_state_live = _get_live_state_by_folder(hash, path, db_session) + i_session = UndoSession(s_state_live, delete_files=delete_files) + + try: + await i_session.run_async() + finally: + s_state_indb = SessionStateInDb.from_live_state(i_session.state) + db_session.merge(instance=s_state_indb) + db_session.commit() + + log.info(f"Import Undo done. {hash=} {path=}") + + +def _get_live_state_by_folder( + hash: str, path: str, db_session: Session, create_if_not_exists=False +) -> SessionState: + f_on_disk = FolderInDb.get_current_on_disk(hash, path) + if hash != f_on_disk.hash: + log.warning( + f"Folder content has changed since the job was scheduled for {path}. " + + f"Using new content ({f_on_disk.hash}) instead of {hash}" + ) + + s_state_indb = SessionStateInDb.get_by_hash_and_path( + # we warn about hash change, and want the import to still run + # but on the old hash. + hash=hash, + path=path, + db_session=db_session, + ) + + if s_state_indb is None and create_if_not_exists: + s_state_live = SessionState(f_on_disk) + return s_state_live + + if s_state_indb is None: + # TODO: rq error handling + raise InvalidUsageException( + f"No session state found for {path=} {hash=} " + + f"fresh_hash_on_disk={f_on_disk}, this should not happen." + ) + + log.debug(f"Using existing session state for {path=}") + s_state_live = s_state_indb.to_live_state() + + # we need this expunge, otherwise we cannot overwrite session states: + # If object id is in session we cant add a new object to the session with the + # same id this will raise (see below session.merge) + db_session.expunge_all() + + return s_state_live + + +def delete_items(task_ids: list[str], delete_files: bool = True): + lib = _open_library(get_config()) + for task_id in task_ids: + delete_from_beets(task_id, delete_files=delete_files, lib=lib) diff --git a/backend/beets_flask/invoker/job.py b/backend/beets_flask/invoker/job.py index 6f66b676..89fc505b 100644 --- a/backend/beets_flask/invoker/job.py +++ b/backend/beets_flask/invoker/job.py @@ -1,34 +1,34 @@ -from __future__ import annotations - -from typing import TYPE_CHECKING, NotRequired, TypedDict - -if TYPE_CHECKING: - from rq.job import Job - - from .enqueue import EnqueueKind - - -class ExtraJobMeta(TypedDict): - job_frontend_ref: NotRequired[str | None] - - -class RequiredJobMeta(TypedDict): - folder_hash: str - folder_path: str - job_id: str - job_kind: str # PS: EnqueueKind not json serializable - - -class JobMeta(RequiredJobMeta, ExtraJobMeta): - pass - - -def _set_job_meta( - job: Job, hash: str, path: str, kind: EnqueueKind, extra: ExtraJobMeta -): - job.meta["folder_hash"] = hash - job.meta["folder_path"] = path - job.meta["job_id"] = job.id - job.meta["job_kind"] = kind.value - job.meta["job_frontend_ref"] = extra.get("job_frontend_ref", None) - job.save_meta() +from __future__ import annotations + +from typing import TYPE_CHECKING, NotRequired, TypedDict + +if TYPE_CHECKING: + from rq.job import Job + + from .enqueue import EnqueueKind + + +class ExtraJobMeta(TypedDict): + job_frontend_ref: NotRequired[str | None] + + +class RequiredJobMeta(TypedDict): + folder_hash: str + folder_path: str + job_id: str + job_kind: str # PS: EnqueueKind not json serializable + + +class JobMeta(RequiredJobMeta, ExtraJobMeta): + pass + + +def _set_job_meta( + job: Job, hash: str, path: str, kind: EnqueueKind, extra: ExtraJobMeta +): + job.meta["folder_hash"] = hash + job.meta["folder_path"] = path + job.meta["job_id"] = job.id + job.meta["job_kind"] = kind.value + job.meta["job_frontend_ref"] = extra.get("job_frontend_ref", None) + job.save_meta() diff --git a/backend/beets_flask/logger.py b/backend/beets_flask/logger.py index d47290d6..211ecfe8 100644 --- a/backend/beets_flask/logger.py +++ b/backend/beets_flask/logger.py @@ -1,64 +1,64 @@ -import logging -import logging.config -import os - -LOGGING_CONFIG = { - "version": 1, - "disable_existing_loggers": True, - "formatters": { - "standard": { - # https://docs.python.org/3/library/logging.html#logrecord-attributes - "format": "[%(levelname)s] %(name)s: %(message)s" - }, - "debug": { - "format": "[%(levelname)-5s] %(asctime)s %(name)s %(filename)-8s:%(lineno)d %(message)s" - }, - }, - "handlers": { - "console": { - "class": "logging.StreamHandler", - "level": "INFO", - "formatter": "standard", - "stream": "ext://sys.stdout", - }, - "file": { - "class": "logging.handlers.RotatingFileHandler", - "level": "DEBUG", - "formatter": "debug", - "filename": os.environ.get("BEETSFLASKLOG", "./beets-flask.log"), - "maxBytes": 1048576, # 1 MB - "backupCount": 3, - }, - }, - "loggers": { - "": { # root logger - "handlers": ["console", "file"], - "level": os.getenv("LOG_LEVEL_OTHERS", logging.WARNING), - "propagate": False, - }, - "beets-flask": { - "handlers": ["console", "file"], - "level": os.getenv("LOG_LEVEL_BEETSFLASK", logging.INFO), - "propagate": False, - }, - }, -} - -# On testing only log to console -if "PYTEST_VERSION" in os.environ: - # Configure minimal logging for pytest - logging.basicConfig( - format="[%(levelname)-5s] %(asctime)s %(name)s %(filename)-8s:%(lineno)d %(message)s", - ) - logging.getLogger("beets-flask").setLevel(logging.DEBUG) -else: - logging.config.dictConfig(LOGGING_CONFIG) - -log = logging.getLogger("beets-flask") - -rq_name = os.getenv("RQ_JOB_ID", None) -if rq_name: - log = log.getChild(rq_name[:4]) - - -log.debug("Logging configured!") +import logging +import logging.config +import os + +LOGGING_CONFIG = { + "version": 1, + "disable_existing_loggers": True, + "formatters": { + "standard": { + # https://docs.python.org/3/library/logging.html#logrecord-attributes + "format": "[%(levelname)s] %(name)s: %(message)s" + }, + "debug": { + "format": "[%(levelname)-5s] %(asctime)s %(name)s %(filename)-8s:%(lineno)d %(message)s" + }, + }, + "handlers": { + "console": { + "class": "logging.StreamHandler", + "level": "INFO", + "formatter": "standard", + "stream": "ext://sys.stdout", + }, + "file": { + "class": "logging.handlers.RotatingFileHandler", + "level": "DEBUG", + "formatter": "debug", + "filename": os.environ.get("BEETSFLASKLOG", "./beets-flask.log"), + "maxBytes": 1048576, # 1 MB + "backupCount": 3, + }, + }, + "loggers": { + "": { # root logger + "handlers": ["console", "file"], + "level": os.getenv("LOG_LEVEL_OTHERS", logging.WARNING), + "propagate": False, + }, + "beets-flask": { + "handlers": ["console", "file"], + "level": os.getenv("LOG_LEVEL_BEETSFLASK", logging.INFO), + "propagate": False, + }, + }, +} + +# On testing only log to console +if "PYTEST_VERSION" in os.environ: + # Configure minimal logging for pytest + logging.basicConfig( + format="[%(levelname)-5s] %(asctime)s %(name)s %(filename)-8s:%(lineno)d %(message)s", + ) + logging.getLogger("beets-flask").setLevel(logging.DEBUG) +else: + logging.config.dictConfig(LOGGING_CONFIG) + +log = logging.getLogger("beets-flask") + +rq_name = os.getenv("RQ_JOB_ID", None) +if rq_name: + log = log.getChild(rq_name[:4]) + + +log.debug("Logging configured!") diff --git a/backend/beets_flask/redis.py b/backend/beets_flask/redis.py index 1f368ca8..5af1dcad 100644 --- a/backend/beets_flask/redis.py +++ b/backend/beets_flask/redis.py @@ -1,70 +1,70 @@ -import asyncio -import time -from concurrent.futures import ThreadPoolExecutor - -from redis import Redis -from rq import Queue -from rq.job import Job - -# Setup redis connection -redis_conn = Redis() - -# Init our different queues -preview_queue = Queue("preview", connection=redis_conn, default_timeout=600) -import_queue = Queue("import", connection=redis_conn, default_timeout=600) - - -queues = [preview_queue, import_queue] - - -async def wait_for_job_results( - job: Job, poll_interval: float = 0.5, timeout: float = 300 -): - """Wait for a job to finish and return the result. - - Parameters - ---------- - job : rq.job.Job - The job to wait for. - poll_interval : float, optional - The interval to poll the job status, by default 0.5 - timeout : float, optional - The timeout for the job, by default 300 - - Raises - ------ - Exception - If the job fails or times out. - - Returns - ------- - Any - The result of the job. - """ - - start_time = time.time() - - with ThreadPoolExecutor() as executor: - while True: - # Check if the timeout has been exceeded - elapsed_time = time.time() - start_time - if elapsed_time > timeout: - raise Exception(f"Job timed out after {timeout} seconds") - - await asyncio.get_event_loop().run_in_executor(executor, job.refresh) - - if job.is_finished: - return job.return_value(False) - if job.is_failed: - raise Exception(f"Job failed: {job.exc_info}") - # Wait for the job to finish - await asyncio.sleep(poll_interval) - - -__all__ = [ - "queues", - "import_queue", - "preview_queue", - "redis_conn", - "wait_for_job_results", -] +import asyncio +import time +from concurrent.futures import ThreadPoolExecutor + +from redis import Redis +from rq import Queue +from rq.job import Job + +# Setup redis connection +redis_conn = Redis() + +# Init our different queues +preview_queue = Queue("preview", connection=redis_conn, default_timeout=600) +import_queue = Queue("import", connection=redis_conn, default_timeout=600) + + +queues = [preview_queue, import_queue] + + +async def wait_for_job_results( + job: Job, poll_interval: float = 0.5, timeout: float = 300 +): + """Wait for a job to finish and return the result. + + Parameters + ---------- + job : rq.job.Job + The job to wait for. + poll_interval : float, optional + The interval to poll the job status, by default 0.5 + timeout : float, optional + The timeout for the job, by default 300 + + Raises + ------ + Exception + If the job fails or times out. + + Returns + ------- + Any + The result of the job. + """ + + start_time = time.time() + + with ThreadPoolExecutor() as executor: + while True: + # Check if the timeout has been exceeded + elapsed_time = time.time() - start_time + if elapsed_time > timeout: + raise Exception(f"Job timed out after {timeout} seconds") + + await asyncio.get_event_loop().run_in_executor(executor, job.refresh) + + if job.is_finished: + return job.return_value(False) + if job.is_failed: + raise Exception(f"Job failed: {job.exc_info}") + # Wait for the job to finish + await asyncio.sleep(poll_interval) + + +__all__ = [ + "queues", + "import_queue", + "preview_queue", + "redis_conn", + "wait_for_job_results", +] diff --git a/backend/beets_flask/server/app.py b/backend/beets_flask/server/app.py index 49a4d101..23ae108a 100644 --- a/backend/beets_flask/server/app.py +++ b/backend/beets_flask/server/app.py @@ -1,86 +1,86 @@ -from __future__ import annotations - -import json -import os -from dataclasses import asdict, is_dataclass -from datetime import date, datetime -from typing import TYPE_CHECKING, Any - -from quart import Quart - -from ..config.flask_config import ServerConfig, init_server_config -from ..logger import log - -if TYPE_CHECKING: - from ..config.flask_config import ServerConfig - - -def create_app(config: str | ServerConfig | None = None) -> Quart: - config = config or os.getenv("BEETSFLASK_ENV", None) - # create and configure the app - app = Quart(__name__, instance_relative_config=True) - - config = init_server_config(config) - app.config.from_object(config) - # make routes with and without trailing slahes the same - app.url_map.strict_slashes = False - app.json = CustomProvider(app) - - global socketio - # app.wsgi_app = socketio.WSGIApp(sio, app.wsgi_app) - - # sqlite - from ..database import setup_database - - setup_database(app) - - # Register different blueprints & websocket routes - # In production, we use the frontend.py route to deliver vite's dist folder - from .routes import register_routes - from .websocket import register_socketio - - register_routes(app) - register_socketio(app) - - log.debug("Quart app created!") - - return app - - -# ------------------------------- Json encoder ------------------------------- # -# Allows to serialize bytes and datetime objects in dictionaries to json -# The default encoder does not support this! -# Has to be added to the app with app.json = CustomProvider(app) -# FIXME: We might be able to remove this once our serialized state does not -# contain bytes or datetime objects - -from enum import Enum - -from quart.json.provider import DefaultJSONProvider - - -class CustomProvider(DefaultJSONProvider): - def dumps(self, obj: Any, **kwargs: Any) -> str: - return json.dumps(obj, cls=Encoder, **kwargs) - - -class Encoder(json.JSONEncoder): - def default(self, o): - if isinstance(o, bytes): - # Mainly used for paths - # b'/path/to/file' -> '/path/to/file' - # Might yield strange results for other byte objects - return o.decode("utf-8") - - if isinstance(o, (datetime, date)): - return o.isoformat() - - # Dataclasses are not serializable by default - if is_dataclass(o) and not isinstance(o, type): - return asdict(o) - - # Enum values are not serializable by default - if isinstance(o, Enum): - return o.value - - return json.JSONEncoder.default(self, o) +from __future__ import annotations + +import json +import os +from dataclasses import asdict, is_dataclass +from datetime import date, datetime +from typing import TYPE_CHECKING, Any + +from quart import Quart + +from ..config.flask_config import ServerConfig, init_server_config +from ..logger import log + +if TYPE_CHECKING: + from ..config.flask_config import ServerConfig + + +def create_app(config: str | ServerConfig | None = None) -> Quart: + config = config or os.getenv("BEETSFLASK_ENV", None) + # create and configure the app + app = Quart(__name__, instance_relative_config=True) + + config = init_server_config(config) + app.config.from_object(config) + # make routes with and without trailing slahes the same + app.url_map.strict_slashes = False + app.json = CustomProvider(app) + + global socketio + # app.wsgi_app = socketio.WSGIApp(sio, app.wsgi_app) + + # sqlite + from ..database import setup_database + + setup_database(app) + + # Register different blueprints & websocket routes + # In production, we use the frontend.py route to deliver vite's dist folder + from .routes import register_routes + from .websocket import register_socketio + + register_routes(app) + register_socketio(app) + + log.debug("Quart app created!") + + return app + + +# ------------------------------- Json encoder ------------------------------- # +# Allows to serialize bytes and datetime objects in dictionaries to json +# The default encoder does not support this! +# Has to be added to the app with app.json = CustomProvider(app) +# FIXME: We might be able to remove this once our serialized state does not +# contain bytes or datetime objects + +from enum import Enum + +from quart.json.provider import DefaultJSONProvider + + +class CustomProvider(DefaultJSONProvider): + def dumps(self, obj: Any, **kwargs: Any) -> str: + return json.dumps(obj, cls=Encoder, **kwargs) + + +class Encoder(json.JSONEncoder): + def default(self, o): + if isinstance(o, bytes): + # Mainly used for paths + # b'/path/to/file' -> '/path/to/file' + # Might yield strange results for other byte objects + return o.decode("utf-8") + + if isinstance(o, (datetime, date)): + return o.isoformat() + + # Dataclasses are not serializable by default + if is_dataclass(o) and not isinstance(o, type): + return asdict(o) + + # Enum values are not serializable by default + if isinstance(o, Enum): + return o.value + + return json.JSONEncoder.default(self, o) diff --git a/backend/beets_flask/server/exceptions.py b/backend/beets_flask/server/exceptions.py index 0375d1cc..07d3eb1e 100644 --- a/backend/beets_flask/server/exceptions.py +++ b/backend/beets_flask/server/exceptions.py @@ -1,212 +1,212 @@ -import traceback -from collections.abc import Awaitable, Callable -from functools import wraps -from typing import NotRequired, ParamSpec, TypedDict, TypeVar - -from beets_flask.logger import log - - -class SerializedException(TypedDict): - """Serialized exception format. - - This is used to serialize exceptions to a common format. - The format is as follows: - { - "type": "Exception type", - "message": "Error message", - "description": "Error description (optional)" - } - """ - - type: str - message: str - description: NotRequired[str | None] - trace: NotRequired[str | None] - - -class ApiException(Exception): - """Base class for all API errors.""" - - persist_in_db: bool - """If true, the exception will be stored in the database on - raise in sessions. - TODO: Think about exception hierarchy. - """ - status_code: int = 500 - - def __init__( - self, *args, status_code: int | None = None, persist_in_db: bool = True - ): - super().__init__(*args) - if status_code is not None: - self.status_code = status_code - self.persist_in_db = persist_in_db - - -class InvalidUsageException(ApiException): - """Invalid usage of the API. - - This is used to indicate that the API was used incorrectly. - """ - - status_code: int = 400 - - -class NotFoundException(ApiException): - """Resource not found. - - This is used to indicate that the requested resource was not found. - """ - - status_code: int = 404 - - -class IntegrityException(ApiException): - """Integrity error. - - This is used to indicate that the requested resource was not found. - """ - - status_code: int = 409 - - -class NotImportedException(ApiException): - """Not imported error. - - So far only used for the auto import session, when the best - match is worse than the threshold. - """ - - status_code: int = 409 - - -class NoCandidatesFoundException(ApiException): - """No candidates found error. - - Raised when an online search does not return any candidates. - Could be raised from automatic search (without searchid) but also when manually - adding more candidates via interactive search (searchid given). - """ - - status_code: int = 409 - - def __init__( - self, *args, status_code: int | None = None, persist_in_db: bool = True - ): - if not args: - error_text = "Lookup found no candidates. " + self.metadata_plugin_info() - args = (error_text,) - - super().__init__(*args, status_code=status_code, persist_in_db=persist_in_db) - - @classmethod - def metadata_plugin_info(cls) -> str: - # Get enabled metadata source plugins to give a better error message - error_text = "" - try: - from beets.metadata_plugins import find_metadata_source_plugins - - meta_plugins: list[str] = [ - p.data_source for p in find_metadata_source_plugins() - ] - if len(meta_plugins) > 0: - error_text += f"Used '{', '.join(meta_plugins)}' as metadata source(s)." - else: - error_text += "No source plugins are enabled." - - except: - error_text += "Could not determine enabled metadata source plugins." - return error_text - - -class UserException(Exception): - """Base class for errors caused by user input or config.""" - - status_code: int = 422 - - def __init__(self, *args, status_code: int | None = None): - super().__init__(*args) - if status_code is not None: - self.status_code = status_code - - -class DuplicateException(UserException): - """Duplicate error. - - Raised when we have trouble resolving duplicates in the beets library. - Users should check their config and api usage. - """ - - status_code: int = 422 - - -def to_serialized_exception( - exception: Exception, -) -> SerializedException: - """Convert an exception to a serialized format. - - Parameters - ---------- - exception : Exception | None - The exception to serialize. - - Returns - ------- - SerializedException - The serialized exception. - """ - - if exception is None: - return None - - tb: str | None = None - - if exception.__traceback__ is not None: - tb = "".join(traceback.format_tb(exception.__traceback__)) - - return SerializedException( - type=exception.__class__.__name__, - message=str(exception), - description=exception.__doc__, - trace=tb, - ) - - -P = ParamSpec("P") # Parameters -R = TypeVar("R") # Return - - -def exception_as_return_value( - f: Callable[P, Awaitable[R]], -) -> Callable[P, Awaitable[R | SerializedException]]: - """Decorator to catch exceptions and return them as a values. - - This is used to catch exceptions in the redis worker and return them - as a values we can use in the frontend. Sadly standard exeption handling - in rq is lacking! - """ - - @wraps(f) - async def wrapper(*args: P.args, **kwargs: P.kwargs) -> R | SerializedException: - try: - return await f(*args, **kwargs) - # Some exceptions are not serializable, so we need to convert them to a - # serialized format. E.g. OSErrors - except ApiException as e: - log.info(e) - return to_serialized_exception(e) - except Exception as e: - log.exception(e) - return to_serialized_exception(e) - - return wrapper - - -__all__ = [ - "SerializedException", - "ApiException", - "InvalidUsageException", - "NotFoundException", - "IntegrityException", - "to_serialized_exception", -] +import traceback +from collections.abc import Awaitable, Callable +from functools import wraps +from typing import NotRequired, ParamSpec, TypedDict, TypeVar + +from beets_flask.logger import log + + +class SerializedException(TypedDict): + """Serialized exception format. + + This is used to serialize exceptions to a common format. + The format is as follows: + { + "type": "Exception type", + "message": "Error message", + "description": "Error description (optional)" + } + """ + + type: str + message: str + description: NotRequired[str | None] + trace: NotRequired[str | None] + + +class ApiException(Exception): + """Base class for all API errors.""" + + persist_in_db: bool + """If true, the exception will be stored in the database on + raise in sessions. + TODO: Think about exception hierarchy. + """ + status_code: int = 500 + + def __init__( + self, *args, status_code: int | None = None, persist_in_db: bool = True + ): + super().__init__(*args) + if status_code is not None: + self.status_code = status_code + self.persist_in_db = persist_in_db + + +class InvalidUsageException(ApiException): + """Invalid usage of the API. + + This is used to indicate that the API was used incorrectly. + """ + + status_code: int = 400 + + +class NotFoundException(ApiException): + """Resource not found. + + This is used to indicate that the requested resource was not found. + """ + + status_code: int = 404 + + +class IntegrityException(ApiException): + """Integrity error. + + This is used to indicate that the requested resource was not found. + """ + + status_code: int = 409 + + +class NotImportedException(ApiException): + """Not imported error. + + So far only used for the auto import session, when the best + match is worse than the threshold. + """ + + status_code: int = 409 + + +class NoCandidatesFoundException(ApiException): + """No candidates found error. + + Raised when an online search does not return any candidates. + Could be raised from automatic search (without searchid) but also when manually + adding more candidates via interactive search (searchid given). + """ + + status_code: int = 409 + + def __init__( + self, *args, status_code: int | None = None, persist_in_db: bool = True + ): + if not args: + error_text = "Lookup found no candidates. " + self.metadata_plugin_info() + args = (error_text,) + + super().__init__(*args, status_code=status_code, persist_in_db=persist_in_db) + + @classmethod + def metadata_plugin_info(cls) -> str: + # Get enabled metadata source plugins to give a better error message + error_text = "" + try: + from beets.metadata_plugins import find_metadata_source_plugins + + meta_plugins: list[str] = [ + p.data_source for p in find_metadata_source_plugins() + ] + if len(meta_plugins) > 0: + error_text += f"Used '{', '.join(meta_plugins)}' as metadata source(s)." + else: + error_text += "No source plugins are enabled." + + except: + error_text += "Could not determine enabled metadata source plugins." + return error_text + + +class UserException(Exception): + """Base class for errors caused by user input or config.""" + + status_code: int = 422 + + def __init__(self, *args, status_code: int | None = None): + super().__init__(*args) + if status_code is not None: + self.status_code = status_code + + +class DuplicateException(UserException): + """Duplicate error. + + Raised when we have trouble resolving duplicates in the beets library. + Users should check their config and api usage. + """ + + status_code: int = 422 + + +def to_serialized_exception( + exception: Exception, +) -> SerializedException: + """Convert an exception to a serialized format. + + Parameters + ---------- + exception : Exception | None + The exception to serialize. + + Returns + ------- + SerializedException + The serialized exception. + """ + + if exception is None: + return None + + tb: str | None = None + + if exception.__traceback__ is not None: + tb = "".join(traceback.format_tb(exception.__traceback__)) + + return SerializedException( + type=exception.__class__.__name__, + message=str(exception), + description=exception.__doc__, + trace=tb, + ) + + +P = ParamSpec("P") # Parameters +R = TypeVar("R") # Return + + +def exception_as_return_value( + f: Callable[P, Awaitable[R]], +) -> Callable[P, Awaitable[R | SerializedException]]: + """Decorator to catch exceptions and return them as a values. + + This is used to catch exceptions in the redis worker and return them + as a values we can use in the frontend. Sadly standard exeption handling + in rq is lacking! + """ + + @wraps(f) + async def wrapper(*args: P.args, **kwargs: P.kwargs) -> R | SerializedException: + try: + return await f(*args, **kwargs) + # Some exceptions are not serializable, so we need to convert them to a + # serialized format. E.g. OSErrors + except ApiException as e: + log.info(e) + return to_serialized_exception(e) + except Exception as e: + log.exception(e) + return to_serialized_exception(e) + + return wrapper + + +__all__ = [ + "SerializedException", + "ApiException", + "InvalidUsageException", + "NotFoundException", + "IntegrityException", + "to_serialized_exception", +] diff --git a/backend/beets_flask/server/routes/__init__.py b/backend/beets_flask/server/routes/__init__.py index f0843e2c..6434f7e2 100644 --- a/backend/beets_flask/server/routes/__init__.py +++ b/backend/beets_flask/server/routes/__init__.py @@ -1,33 +1,33 @@ -from quart import Blueprint, Quart - -from .art_preview import art_blueprint -from .config import config_bp -from .db_models import register_state_models -from .exception import error_bp -from .frontend import frontend_bp -from .inbox import inbox_bp -from .library import library_bp -from .monitor import monitor_bp - -backend_bp = Blueprint("backend", __name__, url_prefix="/api_v1") - -# Register all backend blueprints -backend_bp.register_blueprint(art_blueprint) -backend_bp.register_blueprint(config_bp) -backend_bp.register_blueprint(error_bp) -backend_bp.register_blueprint(frontend_bp) -backend_bp.register_blueprint(inbox_bp) -backend_bp.register_blueprint(library_bp) -backend_bp.register_blueprint(monitor_bp) - - -def register_routes(app: Quart): - # Register database state models - # to api blueprint i.e. /api_v1/session, /api_v1/task & /api_v1/candidate - register_state_models(backend_bp) - - app.register_blueprint(backend_bp) - app.register_blueprint(frontend_bp) - - -__all__ = ["register_routes"] +from quart import Blueprint, Quart + +from .art_preview import art_blueprint +from .config import config_bp +from .db_models import register_state_models +from .exception import error_bp +from .frontend import frontend_bp +from .inbox import inbox_bp +from .library import library_bp +from .monitor import monitor_bp + +backend_bp = Blueprint("backend", __name__, url_prefix="/api_v1") + +# Register all backend blueprints +backend_bp.register_blueprint(art_blueprint) +backend_bp.register_blueprint(config_bp) +backend_bp.register_blueprint(error_bp) +backend_bp.register_blueprint(frontend_bp) +backend_bp.register_blueprint(inbox_bp) +backend_bp.register_blueprint(library_bp) +backend_bp.register_blueprint(monitor_bp) + + +def register_routes(app: Quart): + # Register database state models + # to api blueprint i.e. /api_v1/session, /api_v1/task & /api_v1/candidate + register_state_models(backend_bp) + + app.register_blueprint(backend_bp) + app.register_blueprint(frontend_bp) + + +__all__ = ["register_routes"] diff --git a/backend/beets_flask/server/routes/art_preview.py b/backend/beets_flask/server/routes/art_preview.py index f932d8c6..6988a065 100644 --- a/backend/beets_flask/server/routes/art_preview.py +++ b/backend/beets_flask/server/routes/art_preview.py @@ -1,109 +1,109 @@ -"""Fetch art via ids. - -Allows fetching of art via different ids. At the moment we support: -- Spotify album ids (if spotify plugin is enabled) -""" - -import os -from urllib.parse import quote_plus - -import aiohttp -from quart import Blueprint, jsonify, redirect, request, url_for - -from beets_flask.logger import log -from beets_flask.utility import AUDIO_EXTENSIONS - -art_blueprint = Blueprint("art", __name__, url_prefix="/art") - - -@art_blueprint.route("", methods=["GET"]) -async def redirect_external_art(): - """Get Spotify album art.""" - - # Check that url query param is set - url = request.args.get("url") - if not url: - return jsonify({"error": "url query param is required."}), 400 - - # Check that url is a valid spotify url - redirect_url: str | None = None - if "spotify" in url: - redirect_url = await get_spotify_art(url) - elif "musicbrainz" in url: - redirect_url = await get_musicbrainz_art(url) - elif url.startswith("file://"): - return await get_folder_art(url) - - if redirect_url: - return redirect(redirect_url, code=302) - else: - return jsonify({"error": "No art found."}), 404 - - -async def get_spotify_art(url: str) -> str | None: - """Uses spotify oembed to redirect to the album art. - - See https://developer.spotify.com/documentation/embeds/reference/oembed - - Returns the url the the art. - """ - print(f"https://embed.spotify.com/oembed?url={quote_plus(url)}") - async with aiohttp.ClientSession() as session: - async with session.get( - f"https://embed.spotify.com/oembed?url={quote_plus(url)}" - ) as response: - if response.status == 200: - data = await response.json() - return data.get("thumbnail_url") - else: - log.error(f"Error fetching Spotify art: {response.status}") - return None - - -async def get_musicbrainz_art(url: str) -> str | None: - """Uses musicbrainz oembed to redirect to the album art. - - See https://musicbrainz.org/doc/Cover_Art_Archive/API - - Returns the url the the art. - """ - - # Extract the release id from the url - # musicbrainz urls look like this: - # https://musicbrainz.org/release/2b5f7e4d-2a1c-4f6d-8a0c-7b8b9e3a1f3f - release_id = url.split("/")[-1] - - return f"https://coverartarchive.org/release/{release_id}/front-250" - - -async def get_folder_art(url: str): - """Infers the folder art from a given file path. - - This is a bit of a hack, but it works for now. - url="file:///path/to/music/folder" - """ - - # Check first file for and embedded cover art - path = url.split("file://")[-1] - print(path) - # Check if exists - if not os.path.exists(path): - return jsonify({"error": f"Path '{path}' does not exist."}), 404 - - # Get first file in folder - files = [ - f - for f in os.listdir(path) - if f.endswith(tuple(["." + e for e in AUDIO_EXTENSIONS])) - ] - if not files or len(files) < 1: - return jsonify({"error": "No audio files found in folder."}), 404 - - # Redirect to file art endpoint /file/