fix: reduce bind-mount mariadb corruption risk

.gitignore

@@ -16,6 +16,7 @@ server/images
 
 .env
 .env.dev
+.env.bak.*
 
 # Local dev exclusion
 ffmpeg.exe
@@ -57,6 +58,9 @@ downloads/*
 # Backup archive location
 backups/
 
+# Database migration safety backups
+database.bind-mount-backup.*/
+
 # Superpowers brainstorming session artifacts (local-only, ephemeral mockups)
 .superpowers/
 

README.md

@@ -57,6 +57,8 @@ You'll need Docker, Docker Compose, Git, and a Bash shell (Git Bash on Windows).
 
 > Want to try unreleased features? See [Using Development Builds](docs/DEVELOPMENT.md#using-development-builds) for the bleeding-edge `dev-latest` image.
 
+> **Database note for Docker Desktop/ARM/NAS users:** Fresh installs started with `./start.sh` use Docker named-volume storage for MariaDB. Existing bind-mounted installs and plain `docker compose up -d` installs may use `./database`; on virtualized filesystems this can be risky for MariaDB schema migrations. If you see `Table ... doesn't exist in engine` or `Incorrect information in file` errors, or if you want to proactively migrate, see [Database Management](docs/DATABASE.md#migrating-from-bind-mount-to-named-volume).
+
 ## Documentation
 
 ### Setup & Configuration

docker-compose.arm.yml

@@ -1,5 +1,7 @@
-# Override file for ARM-based systems (Apple Silicon, Raspberry Pi, etc.)
-# This uses a named volume for MariaDB to work around virtiofs bugs on ARM
+# Named-volume database override.
+# The filename is historical: it was originally added for ARM systems, but it is
+# also useful on Docker Desktop and NAS/virtualized filesystems where bind-mounted
+# MariaDB data can corrupt during schema migrations.
 
 services:
   youtarr-db:
@@ -8,4 +10,3 @@ services:
 
 volumes:
   youtarr-db-data:
-

docker-compose.yml

@@ -15,10 +15,12 @@
 # mkdir -p config jobs server/images /path/to/youtube_videos
 # sudo chown -R 1000:1000 config jobs server/images /path/to/youtube_videos
 #
-# 2) Database setup notes for Synology and ARM-based systems (Apple Silicon, Raspberry Pi):
-# There are known issues with using bind mounts for MariaDB data directories due to permission and virtiofs bugs.
-# The start scripts automatically detect ARM architecture and use docker-compose.arm.yml override to switch to named volumes.
-# For Synology or manual override, run: docker compose -f docker-compose.yml -f docker-compose.arm.yml up -d
+# 2) Database setup notes:
+# The default bundled MariaDB uses ./database as a bind mount for backwards compatibility.
+# On Docker Desktop (Windows/macOS), ARM hosts, and some NAS/virtualized filesystems,
+# bind-mounted MariaDB data can corrupt during schema migrations. For a safer named
+# volume, use docker-compose.arm.yml or run ./scripts/migrate-to-named-volume.sh
+# for an existing bind-mounted install.
 
 services:
   youtarr-db:
@@ -96,6 +98,6 @@ networks:
   default:
     name: youtarr-network
 
-# Named volume definition (used by docker-compose.arm.yml override for ARM systems)
+# Named volume definition (used by docker-compose.arm.yml override)
 volumes:
   youtarr-db-data:

docs/DATABASE.md

@@ -45,52 +45,102 @@ Youtarr uses MariaDB/MySQL for storing:
 
 ### Storage Options
 
-#### Option 1: Bind Mount (Default)
+#### Option 1: Bind Mount (legacy / pre-existing installs)
 ```yaml
 volumes:
   - ./database:/var/lib/mysql
 ```
 - Data stored in `./database` directory on the host
-- May have permission issues on Synology/QNAP and or virtiofs issues on Apple Silicon macOS, leading to database issues/corruption
+- Kept for backwards compatibility with existing bind-mounted installs and plain `docker compose up -d` users
+- Works well on native Linux Docker hosts
+- Can have permission issues on Synology/QNAP
+- Can corrupt during MariaDB schema migrations on Docker Desktop for Windows/macOS, ARM hosts, and some virtualized filesystems
 
-#### Option 2: Named Volume (Recommended for Synology/Apple)
+#### Option 2: Named Volume (Recommended for Docker Desktop/ARM/NAS)
 ```yaml
 volumes:
   - youtarr-db-data:/var/lib/mysql
 ```
 - Better compatibility with Synology/QNAP
-- Avoids permission issues
-- Required for macOS Apple Silicon
-- Not easily visible on host
+- Avoids the virtualized-filesystem write semantics problem that can affect bind-mounted MariaDB
+- Used automatically for fresh installs started with `./start.sh` on every platform (Linux included, since v1.69)
+- Recommended for Docker Desktop on Windows/macOS, ARM systems, and NAS setups
+- Not easily visible on host: data lives under `/var/lib/docker/volumes/<project>_youtarr-db-data/_data` rather than `./database/`. `./scripts/backup.sh` dumps from the running MariaDB container when it is already up; when it has to start MariaDB for a backup, it detects whether this install uses the bind mount or named volume first.
 
-### Switching to Named Volume
+### Migrating from Bind Mount to Named Volume
 
-If experiencing permission errors on Synology/QNAP or corruption on Apple Silicon:
+If you already have Youtarr data in `./database/`, do **not** switch the compose mount by hand unless you intentionally want to start with an empty database. Use the migration helper instead:
 
-**IMPORTANT**: Changing your DB volume mount will *not* migrate your existing database! If you are not experiencing problems, leave this setting alone!
+```bash
+./scripts/migrate-to-named-volume.sh
+```
 
-1. Stop the stack:
-   `docker compose down` or `./stop.sh`
+What the script does (in this order, so any failure leaves the simplest possible recovery state):
+1. Runs a pre-flight permissions check so it fails fast (instead of stalling on an interactive `sudo` prompt) if it cannot write to the project directory.
+2. Stops Youtarr.
+3. Starts the existing bind-mounted MariaDB long enough to run `mysqldump` and to capture per-table row counts.
+4. Renames `./database/` to `./database.bind-mount-backup.<timestamp>/` so the original files are preserved.
+5. Starts a fresh named-volume MariaDB and imports the dump.
+6. Verifies that the table set matches the source **and** that every table has the same row count as the source.
+7. **Only after verification succeeds**, snapshots `.env` to `./.env.bak.<timestamp>` and pins `COMPOSE_PATH_SEPARATOR=:` and `COMPOSE_FILE=docker-compose.yml:docker-compose.arm.yml` in `.env`. This means a failure during step 5 or 6 leaves `.env` untouched, and recovery is just `mv ./database.bind-mount-backup.<timestamp> ./database` plus removing the partial named volume.
+8. Brings the full stack (app + database) back up so Youtarr is immediately usable.
 
-2. Edit `docker-compose.yml`:
-   ```yaml
-   # Change from:
-   volumes:
-     - ./database:/var/lib/mysql
+**What the migration does *not* copy**: `mysqldump` runs with `--single-transaction --routines --triggers --events`. Schema, data, stored routines, triggers, and events all migrate. MariaDB users and `GRANT` statements (anything in `mysql.user` / `mysql.db`) do **not**. The default Youtarr install only uses the bundled `root` user, so this is a no-op for almost everyone. If you have created additional database users on the bundled MariaDB, recreate them after the migration completes.
 
-   # To:
-   volumes:
-     - youtarr-db-data:/var/lib/mysql
-   ```
+**Password note**: for the bundled `root` database user, `DB_ROOT_PASSWORD` seeds the root password when a fresh MariaDB data directory is initialized, while Youtarr connects with `DB_PASSWORD`. The migration requires those two values to match before it creates the new named-volume database.
 
-3. Add volume definition at bottom of file:
-   ```yaml
-   volumes:
-     youtarr-db-data:
+After it completes, the stack is already running. Subsequent restarts can use any of:
+
+```bash
+./start.sh                                                       # recommended
+docker compose up -d                                             # the script pins COMPOSE_FILE in .env
+docker compose -f docker-compose.yml -f docker-compose.arm.yml up -d  # explicit override
+```
+
+### Reverting to Bind Mount
+
+The migration is reversible:
+
+1. Stop the stack:
+   ```bash
+   ./stop.sh
+   ```
+2. Restore the `.env` snapshot:
+   ```bash
+   mv ./.env.bak.<timestamp> .env
    ```
+3. Remove the named volume for this install. The name is usually `<project>_youtarr-db-data`:
+   ```bash
+   docker volume ls --format '{{.Name}}' | grep -E '(^|_)youtarr-db-data$'
+   docker volume rm <volume-name>
+   ```
+4. Restore the original bind-mounted database directory:
+   ```bash
+   mv ./database.bind-mount-backup.<timestamp> ./database
+   ```
+5. Start Youtarr:
+   ```bash
+   ./start.sh
+   ```
+
+Changes made while running on the named volume are not present in the old bind-mounted backup. If you have used the named volume for a while and want to keep those newer changes, take a backup first with `./scripts/backup.sh`.
+
+### Fresh Installs with Named Volume
+
+For a new install with no data to preserve, you can start directly with the named-volume override:
+
+```bash
+docker compose -f docker-compose.yml -f docker-compose.arm.yml up -d
+```
+
+Or pin the override in `.env` so plain `docker compose up -d` uses it:
+
+```env
+COMPOSE_PATH_SEPARATOR=:
+COMPOSE_FILE=docker-compose.yml:docker-compose.arm.yml
+```
 
-4. Restart:
-   `docker compose up -d` or `./start.sh`
+`COMPOSE_PATH_SEPARATOR=:` is important on Windows so Compose parses the file list consistently.
 
 ### Security Considerations
 
@@ -214,11 +264,11 @@ npm run db:create-migration -- --name my-migration-name
 
 #### Common Causes
 1. **Synology/QNAP NAS**: MariaDB runs as UID 999, which may not exist
-2. **macOS Apple Silicon**: virtiofs incompatibility with MariaDB 10.3
+2. **Docker Desktop/ARM/NAS**: virtualized filesystem or permission issues with bind-mounted MariaDB data
 3. **Wrong ownership**: Database files owned by incorrect user
 
 #### Solutions
-1. **Switch to named volume** (see above)
+1. **Migrate to named volume** (see above)
 2. **Fix permissions**:
    ```bash
    # Check current ownership

docs/DEVELOPMENT.md

@@ -14,11 +14,11 @@ The `dev-latest` tag always points at the most recent dev build. Each commit als
 
 This pulls `dialmaster/youtarr:dev-latest` and starts the stack. On later runs, drop `--pull-latest` if you want to stay on the image you already have locally.
 
-### Alternative: bypass `./start.sh` (safer for existing installs on ARM)
+### Alternative: bypass `./start.sh`
 
-`./start.sh` auto-detects ARM hosts (Apple Silicon, Raspberry Pi, ARM Linux) and layers `docker-compose.arm.yml` on top of the default compose file. That override switches MariaDB from the bind mount at `./database/` to a named volume. If you already have data in `./database/` and you run `./start.sh` on ARM for the first time, the container boots against an empty database and it looks like all your channels and videos disappeared. The data is fine, it's still sitting in `./database/`, but the running container isn't pointing at it.
+`./start.sh` layers `docker-compose.arm.yml` on top of the default compose file for fresh installs so MariaDB uses a named volume. Existing installs with real MariaDB data in `./database/` keep using the bind mount and print a migration warning instead. ARM installs continue to use the named-volume override.
 
-To pull the dev image without letting `./start.sh` touch your compose file selection, use docker directly:
+To pull the dev image while managing compose file selection yourself, use docker directly:
 
 ```bash
 ./stop.sh

docs/DOCKER.md

@@ -14,7 +14,7 @@ Youtarr ships four Compose files so each supported runtime can layer the right o
 |------|---------|
 | `docker-compose.yml` | Production defaults with the bundled MariaDB container. Used by `./start.sh`. |
 | `docker-compose.dev.yml` | Development mode: mounts `./server/` and migrations into the container, runs the backend with `node --watch` for hot reload, and uses a separate `youtarr-db-dev` database with its own named volume. Used by `./scripts/start-dev.sh`. See [DEVELOPMENT.md](DEVELOPMENT.md). |
-| `docker-compose.arm.yml` | ARM override (Apple Silicon, Raspberry Pi) that switches the MariaDB data directory to a named volume to avoid virtiofs issues. Layered on top of `docker-compose.yml` via `-f`. |
+| `docker-compose.arm.yml` | Named-volume database override. The filename is historical: it was originally added for ARM systems, but it is also useful on Docker Desktop and NAS/virtualized filesystems. Layered on top of `docker-compose.yml` via `-f`. |
 | `docker-compose.external-db.yml` | Runs Youtarr against an external MariaDB/MySQL instance instead of the bundled database. Used by `./start-with-external-db.sh`. |
 
 ## Container Details
@@ -34,10 +34,10 @@ Youtarr ships four Compose files so each supported runtime can layer the right o
 - **Port**: 3321 (both host and container)
 - **Volumes**:
   - `./database:/var/lib/mysql` - Database persistence (default)
-  - `youtarr-db-data:/var/lib/mysql` - Named volume (required for ARM/Synology)
+  - `youtarr-db-data:/var/lib/mysql` - Named volume (recommended for Docker Desktop/ARM/NAS)
 - **Character Set**: utf8mb4 (full Unicode support)
 
-> **ARM Users**: See [ARM Architecture Notes](#arm-architecture-apple-silicon-raspberry-pi) below.
+> **Docker Desktop/ARM/NAS users**: See [Named-Volume Database Override](#named-volume-database-override) below.
 
 ## ⚠️ Important: Do Not Mount the Migrations Directory
 
@@ -62,25 +62,30 @@ volumes:
 
 If your automation creates a migrations directory, remove it from both directory creation and volume mounts.
 
-## ARM Architecture (Apple Silicon, Raspberry Pi)
+## Named-Volume Database Override
 
-ARM-based systems (Apple Silicon Macs, Raspberry Pi, etc.) have known issues with MariaDB bind mounts due to virtiofs bugs. The start scripts automatically detect ARM and apply the fix.
+Docker Desktop on Windows/macOS, ARM hosts, Synology/QNAP, and some virtualized filesystems can have trouble with MariaDB data stored on a bind mount. The named-volume override avoids that class of issue.
 
 ### Using Start Scripts (Recommended)
 
-The `./start.sh` script automatically detects ARM architecture and applies the correct configuration:
+`./start.sh` automatically uses `docker-compose.arm.yml` for fresh installs on every platform. Existing installs with real MariaDB data in `./database/` keep using the bind mount and print a migration warning instead, including on ARM hosts.
 ```bash
 ./start.sh
 ```
 
+For an existing bind-mounted install, migrate with:
+```bash
+./scripts/migrate-to-named-volume.sh
+```
+
 ### Using Docker Compose Directly
 
-If you prefer running `docker compose` commands directly on ARM systems, use the override file:
+For a fresh install, use the override file:
 ```bash
 docker compose -f docker-compose.yml -f docker-compose.arm.yml up -d
 ```
 
-This uses a named Docker volume instead of a bind mount for MariaDB data, avoiding the virtiofs issues.
+This uses a named Docker volume instead of a bind mount for MariaDB data.
 
 ### Manual Configuration
 
@@ -99,7 +104,7 @@ volumes:
   youtarr-db-data:
 ```
 
-See [Troubleshooting](TROUBLESHOOTING.md#apple-silicon--arm-incorrect-information-in-file-errors) for more details on the underlying issue.
+See [Troubleshooting](TROUBLESHOOTING.md#docker-desktop--arm-incorrect-information-in-file-errors) for more details on the underlying issue.
 
 ## Configuration Setup
 - **Create a .env file** to configure environment variables:
@@ -246,7 +251,7 @@ On most Linux hosts, Docker will auto-create `./database` on first run and Maria
 sudo chown -R 999:999 database
 ```
 
-If you hit `InnoDB: Operating system error number 13` at startup, you have hit this case - see [Switching to Named Volume](DATABASE.md#switching-to-named-volume) in the database docs for an alternative that sidesteps bind-mount permission issues entirely.
+If you hit `InnoDB: Operating system error number 13` at startup, you have hit this case - see [Migrating from Bind Mount to Named Volume](DATABASE.md#migrating-from-bind-mount-to-named-volume) in the database docs for an alternative that sidesteps bind-mount permission issues entirely.
 
 #### 5. Set Permissions
 

docs/INSTALLATION.md

@@ -33,7 +33,7 @@ Choose your preferred installation method
      - `--pull-latest`: Pull latest code from Github and latest image from DockerHub
      - `--debug`: Set log level to debug
 
-   This automatically creates a `.env` file from the included `.env.example` and starts both the Youtarr application and MariaDB database containers.
+   This automatically creates a `.env` file from the included `.env.example` and starts both the Youtarr application and MariaDB database containers. On a fresh install, `./start.sh` uses Docker named-volume storage for MariaDB. If an existing `./database/` MariaDB directory is present, it preserves that bind-mounted database and prints a migration warning.
 
 3. **Access the web interface**:
    Open your browser and navigate to `http://localhost:3087`
@@ -81,11 +81,11 @@ If you prefer to use standard `docker compose up` commands:
    docker compose up -d
    ```
 
-   > **ARM Users (Apple Silicon, Raspberry Pi)**: Use the ARM override to avoid MariaDB volume issues:
+   > **Docker Desktop/ARM/NAS users**: For a fresh install, use the named-volume database override to avoid MariaDB bind-mount issues on virtualized filesystems:
    > ```bash
    > docker compose -f docker-compose.yml -f docker-compose.arm.yml up -d
    > ```
-   > See [Troubleshooting](TROUBLESHOOTING.md#apple-silicon--arm-incorrect-information-in-file-errors) for details.
+   > If you already have data in `./database/`, use `./scripts/migrate-to-named-volume.sh` instead. See [Database Management](DATABASE.md#migrating-from-bind-mount-to-named-volume) and [Troubleshooting](TROUBLESHOOTING.md#docker-desktop--arm-incorrect-information-in-file-errors) for details.
 
 5. **Access the web interface**:
    - Navigate to `http://localhost:3087`
@@ -95,7 +95,7 @@ If you prefer to use standard `docker compose up` commands:
 
 > **Important**: Ensure the path you assign to `YOUTUBE_OUTPUT_DIR` already exists on the host and is writable before starting the stack. Otherwise Docker will create it as root-owned and the container may not be able to write downloads.
 
-This method is **functionally equivalent** to using the start.sh script, but gives you direct control over environment variables. It's the preferred approach for any Docker-native workflow.
+This method gives you direct control over environment variables and compose files, but it is not identical to `./start.sh`: plain `docker compose up -d` uses the legacy bind-mounted database unless you include or pin `docker-compose.arm.yml`.
 
 ### Method 3: Manual Setup Without Git (Advanced Users Only)