Migrate Webserver to Flask and Jinja templates.

.vscode/settings.json

@@ -1,4 +1,5 @@
 {
+  "python-envs.defaultEnvManager": "ms-python.python:pyenv",
   "python.analysis.diagnosticSeverityOverrides": {
     "reportUndefinedVariable": "none"
   },

CLAUDE.md

@@ -4,6 +4,10 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 
 ## Development Commands
 
+**Running Python**
+Developers may have created virtual environments in directories like ".venv" or "venv". Make sure these virtual
+environments are activated before any of the python based tools below.
+
 **Development workflow uses Nox for task automation:**
 ```bash
 nox -s lint          # Code linting with Ruff (auto-fixes issues)
@@ -12,6 +16,7 @@ nox -s type_hints    # Type checking with MyPy
 nox -s smoke_tests   # Quick functionality validation
 nox -s unit_tests    # Full unit test suite
 nox -s babel         # I18n message extraction and compilation
+nox -s web_tests     # Testing the webserver, see below
 ```
 
 **Direct testing with pytest:**
@@ -32,7 +37,13 @@ pip install -r requirements_dev.txt
 If the .venv dir already exists, you can directly source it and run the app.
 
 
+Watch out for .venv directories containing virtual environments, that you need to activate first. 
+
 **Running the application:**
+
+First start the `cedar-detect-server` which is in `bin` (you need to use `-p 50551`, when invoking it).
+Use the correct architecture suffix for cedar-detect-server according to the platform you're running on. 
+
 Development setup has to have run and you should be in .venv virtual environment
 ```bash
 cd python/
@@ -54,13 +65,12 @@ python3.9 -m PiFinder.main -fh --camera debug --keyboard local -x
 - **GPS Process** - Location/time via GPSD or UBlox direct interface
 - **IMU Process** - Motion tracking with BNO055 sensor
 - **Integrator Process** - Combines solver + IMU data for real-time positioning
-- **Web Server Process** - Web interface and SkySafari telescope control integration
+- **Web Server Process** - Web interface and SkySafari integration as a telescope 
 - **Position Server Process** - External protocol support
 
 **State Management:**
 - `SharedStateObj` - Process-shared state using multiprocessing managers
 - `UIState` - UI-specific state management
-- Real-time synchronization of telescope position, GPS coordinates, and solved sky coordinates
 
 **Database Layer:**
 - SQLite backend (`astro_data/pifinder_objects.db`)
@@ -70,7 +80,7 @@ python3.9 -m PiFinder.main -fh --camera debug --keyboard local -x
 
 **Hardware Abstraction:**
 - Camera interface supporting IMX296 (global shutter), IMX290/462, HQ cameras
-- Display system for SSD1351 OLED and ST7789 LCD with red-light preservation
+- Display system for SSD1351 OLED and ST7789 LCD with night vision preservation using red channel only
 - Hardware keypad with PWM brightness control
 - GPS integration via GPSD or direct UBlox protocol
 - IMU sensor integration for motion detection and telescope orientation
@@ -109,6 +119,33 @@ Tests use pytest with custom markers for different test types. The smoke tests p
 - Menu structure and navigation logic
 - Multi-process logging and communication
 - Hardware interface abstractions
+- Website testing
+
+### Website testing setup
+
+**Testing Framework:** Uses Selenium WebDriver with Pytest for automated browser testing of the web interface
+
+**Infrastructure Requirements:**
+- Selenium Grid server at localhost:4444 (configurable via SELENIUM_GRID_URL environment variable). 
+  This server is started outside of the test code, for maximum flexibility
+- Chrome browser in headless mode for test execution
+- Tests automatically skip if Selenium Grid is unavailable
+
+**Test Coverage Areas:**
+- **Web Interface** (`test_web_interface.py`): Basic page loading, image display, status table elements (Mode, coordinates, software version)
+- **Location Management** (`test_web_locations.py`): Location CRUD operations, DMS coordinate entry, default switching, GPS integration via remote interface
+- **Network Configuration** (`test_web_network.py`): WiFi settings form validation, network management, restart flows, modal dialogs
+- **Remote Control** (`test_web_remote.py`): Authentication, virtual keypad, menu navigation, marking menus, API endpoint validation
+- **Equipment Management** (`test_web_equipment.py`): Telescope and eyepiece CRUD operations, active equipment selection, form validation
+- **Observation Tracking** (`test_web_observations.py`): Session list display, observation counters, detail pages, TSV export functionality
+
+**Authentication:** All protected pages use default password "solveit"
+
+**Responsive Testing:** Tests run on both desktop (1920x1080) and mobile (375x667) viewports
+
+**API Integration:** Extensive use of `/api/current-selection` endpoint to validate UI state changes and ensure web interface accurately reflects PiFinder's internal state
+
+**Helper Utilities:** Shared utilities in `web_test_utils.py` for login flows, key simulation, and state validation with recursive dictionary comparison
 
 ## Code Quality
 

docs/source/dev_guide.rst

@@ -322,7 +322,12 @@ The defined sessions are:
   That means extracts strings to translate and updates the `.po`-files in `python/locale/**`
   Then these are compiled into `.mo`-files. Unfortunately, this changes the `.mo`-files in any case,
   even if the there have been no changes to strings or their translation. As this will show up 
-  as changes to checked-in, this is not run by default. 
+  as changes to checked-in, this is not run by default.
+
+- web_tests -> Runs PyTest and executes all tests marked as WEB. Web tests use Selenium 
+  to automate browser testing of the PiFinder web interface. These tests require a 
+  running Selenium Grid server and a running PiFinder web server. You can test against a real PiFinder 
+  or a locally running instance. See the sections below for setup instructions. 
 
 
 CI/CD
@@ -334,6 +339,79 @@ your fork to run the existing automation to validate your code as you develop.
 
 If you need help, reach out via email or discord.  We are happy to help :-)
 
+Website Tests
+.............
+
+The PiFinder web interface can be tested using automated browser tests powered by Selenium.
+These tests verify functionality across different viewports (desktop and mobile) and ensure
+the web interface works correctly.
+
+The tests exercise the remote control features of PiFinder, changing **the state of the PiFinder** and
+therefore should **not be run** against a PiFinder you are actively using for observing.
+
+Running Website Tests locally
+_______________________________
+
+You can run ``pytest -m web --browser <browser> --local`` to run the website tests locally. 
+This will have Selenium launch a browser on your local machine and run the tests against a locally running instance of PiFinder. 
+The respective browsers need to be installed on your machine. Recognized browsers are ``chrome``, ``firefox`` and ``safari``. 
+
+Note that when running the tests on Safari, you need to enable "Allow Remote Automation" in the Develop menu of Safari. In addition Safari
+does not support the "headless" mode, so you will see the browser window when running the tests and you cannot use other windows while the tests are running.
+
+If you want to run the tests against a real PiFinder, set the ``PIFINDER_HOMEPAGE`` environment variable to the URL of your PiFinder instance or 
+pass the URL directly as a command line paramters with ``--url``. The PiFinder instance needs to be in the same WiFi as your machine, so that it is reachable via the network.
+
+Running Website Tests remotely
+________________________________
+
+Using Selenium Grid you can set up servers with different operating systems and different browsers to run your tests in parallel. 
+As the PiFinder is designed to have only one client accessing the web interface at a time, we recommend to run one Raspberry Pi per computer 
+instance and browser in the grid. You can install the software on bare bones Raspberry Pi and fake the non-existing hardware.  
+Or you can test against real PiFinders. 
+
+In the following we describe a simple setup with Selenium Grid running locally and running tests againt a locally running instance of PiFinder. 
+You can easily adapt this to more complex setups, e.g. by running the Selenium Grid server on a different machine or testing against a real PiFinder. 
+
+Running against a locally running instance at localhost:8080:
+
+.. code-block:: bash
+
+    cd ~/PiFinder/python
+    . .venv/bin/activate # Optionally active your virtual environment
+    export SELENIUM_GRID_URL=<your selenium grid url which ends in /wd/hub> # Optional, default is http://localhost:4444/wd/hub
+    nox -s web_tests
+
+If you want to test against a real PiFinder, set the ``PIFINDER_HOMEPAGE`` environment variable to the URL of your PiFinder instance:
+
+.. code-block:: bash
+
+    cd ~/PiFinder/python
+    . .venv/bin/activate # Optionally active your virtual environment
+    export SELENIUM_GRID_URL=<your selenium grid url which ends in /wd/hub> # Optional, default is http://localhost:4444/wd/hub
+    export PIFINDER_HOMEPAGE=http://pifinder.local # Change to the URL of your PiFinder, which needs to be in the same WiFi
+    nox -s web_tests
+
+If you run the tests with-out a working Selenium Grid instance, the tests will all be skipped. 
+You can also run individual tests with PyTest directly, use ``SELENIUM_GRID_URL=... PIFINDER_HOMEPAGE=... pytest tests/webstite/test_file.py``.
+
+Note that due to the tests depending on the response times of the PiFinder web server and the Selenium Grid server, there may be occasional timeouts or failures.
+If you encounter such issues, simply re-run the tests. We need to strike a balance between test speed and reliability, and this may require some tuning in the future.
+Note that the tests run approximately 10 minutes.
+
+Setting up Selenium Grid
+___________________________
+
+If you choose to run the website tests using a Selenium Grid server, the easiest way is to download the Selenum Grid server jar 
+from the selenium website, see https://www.selenium.dev/downloads/ and run it with Java:
+
+.. code-block:: bash
+
+    java -jar selenium-server-<version>.jar standalone
+
+If you run the Selenium Grid server this way, the browsers need to be installed on the same machine. 
+You'll have to consult the Selenium documentation for setting up a more complex grid with different machines and browsers.
+
 
 Running/Debugging from the command line
 ---------------------------------------
@@ -364,6 +442,9 @@ PiFinder:
 
     ps aux | grep PiFinder.main | awk '{system("kill -9  " $2)}'
 
+Running cedar-detect-server
+.............................
+
 You will need to start the ``cedar-detect`` process manually, if your development machine is not a PiFinder, 
 as it is started as a separate process on the PiFinder starting with v2.4.0. 
 You can do this by running the following command in another terminal window:
@@ -468,6 +549,23 @@ be retired because the remote server is always started.
 Troubleshooting
 ---------------
 
+Shared Memory location already exists
+.......................................
+
+It can happen that during development the shared memory location 
+``//cedar_detect_image`` is not cleaned up properly, e.g. because of a crash. 
+In this case, you can simply remove the shared memory location with the following command: 
+
+.. code-block:: bash
+
+    sudo rm /dev/shm/cedar_detect_image
+
+on Linux or with the following command on MacOS:
+
+.. code-block:: bash
+
+    python -c "import _posixshmem; _posixshmem.shm_unlink('//cedar_detect_image')"
+
 My app crashes
 ..............
 

python/PiFinder/config.py

@@ -9,6 +9,9 @@
 from pathlib import Path
 from PiFinder import utils, equipment, locations
 from typing import Any
+import logging
+
+logger = logging.getLogger("config")
 
 
 class Config:
@@ -34,7 +37,7 @@ def load_config(self):
             self._config_dict = {}
         else:
             with open(self.config_file_path, "r") as config_file:
-                print("Loading config from", self.config_file_path)
+                logger.info("Loading config from %s", self.config_file_path)
                 self._config_dict = json.load(config_file)
 
         # open default default_config

python/PiFinder/gps_ubx_parser.py

@@ -446,7 +446,7 @@ def _parse_nav_posecef(self, data: bytes) -> dict:
         ecefZ = int.from_bytes(data[12:16], "little", signed=True) / 100.0
         result = {}
         if ecefX == 0 or ecefY == 0 or ecefZ == 0:
-            logging.debug(
+            logger.debug(
                 f"nav_posecef zeroes: ecefX: {ecefX}, ecefY: {ecefY}, ecefZ: {ecefZ}"
             )
         else:

python/PiFinder/keyboard_interface.py

@@ -21,6 +21,7 @@ class KeyboardInterface:
     ALT_UP = 104
     ALT_DOWN = 105
     ALT_RIGHT = 106
+    ALT_SQUARE = 107
     ALT_0 = 110
     LNG_LEFT = 200
     LNG_UP = 201

python/PiFinder/keyboard_local.py

@@ -34,6 +34,22 @@ def __init__(self, q):
         except ModuleNotFoundError:
             logger.error("pyhotkey not supported on pi hardware")
             return
+        # pynput bug on macOS: KeyCode.__repr__ crashes with TypeError when vk is None.
+        # PyHotKey calls repr(key) to look up hotkeys, so patch it to be safe.
+        try:
+            from pynput.keyboard import KeyCode
+
+            _orig_repr = KeyCode.__repr__
+
+            def _safe_repr(self):
+                try:
+                    return _orig_repr(self)
+                except TypeError:
+                    return "<unknown>"
+
+            KeyCode.__repr__ = _safe_repr
+        except Exception:
+            pass
         try:
             self.q = q
             # Configure unmodified keys

python/PiFinder/main.py

@@ -296,13 +296,6 @@ def main(
     """
     global display_device, display_hardware
 
-    display_device = get_display(display_hardware)
-    init_keypad_pwm()
-    setup_dirs()
-
-    # Instantiate base keyboard class for keycode
-    keyboard_base = keyboard_interface.KeyboardInterface()
-
     # init queues
     console_queue: Queue = Queue()
     keyboard_queue: Queue = Queue()
@@ -326,6 +319,13 @@ def main(
     # Start log consolidation process first.
     log_helper.start()
 
+    display_device = get_display(display_hardware)
+    init_keypad_pwm()
+    setup_dirs()
+
+    # Instantiate base keyboard class for keycode
+    keyboard_base = keyboard_interface.KeyboardInterface()
+
     os_detail, platform, arch = utils.get_os_info()
     logger.info("PiFinder running on %s, %s, %s", os_detail, platform, arch)
 
@@ -580,7 +580,7 @@ def main(
                                     )  # Only if new error is smaller
                                 )
                             ):
-                                logger.info(
+                                logger.debug(
                                     f"Updating GPS location: new content: {gps_content}, old content: {location}"
                                 )
                                 location.lat = gps_content["lat"]