fix(cli): harden extension registration and discovery workflows (#2499)

* chore: update community catalog with latest extension versions

- Update memory-md from 0.7.9 to 0.8.0
- Update architecture-guard from 1.6.7 to 1.8.0

* fix(cli): harden extension registration with project-level tracking in extensions.yml

* test(cli): add comprehensive unit tests for extension registration logic

* chore: remove out-of-scope catalog changes

* refactor: address PR feedback for extension registration hardening

* fix: harden extension registration defensive logic and add comprehensive unregister_hooks tests

- Add dict guard to register_hooks() to handle corrupted extensions.yml (non-dict root)
- Add 5 comprehensive tests for unregister_hooks() workflow:
  * Full workflow with hooks + installed list removal
  * Resilience when config has no 'hooks' key
  * Corrupted YAML handling
  * Multiple extension scenarios
  * All 11 tests passing

* fix: sanitize installed to strings, guard unregister_hooks dict, handle null hook values

- register_extension(): filter non-string entries from installed before sort
- register_hooks(): normalize hooks to {} when missing or not a dict
- unregister_hooks(): add isinstance(config, dict) guard before key checks
- unregister_hooks(): coerce null/scalar hook lists to [] before iteration
- tests: add 3 regression tests for no-hooks manifest, mixed-type installed, null hook values
- All 14 tests passing

* fix(cli): persist sanitization results and harden hook registration

* Harden extension registration to always persist sanitization results

* Hardening extension registration: support mapping entries, improve persistence, and fix update rollback

* fix(cli): harden extension update and unregistration workflows

* fix(cli): move update sentinels outside try block to prevent NameError on rollback

* fix(cli): sanitize hook event lists in register_hooks to prevent crashes

* fix(cli): deduplicate hook entries and harden rollback hooks-restore guards

* test(cli): add regression tests for extension update and rollback hardening

* fix(cli): deduplicate installed list by id in register_extension

* fix(cli): consolidate and harden extension update rollback logic

* fix(cli): initialize backup_registry_entry before try block to prevent UnboundLocalError on rollback

* fix(tests): return Path from download_extension mock and add Path import

* fix(cli): normalize get_project_config() return to dict; deduplicate in unregister_extension()

* fix(cli): normalize hooks/installed/settings in get_project_config(); use tmp_path-scoped zip in tests

* fix(cli): set modified=True on hook coercion in rollback; sanitize hook event values in get_project_config(); harden test assertions

* fix(cli): filter non-dict hook entries in get_project_config(); remove dead MISSING sentinel

* fix(cli): gate extensions.yml rollback on backup_hooks is not None; update stale comment

* fix(cli): move _AgentReg import outside try block; assert result.exception is None in tests

* fix(extensions): consistent key order in default config; deep-copy backup_installed

* test: fix misleading comment; assert exit_code==1 in rollback test

* test: clean up duplicate imports in hardening tests

* refactor(extensions): extract _sanitize_installed_list helper; strengthen hook unregister assertion

* fix(extensions): validate extension IDs in _sanitize_installed_list; clarify test comment

Author: DyanGalih

src/specify_cli/__init__.py

@@ -4295,6 +4295,10 @@ def extension_update(
         failed_updates = []
         registrar = CommandRegistrar()
         hook_executor = HookExecutor(project_root)
+        from .agents import CommandRegistrar as _AgentReg  # used in backup and rollback paths
+
+        # UNSET sentinel: backup not yet captured (exception before backup step)
+        UNSET = object()
 
         for update in updates_available:
             extension_id = update["id"]
@@ -4308,8 +4312,9 @@ def extension_update(
             backup_config_dir = backup_base / "config"
 
             # Store backup state
-            backup_registry_entry = None
-            backup_hooks = None  # None means no hooks key in config; {} means hooks key existed
+            backup_registry_entry = None  # None means registry entry not yet captured
+            backup_installed = UNSET  # Original installed list from extensions.yml
+            backup_hooks = None  # None means backup step 4 not yet reached; {} or {...} means backup was captured
             backed_up_command_files = {}
 
             try:
@@ -4334,8 +4339,7 @@ def extension_update(
                         shutil.copy2(cfg_file, backup_config_dir / cfg_file.name)
 
                 # 3. Backup command files for all agents
-                from .agents import CommandRegistrar as _AgentReg
-                registered_commands = backup_registry_entry.get("registered_commands", {})
+                registered_commands = backup_registry_entry.get("registered_commands", {}) if isinstance(backup_registry_entry, dict) else {}
                 for agent_name, cmd_names in registered_commands.items():
                     if agent_name not in registrar.AGENT_CONFIGS:
                         continue
@@ -4360,14 +4364,20 @@ def extension_update(
                                 shutil.copy2(prompt_file, backup_prompt_path)
                                 backed_up_command_files[str(prompt_file)] = str(backup_prompt_path)
 
-                # 4. Backup hooks from extensions.yml
-                # Use backup_hooks=None to indicate config had no "hooks" key (don't create on restore)
-                # Use backup_hooks={} to indicate config had "hooks" key with no hooks for this extension
+                # 4. Backup hooks and installed list from extensions.yml
+                # get_project_config() always normalizes installed->[] and hooks->{},
+                # so no sentinel is needed to distinguish key-absent from key-empty.
                 config = hook_executor.get_project_config()
-                if "hooks" in config:
-                    backup_hooks = {}  # Config has hooks key - preserve this fact
-                    for hook_name, hook_list in config["hooks"].items():
-                        ext_hooks = [h for h in hook_list if h.get("extension") == extension_id]
+                if isinstance(config, dict):
+                    import copy
+                    # Deep-copy so nested mapping entries (e.g. version-pin dicts)
+                    # are not affected by in-place mutations during the update.
+                    backup_installed = copy.deepcopy(config.get("installed", []))
+                    backup_hooks = {}
+                    for hook_name, hook_list in config.get("hooks", {}).items():
+                        if not isinstance(hook_list, list):
+                            continue
+                        ext_hooks = [h for h in hook_list if isinstance(h, dict) and h.get("extension") == extension_id]
                         if ext_hooks:
                             backup_hooks[hook_name] = ext_hooks
 
@@ -4520,35 +4530,51 @@ def extension_update(
                             original_file.parent.mkdir(parents=True, exist_ok=True)
                             shutil.copy2(backup_file, original_file)
 
-                    # Restore hooks in extensions.yml
-                    # - backup_hooks=None means original config had no "hooks" key
-                    # - backup_hooks={} or {...} means config had hooks key
-                    config = hook_executor.get_project_config()
-                    if "hooks" in config:
+                    # Restore metadata in extensions.yml (hooks and installed list).
+                    # Only run if backup step 4 was reached (backup_hooks is not None);
+                    # otherwise we have no safe baseline to restore from and could corrupt
+                    # the config by removing pre-existing hooks.
+                    if backup_hooks is not None:
+                        config = hook_executor.get_project_config()
+                        if not isinstance(config, dict):
+                            config = {}
+
                         modified = False
 
-                        if backup_hooks is None:
-                            # Original config had no "hooks" key; remove it entirely
-                            del config["hooks"]
+                        # 1. Restore hooks in extensions.yml
+                        if not isinstance(config.get("hooks"), dict):
+                            config["hooks"] = {}
                             modified = True
-                        else:
-                            # Remove any hooks for this extension added by failed install
-                            for hook_name, hooks_list in config["hooks"].items():
-                                original_len = len(hooks_list)
-                                config["hooks"][hook_name] = [
-                                    h for h in hooks_list
-                                    if h.get("extension") != extension_id
-                                ]
-                                if len(config["hooks"][hook_name]) != original_len:
-                                    modified = True
-
-                            # Add back the backed up hooks if any
-                            if backup_hooks:
-                                for hook_name, hooks in backup_hooks.items():
-                                    if hook_name not in config["hooks"]:
-                                        config["hooks"][hook_name] = []
-                                    config["hooks"][hook_name].extend(hooks)
-                                    modified = True
+
+                        # Remove any hooks for this extension added by the failed install
+                        for hook_name in list(config["hooks"].keys()):
+                            hooks_list = config["hooks"][hook_name]
+                            if not isinstance(hooks_list, list):
+                                config["hooks"][hook_name] = []
+                                modified = True
+                                continue
+
+                            original_len = len(hooks_list)
+                            config["hooks"][hook_name] = [
+                                h for h in hooks_list
+                                if isinstance(h, dict) and h.get("extension") != extension_id
+                            ]
+                            if len(config["hooks"][hook_name]) != original_len:
+                                modified = True
+
+                        # Add back the backed-up hooks
+                        if backup_hooks:
+                            for hook_name, hooks in backup_hooks.items():
+                                if not isinstance(config["hooks"].get(hook_name), list):
+                                    config["hooks"][hook_name] = []
+                                config["hooks"][hook_name].extend(hooks)
+                                modified = True
+
+                        # 2. Restore installed list in extensions.yml
+                        if backup_installed is not UNSET:
+                            if config.get("installed") != backup_installed:
+                                config["installed"] = backup_installed
+                                modified = True
 
                         if modified:
                             hook_executor.save_project_config(config)

src/specify_cli/extensions.py

@@ -1190,7 +1190,7 @@ def install_from_directory(
         # was used during project initialisation (feature parity).
         registered_skills = self._register_extension_skills(manifest, dest_dir)
 
-        # Register hooks
+        # Register hooks and update installed list in extensions.yml
         hook_executor = HookExecutor(self.project_root)
         hook_executor.register_hooks(manifest)
 
@@ -2481,7 +2481,32 @@ def get_project_config(self) -> Dict[str, Any]:
             }
 
         try:
-            return yaml.safe_load(self.config_file.read_text(encoding="utf-8")) or {}
+            result = yaml.safe_load(self.config_file.read_text(encoding="utf-8"))
+            # Coerce non-dict root (including None for an empty file) to the
+            # fully-normalized default so callers always get guaranteed fields.
+            if not isinstance(result, dict):
+                return {
+                    "installed": [],
+                    "settings": {"auto_execute_hooks": True},
+                    "hooks": {},
+                }
+            # Normalize nested fields so read-only callers like get_hooks_for_event()
+            # never see non-dict hooks or non-list installed (Feedback)
+            if not isinstance(result.get("hooks"), dict):
+                result["hooks"] = {}
+            if not isinstance(result.get("installed"), list):
+                result["installed"] = []
+            if not isinstance(result.get("settings"), dict):
+                result["settings"] = {"auto_execute_hooks": True}
+            # Sanitize hook event values: coerce non-list values to [] and filter
+            # non-dict items so get_hooks_for_event() can safely call .get() (Feedback)
+            for event_key in list(result["hooks"]):
+                event_val = result["hooks"][event_key]
+                if not isinstance(event_val, list):
+                    result["hooks"][event_key] = []
+                else:
+                    result["hooks"][event_key] = [h for h in event_val if isinstance(h, dict)]
+            return result
         except (yaml.YAMLError, OSError, UnicodeError):
             return {
                 "installed": [],
@@ -2501,25 +2526,141 @@ def save_project_config(self, config: Dict[str, Any]):
             encoding="utf-8",
         )
 
+    def register_extension(self, extension_id: str):
+        """Add extension to the installed list in project config.
+
+        Args:
+            extension_id: ID of extension to register
+        """
+        config = self.get_project_config()
+
+        # Ensure config is a dict (defensive)
+        if not isinstance(config, dict):
+            config = {}
+
+        raw_installed = config.get("installed")
+        sanitized = self._sanitize_installed_list(raw_installed, add_id=extension_id)
+
+        if sanitized != raw_installed:
+            config["installed"] = sanitized
+            self.save_project_config(config)
+
+    def unregister_extension(self, extension_id: str):
+        """Remove extension from the installed list in project config.
+
+        Args:
+            extension_id: ID of extension to unregister
+        """
+        config = self.get_project_config()
+
+        if not isinstance(config, dict):
+            config = {}
+
+        raw_installed = config.get("installed")
+        sanitized = self._sanitize_installed_list(raw_installed, remove_id=extension_id)
+
+        # Always persist if sanitized state differs from raw config (ensures normalization)
+        if sanitized != raw_installed:
+            config["installed"] = sanitized
+            self.save_project_config(config)
+
+    @staticmethod
+    def _sanitize_installed_list(
+        raw: object,
+        *,
+        add_id: str = "",
+        remove_id: str = "",
+    ) -> list:
+        """Normalize, deduplicate, and optionally add/remove an extension id.
+
+        Shared by register_extension() and unregister_extension() to prevent
+        the two paths from drifting.
+
+        Args:
+            raw: The raw value from config["installed"] (may be non-list).
+            add_id: If non-empty, ensure this id is present (plain-string fallback).
+            remove_id: If non-empty, remove this id from the list.
+
+        Returns:
+            A sanitized, deduplicated, alphabetically-sorted list.
+        """
+        _VALID_ID = re.compile(r'^[a-z0-9-]+$')
+
+        installed = raw if isinstance(raw, list) else []
+
+        # Keep only entries whose resolved id is a non-empty string matching
+        # the extension-id format (^[a-z0-9-]+$), same rule ExtensionManifest enforces.
+        def _valid_entry(x: object) -> bool:
+            if isinstance(x, str):
+                return bool(_VALID_ID.match(x.strip()))
+            if isinstance(x, dict):
+                eid = x.get("id")
+                return isinstance(eid, str) and bool(_VALID_ID.match(eid.strip()))
+            return False
+
+        valid = [x for x in installed if _valid_entry(x)]
+
+        # Deduplicate by id: prefer dict (richer metadata) over plain string
+        seen: dict = {}  # id -> entry (dict preferred over str)
+        for x in valid:
+            eid = x.strip() if isinstance(x, str) else x.get("id", "").strip()
+            if eid not in seen or isinstance(x, dict):
+                seen[eid] = x
+
+        # Validate add_id against the same regex before inserting
+        if add_id and _VALID_ID.match(add_id.strip()) and add_id not in seen:
+            seen[add_id] = add_id
+
+        if remove_id:
+            seen.pop(remove_id, None)
+
+        def _sort_key(x: object) -> str:
+            return x if isinstance(x, str) else x.get("id", "")  # type: ignore[return-value]
+
+        return sorted(seen.values(), key=_sort_key)
+
     def register_hooks(self, manifest: ExtensionManifest):
         """Register extension hooks in project config.
 
         Args:
             manifest: Extension manifest with hooks to register
         """
+        # Always ensure the extension is in the installed list
+        self.register_extension(manifest.id)
+
         if not hasattr(manifest, "hooks") or not manifest.hooks:
             return
 
         config = self.get_project_config()
 
-        # Ensure hooks dict exists
-        if "hooks" not in config:
+        # Ensure config is a dict (defensive)
+        changed = False
+        if not isinstance(config, dict):
+            config = {}
+            changed = True
+
+        # Ensure hooks dict exists and is a mapping
+        if "hooks" not in config or not isinstance(config["hooks"], dict):
             config["hooks"] = {}
+            changed = True
+        else:
+            # Sanitize existing hook lists to prevent crashes in downstream code (Feedback)
+            for h_name in list(config["hooks"].keys()):
+                h_list = config["hooks"][h_name]
+                if not isinstance(h_list, list):
+                    config["hooks"][h_name] = []
+                    changed = True
+                else:
+                    sanitized_h_list = [h for h in h_list if isinstance(h, dict)]
+                    if len(sanitized_h_list) != len(h_list):
+                        config["hooks"][h_name] = sanitized_h_list
+                        changed = True
 
         # Register each hook
         for hook_name, hook_config in manifest.hooks.items():
-            if hook_name not in config["hooks"]:
+            if hook_name not in config["hooks"] or not isinstance(config["hooks"][hook_name], list):
                 config["hooks"][hook_name] = []
+                changed = True
 
             # Add hook entry
             hook_entry = {
@@ -2534,40 +2675,53 @@ def register_hooks(self, manifest: ExtensionManifest):
                 "condition": hook_config.get("condition"),
             }
 
-            # Check if already registered
-            existing = [
-                h
-                for h in config["hooks"][hook_name]
-                if h.get("extension") == manifest.id
+            # Deduplicate: remove all existing entries for this extension on this
+            # hook event, then append the single canonical entry. This prevents
+            # multiple hooks firing when hand-edited or older versions leave
+            # duplicate entries behind. (Feedback from review)
+            original_list = config["hooks"][hook_name]
+            deduped = [
+                h for h in original_list
+                if not (isinstance(h, dict) and h.get("extension") == manifest.id)
             ]
+            deduped.append(hook_entry)
+            if deduped != original_list:
+                config["hooks"][hook_name] = deduped
+                changed = True
 
-            if not existing:
-                config["hooks"][hook_name].append(hook_entry)
-            else:
-                # Update existing
-                for i, h in enumerate(config["hooks"][hook_name]):
-                    if h.get("extension") == manifest.id:
-                        config["hooks"][hook_name][i] = hook_entry
-
-        self.save_project_config(config)
+        if changed:
+            self.save_project_config(config)
 
     def unregister_hooks(self, extension_id: str):
         """Remove extension hooks from project config.
 
         Args:
             extension_id: ID of extension to unregister
         """
+        # Always remove from installed list (Feedback from review)
+        self.unregister_extension(extension_id)
+
         config = self.get_project_config()
 
-        if "hooks" not in config:
+        if not isinstance(config, dict):
+            config = {}
+            # We don't save yet, as there are no hooks to unregister, 
+            # but unregister_extension above might have already saved a normalized config.
+            return
+
+        if "hooks" not in config or not isinstance(config["hooks"], dict):
             return
 
         # Remove hooks for this extension
-        for hook_name in config["hooks"]:
+        for hook_name in list(config["hooks"].keys()):
+            hook_list = config["hooks"][hook_name]
+            if not isinstance(hook_list, list):
+                config["hooks"][hook_name] = []
+                continue
             config["hooks"][hook_name] = [
                 h
-                for h in config["hooks"][hook_name]
-                if h.get("extension") != extension_id
+                for h in hook_list
+                if isinstance(h, dict) and h.get("extension") != extension_id
             ]
 
         # Clean up empty hook arrays