fix: nova workaround removal + CCZ truncation + update_yaml shallow-merge + nova_scopes probe + docs (0.13.116)

.claude-plugin/marketplace.json

@@ -6,13 +6,13 @@
     "url": "https://github.com/jjackson"
   },
   "metadata": {
-    "version": "0.13.115"
+    "version": "0.13.116"
   },
   "plugins": [
     {
       "name": "ace",
       "source": "./",
-      "version": "0.13.115",
+      "version": "0.13.116",
       "description": "AI Connect Engine — orchestrates the CRISPR-Connect lifecycle from idea through app building, Connect setup, LLO management, and closeout"
     }
   ]

.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "ace",
-  "version": "0.13.115",
+  "version": "0.13.116",
   "description": "AI Connect Engine — orchestrates the CRISPR-Connect lifecycle from idea through app building, Connect setup, LLO management, and closeout",
   "author": {
     "name": "Jonathan Jackson",

CHANGELOG.md

@@ -5,6 +5,44 @@ All notable changes to the ACE plugin will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and the plugin follows [semantic versioning](https://semver.org/spec/v2.0.0.html).
 
+## 0.13.116 — 2026-05-08
+
+**`commcare_download_ccz` now writes the CCZ to disk and returns a path. Inline `ccz_base64` removed (BREAKING for any caller that reads the field).**
+
+The MCP transport silently truncates large base64 payloads — a 29 KB CCZ came back ~2.5 KB short of the original, and `unzip` rejected the corrupted ZIP. Default `MAX_MCP_OUTPUT_TOKENS` is ~10K tokens; ~40 KB of base64 is ~20 K tokens, well over the cap. Workaround during `turmeric/20260508-1951` was to call `CommCareBackend.downloadCcz` directly via tsx.
+
+- `mcp/connect/backends/commcare.ts` — `downloadCcz()` now writes bytes to `${CLAUDE_PLUGIN_DATA}/ccz-cache/<id>-<status>-<sha8>.ccz`, returning `{status, size_bytes, ccz_path, ccz_sha256, connect_markers, projected_connect_state}`. The `ccz_base64` field is gone. CCZs > 25 MB still get written + return the path; only the in-memory inflate (`connect_markers` / `projected_connect_state`) is skipped.
+- `mcp/connect-server.ts` — tool description updated to lead with the new contract.
+- `skills/commcare-form-patch/SKILL.md`, `skills/app-release/SKILL.md` — patched to reference `ccz_path` instead of `ccz_base64`.
+- `test/mcp/connect/unit/commcare-download-ccz.test.ts` — added a unit test verifying file is written, path is returned, sha256 matches, and `ccz_base64` is absent.
+
+**Operator/caller action on update:** anything that decoded `ccz_base64` must now `fs.readFileSync(result.ccz_path)`. Cache files persist across calls — clean up `${CLAUDE_PLUGIN_DATA}/ccz-cache/` if disk usage matters.
+
+**Other improvements in this version:**
+
+- `mcp/google-drive-server.ts` `update_yaml_file` — added `mergeMode: 'shallow' | 'deep'` parameter (default `'shallow'`, backward-compat). The default top-level shallow merge wiped sibling keys when patching nested structures (`phases.commcare-setup.steps.X` would replace the entire `phases` block, wiping `phases.design-review`); pass `mergeMode: 'deep'` to recursively merge plain objects (arrays/primitives still replace). Hit in `turmeric/20260508-1951` requiring manual `phases.design-review` restoration.
+- `bin/ace-doctor` — added `nova_scopes` probe immediately after `nova_auth` to validate per-tool scopes (`nova.hq.read` for `get_hq_connection`, `nova.hq.write` for `upload_app_to_hq`). Pre-0.13.116 a scope-missing key passed `nova_auth` and only failed mid-Phase-2.
+- `skills/commcare-form-patch/SKILL.md` — clarified `assessment-removal` patch class strips any `commcareconnect`-namespaced wrapper (including `<module xmlns=...>`), not just `<assessment>`.
+- `skills/app-test-cases/SKILL.md` — clarified `mobile_resolve_selectors` is a static lookup; no AVD/APK required.
+- `agents/commcare-setup.md` — added "Recovery: orphan apps from architect-vs-PAT identity split" subsection (voidcraft-labs/nova-plugin#13).
+
+## 0.13.111 — 2026-05-09
+
+**Drop the Nova MCP user-scope-override workaround now that the Nova plugin reads `NOVA_API_KEY` natively (voidcraft-labs/nova-plugin#11).**
+
+The Nova MCP server has supported PAT auth since early May, but the plugin's MCP config never surfaced it — Claude Code defaulted to OAuth, and ACE worked around it by registering a user-scope MCP entry at the same URL with `claude mcp add nova ... --header "Authorization: Bearer $NOVA_API_KEY"`, relying on URL-signature dedup to suppress the plugin entry. The override was fragile: it silently fell off across CLI updates, with no doctor signal until a Phase 2 upload halted mid-run on a missing `upload_app_to_hq` tool (this hit `turmeric/20260508-1951` and is the immediate trigger for this version).
+
+The right fix is upstream — add `headers.Authorization = "Bearer ${NOVA_API_KEY}"` to the plugin's `.mcp.json`. That issue (voidcraft-labs/nova-plugin#11) is filed; this version assumes it lands and removes the ACE-side workaround:
+
+- `bin/ace-setup` — deleted the entire ~40-line "Nova MCP user-scope override" block. Replaced with a one-time idempotent `claude mcp remove nova --scope user` so existing installs clean up the now-obsolete entry.
+- `bin/ace-doctor` — updated the `nova_env` and `nova_auth` probe comments + remediation strings to reflect the native PAT path. The probes themselves are unchanged (still useful — `nova_env` confirms `NOVA_API_KEY` is in `.env`; `nova_auth` confirms the bearer is accepted by the server).
+- `agents/commcare-setup.md` § Subagent inheritance — re-explained the inheritance via plugin-side `${NOVA_API_KEY}` expansion instead of the user-scope override.
+- `playbook/integrations/nova-integration.md` — full rewrite. Removed the override architecture; documented the native PAT path including the **shell-export step** required because Claude Code reads `${NOVA_API_KEY}` from its own process env, not from `$CLAUDE_PLUGIN_DATA/.env`. Added nova-plugin#11 to the Resolved Blockers section.
+
+**Operator action on update:** export `NOVA_API_KEY` in your shell rc (one-time per machine — see playbook/integrations/nova-integration.md § Install + auth step 5), then restart Claude Code. The next session will use the plugin's native PAT path.
+
+If you're updating from a version that registered the user-scope override, `bin/ace-setup`'s one-time cleanup will remove it on the next `/ace:setup --force-env`.
+
 ## 0.13.66 — 2026-05-06
 
 **Fix #129: swap `googleapis` for per-API subpackages — node_modules drops 332 MB → 141 MB.**

VERSION

@@ -1 +1 @@
-0.13.115
+0.13.116

agents/commcare-setup.md

@@ -97,9 +97,26 @@ Do NOT dispatch the architect until `get_hq_connection` returns
 Apply the same gate at the start of any later subagent dispatch in
 this phase that calls Nova tools (e.g. coverage retries) — but the
 parent's auth state is what matters. Subagents inherit Nova's MCP
-connection because the user-scope override registers it once for the
-session; every subagent dispatch sees the same `get_hq_connection`
-result.
+connection because the plugin's MCP entry expands `${NOVA_API_KEY}`
+once per session start; every subagent dispatch sees the same
+`get_hq_connection` result.
+
+#### Recovery: orphan apps from architect-vs-PAT identity split
+
+If `upload_app_to_hq` returns `error_type: not_found` for an `app_id`
+that the architect just successfully reported building, the architect
+ran under a different Nova identity than the level-0 PAT — the app
+exists in Nova storage but is owned by an account whose PAT is not
+visible to the level-0 session, so the upload tool can't find it.
+(Filed upstream as voidcraft-labs/nova-plugin#13.)
+
+Recovery is to re-dispatch the architect from level 0. The fresh
+dispatch uses the post-fix MCP context, which expands `${NOVA_API_KEY}`
+to the level-0 PAT identity; the rebuild and the upload then run under
+the same identity. Cost: ~10–15 min to rebuild each app. Canonical
+instance: `turmeric/20260508-1951` Phase 2 — both Learn and Deliver
+apps were orphaned and had to be rebuilt before `upload_app_to_hq`
+could find them.
 
 ### Step 1: PDD to Apps (sequential)
 Invoke `pdd-to-learn-app`, then `pdd-to-deliver-app`.

bin/ace-doctor

@@ -961,12 +961,13 @@ if [ -n "$ENV_FILE" ]; then
 
   # ── Nova (CommCare app builder MCP) ──────────────────────────────────
   # Presence check only — live HTTP probe runs in [Auth liveness] below.
-  # Drift detection: if NOVA_API_KEY is missing or unresolved (still
-  # `op://...`), bin/ace-setup's user-scope MCP override will be skipped
-  # and /nova:* falls back to OAuth — broken under concurrent worktrees.
+  # The Nova plugin reads NOVA_API_KEY natively (post nova-plugin#11) and
+  # expands it into the MCP entry's Authorization: Bearer header. Without
+  # it the plugin falls back to interactive OAuth — broken under concurrent
+  # worktrees per voidcraft-labs/nova-plugin#9.
   NOVA_KEY_VAL="$(get_env NOVA_API_KEY)"
   if [ -z "$NOVA_KEY_VAL" ]; then
-    warn "nova_env: NOVA_API_KEY missing or unresolved" "mint at https://commcare.app/settings as the ACE Gmail identity, save to 1Password item \"ACE - Nova\" / field \`api_key\`, then /ace:setup --force-env"
+    warn "nova_env: NOVA_API_KEY missing or unresolved" "mint at https://commcare.app/settings as the ACE Gmail identity, save to 1Password item \"ACE - Nova\" / field \`api_key\`, then /ace:setup --force-env, then export NOVA_API_KEY in your shell rc so Claude Code's MCP header expansion picks it up"
   else
     pass "nova_env: NOVA_API_KEY present"
   fi
@@ -1187,10 +1188,10 @@ else
 fi
 
 # 6. nova — POST initialize JSON-RPC frame to https://mcp.commcare.app/mcp
-# with bearer auth. 200 = bearer accepted (the user-scope override
-# bin/ace-setup registered is alive). 401 = key invalid/revoked. The
-# Accept header includes text/event-stream because Nova returns SSE
-# frames for the response body even though the request is plain JSON.
+# with bearer auth. 200 = bearer accepted by Nova's PAT auth path.
+# 401 = key invalid/revoked. The Accept header includes text/event-stream
+# because Nova returns SSE frames for the response body even though the
+# request is plain JSON.
 if [ -n "$NOVA_KEY_VAL" ]; then
   NOVA_REACH_CODE="$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \
     -X POST \
@@ -1210,13 +1211,102 @@ if [ -n "$NOVA_KEY_VAL" ]; then
       warn "nova_auth: cannot reach https://mcp.commcare.app/mcp (network/DNS/timeout)" "check VPN/proxy; net_nova_mcp probe above should also flag this"
       ;;
     *)
-      warn "nova_auth: POST https://mcp.commcare.app/mcp → HTTP $NOVA_REACH_CODE" "investigate Nova MCP health; if persists, /ace:setup --force-env to re-register the user-scope override"
+      warn "nova_auth: POST https://mcp.commcare.app/mcp → HTTP $NOVA_REACH_CODE" "investigate Nova MCP health; if persists, rotate the key at https://commcare.app/settings and re-inject .env"
       ;;
   esac
 else
   warn "nova_auth: skipped — NOVA_API_KEY not set" "see nova_env warning above; mint at https://commcare.app/settings"
 fi
 
+# 6a. nova_scopes — validate the PAT actually has the per-tool scopes ACE
+# needs. `nova_auth` only confirms the bearer is accepted at the transport
+# layer; a key without `nova.hq.read` / `nova.hq.write` will pass nova_auth
+# and only fail mid-Phase-2 / app-deploy with a misleading "scope_missing".
+# Probe by calling `get_hq_connection` (the cheapest read tool that exercises
+# `nova.hq.read`) and parse the response — branches:
+#   - 2xx + configured:true + domain.name → pass (read scope + HQ key bound)
+#   - 2xx + configured:false             → warn (read scope OK, no HQ key in Nova)
+#   - 2xx + error_type:scope_missing     → fail (rotate / regrant scope)
+#   - other                              → warn (unexpected response)
+if [ -n "$NOVA_KEY_VAL" ]; then
+  NOVA_SCOPE_BODY="$(curl -sS --max-time 5 \
+    -X POST \
+    -H "Authorization: Bearer $NOVA_KEY_VAL" \
+    -H "Content-Type: application/json" \
+    -H "Accept: application/json, text/event-stream" \
+    -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"get_hq_connection","arguments":{}}}' \
+    "https://mcp.commcare.app/mcp" 2>/dev/null || echo "")"
+  NOVA_SCOPE_CODE="$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \
+    -X POST \
+    -H "Authorization: Bearer $NOVA_KEY_VAL" \
+    -H "Content-Type: application/json" \
+    -H "Accept: application/json, text/event-stream" \
+    -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"get_hq_connection","arguments":{}}}' \
+    "https://mcp.commcare.app/mcp" 2>/dev/null || echo "000")"
+  # The body may be SSE-wrapped (lines like "data: {...}"); strip the
+  # `data: ` prefix on the JSON-RPC frame line before parsing. python3 is
+  # ubiquitous on macOS + most Linux; jq isn't always present.
+  NOVA_SCOPE_PARSED="$(printf '%s' "$NOVA_SCOPE_BODY" | python3 -c '
+import sys, json, re
+raw = sys.stdin.read()
+# Pull JSON object from SSE-style "data: {...}" or plain JSON.
+m = re.search(r"data:\s*(\{.*\})", raw)
+payload = m.group(1) if m else raw.strip()
+try:
+    obj = json.loads(payload)
+except Exception:
+    print("PARSE_ERROR")
+    sys.exit(0)
+# JSON-RPC tools/call wraps the actual result in result.content[0].text
+# as a JSON string. Unwrap it.
+inner = None
+try:
+    inner_raw = obj.get("result", {}).get("content", [{}])[0].get("text", "")
+    inner = json.loads(inner_raw) if inner_raw else None
+except Exception:
+    inner = obj.get("result") or obj
+target = inner if isinstance(inner, dict) else (obj.get("result") if isinstance(obj.get("result"), dict) else obj)
+err_type = target.get("error_type") if isinstance(target, dict) else None
+required = target.get("required_scope") if isinstance(target, dict) else None
+configured = target.get("configured") if isinstance(target, dict) else None
+domain_name = None
+if isinstance(target, dict):
+    d = target.get("domain")
+    if isinstance(d, dict):
+        domain_name = d.get("name")
+print(f"err_type={err_type}\nrequired={required}\nconfigured={configured}\ndomain_name={domain_name}")
+' 2>/dev/null || echo "PARSE_ERROR")"
+  NOVA_ERR_TYPE="$(printf '%s' "$NOVA_SCOPE_PARSED" | sed -n 's/^err_type=//p')"
+  NOVA_REQUIRED_SCOPE="$(printf '%s' "$NOVA_SCOPE_PARSED" | sed -n 's/^required=//p')"
+  NOVA_CONFIGURED="$(printf '%s' "$NOVA_SCOPE_PARSED" | sed -n 's/^configured=//p')"
+  NOVA_DOMAIN_NAME="$(printf '%s' "$NOVA_SCOPE_PARSED" | sed -n 's/^domain_name=//p')"
+
+  case "$NOVA_SCOPE_CODE" in
+    2*)
+      if [ "$NOVA_ERR_TYPE" = "scope_missing" ]; then
+        REQ_TXT="${NOVA_REQUIRED_SCOPE:-unknown}"
+        fail "nova_scopes: NOVA_API_KEY missing scope $REQ_TXT" "edit at https://commcare.app/settings → API tokens → grant HQ Read + HQ Write"
+      elif [ "$NOVA_CONFIGURED" = "True" ] && [ -n "$NOVA_DOMAIN_NAME" ] && [ "$NOVA_DOMAIN_NAME" != "None" ]; then
+        pass "nova_scopes: HQ Read scope granted; bound to $NOVA_DOMAIN_NAME"
+      elif [ "$NOVA_CONFIGURED" = "False" ]; then
+        warn "nova_scopes: HQ Read granted but no HQ key bound in Nova settings" "paste an HQ API key at https://commcare.app/settings before /ace:run"
+      else
+        TRUNC="$(printf '%s' "$NOVA_SCOPE_BODY" | head -c 200)"
+        warn "nova_scopes: unexpected response $TRUNC" "investigate Nova MCP response shape"
+      fi
+      ;;
+    000)
+      warn "nova_scopes: cannot reach https://mcp.commcare.app/mcp (network/DNS/timeout)" "check VPN/proxy; see nova_auth above"
+      ;;
+    *)
+      TRUNC="$(printf '%s' "$NOVA_SCOPE_BODY" | head -c 200)"
+      warn "nova_scopes: HTTP $NOVA_SCOPE_CODE — $TRUNC" "investigate Nova MCP health; if persists, rotate the key"
+      ;;
+  esac
+else
+  warn "nova_scopes: skipped — NOVA_API_KEY not set" "see nova_env warning above; mint at https://commcare.app/settings"
+fi
+
 # 7. ace-mobile — local-only MCP (Maestro + adb + AVD + APK). It does not
 # authenticate against any remote service at runtime, so there is no
 # "live auth probe" distinct from the others. The auth-adjacent things

bin/ace-setup

@@ -406,46 +406,16 @@ $DIAG"
   fi
 fi
 
-# ---- Nova MCP user-scope override (API-key auth)
+# ---- Nova MCP one-time cleanup of legacy user-scope override
 #
-# The Nova plugin auto-registers an OAuth-mode MCP entry at
-# https://mcp.commcare.app/mcp. Concurrent ACE worktrees on one identity
-# trip a refresh-token cascade upstream of disk (better-auth/oauth-provider
-# deletes all (userId, clientId) refresh rows when a stale token is
-# presented). Adding a user-scope entry at the same URL with a bearer
-# header makes Claude Code's URL-signature dedup suppress the plugin entry
-# and route every Nova call through API-key auth instead. See
-# voidcraft-labs/nova-plugin#9 and `playbook/integrations/nova-integration.md`.
-#
-# Idempotent: removes any existing user-scope nova entry, re-adds with the
-# current key so rotations propagate. Skipped silently when NOVA_API_KEY is
-# missing (e.g. operator hasn't minted one yet) — doctor surfaces it.
-if [ -f "$ENV_OUT" ]; then
-  NOVA_KEY="$(grep -E '^NOVA_API_KEY=' "$ENV_OUT" | head -1 | sed -E 's/^NOVA_API_KEY=//' | sed -E 's/^"(.*)"$/\1/')"
-else
-  NOVA_KEY=""
-fi
-if [ -z "$NOVA_KEY" ] || [ "${NOVA_KEY#op://}" != "$NOVA_KEY" ]; then
-  warn "nova_mcp: NOVA_API_KEY not resolved in $ENV_OUT" \
-       "mint a key at https://commcare.app/settings as the ACE Gmail identity, save to 1Password item \"ACE - Nova\" field \`api_key\`, then re-run /ace:setup --force-env. Until then, /nova:* will fall back to OAuth and break under concurrent worktree use."
-elif ! command -v claude >/dev/null 2>&1; then
-  warn "nova_mcp: claude CLI not on PATH — cannot register user-scope override" \
-       "install Claude Code CLI, then re-run /ace:setup"
-else
-  # Remove any prior user-scope entry; ignore failure (entry may not exist).
+# Pre-2026-05-09 ACE registered a user-scope MCP entry at
+# https://mcp.commcare.app/mcp with a bearer header to route around
+# voidcraft-labs/nova-plugin#9. The Nova plugin now reads NOVA_API_KEY
+# natively (voidcraft-labs/nova-plugin#11), so the user-scope entry is
+# obsolete and could double-register the same URL. Idempotent: tries
+# to remove, ignores any "not found" error.
+if command -v claude >/dev/null 2>&1; then
   claude mcp remove nova --scope user >/dev/null 2>&1 || true
-  ADD_OUT="$(claude mcp add nova "https://mcp.commcare.app/mcp" \
-              --transport http \
-              --scope user \
-              --header "Authorization: Bearer $NOVA_KEY" 2>&1)"
-  ADD_RC=$?
-  if [ "$ADD_RC" -eq 0 ]; then
-    pass "nova_mcp: user-scope bearer override registered at https://mcp.commcare.app/mcp"
-  else
-    fail "nova_mcp: claude mcp add failed (exit $ADD_RC)" "see error below; usually means the claude CLI version doesn't support --header or --scope user yet
-$(printf '%s\n' "$ADD_OUT" | sed 's/^/    /')"
-    mark_fail
-  fi
 fi
 
 # ---- Auto-update hook (opt-in)

mcp/connect-server.ts

@@ -427,6 +427,7 @@ server.tool('commcare_release_build',
 );
 
 server.tool('commcare_download_ccz',
+  'Writes the CCZ to `$CLAUDE_PLUGIN_DATA/ccz-cache/` and returns `ccz_path` for the caller to open. Inline `ccz_base64` was removed in 0.13.116 because the MCP transport silently truncated large base64 payloads — a 29 KB CCZ came back missing 2.5 KB of trailing bytes. Returns `{status, size_bytes, ccz_path, ccz_sha256, connect_markers, projected_connect_state}`. CCZs > 25 MB still get written to disk + return the path, but skip the in-memory inflate (so `connect_markers` and `projected_connect_state` are absent). Use `connect_markers` and `projected_connect_state` for cheap server-side validation; read the file at `ccz_path` only when you need the raw bytes.',
   {
     domain: z.string(),
     app_id: z.string(),