ClawProbe Testing Feedback Report

[shunwah]

Test Environment: macOS 15.3, Node.js v22.14.0, latest OpenClaw (main branch)
Test Date: 2026-03-24
Purpose: Verify ClawProbe installation, feature completeness, output accuracy, and compatibility with OpenClaw.

---

1. Installation and Startup Testing

Test Commands:

npm install -g clawprobe
clawprobe start
clawprobe status

Observations:

• Installation succeeded, but on first run of clawprobe start, the console output showed:

  ✔ Daemon started (pid 12345)
  ✔ Watching ~/.openclaw
  

  However, clawprobe status displayed:

  ⚠ No active session found
  

  In reality, OpenClaw had an active session (agent:main).
  Inspecting ~/.clawprobe/clawprobe.db revealed that the daemon did not correctly resolve the OpenClaw session file paths.

Issues:

• The daemon’s auto-detection of the OpenClaw path may be flawed, especially when OpenClaw is configured to use a non‑default path (e.g., via the OPENCLAW_HOME environment variable).
• Even with the default path, permission issues could prevent reading files under ~/.openclaw/agent.

Suggestions:

• Add a --openclaw-dir flag to manually specify the OpenClaw root directory.
• Print the detected path on startup to aid troubleshooting.

---

2. Basic Command Testing

clawprobe status (when no active session)

Command:

clawprobe status

Output:

⚠ No active session found

Issue:

• The error message is ambiguous; users cannot tell if the daemon isn’t running or if OpenClaw isn’t running.
• Provide more detailed diagnostic info, e.g., list recent sessions or suggest clawprobe session --list to view history.

clawprobe status (when an active session exists)

Command:

clawprobe status

Output (normal):

📊  Agent Status  (active session)
──────────────────────────────────────────────────
  Agent:     main
  Session:   agent:main:workspace:direct:xxx ●
  Model:     moonshot/kimi-k2.5
  Active:    Today 16:41   Compacts: 2

  Context:   87.3K / 200.0K tokens  ███████░░░  44%
  Tokens:    72.4K in / 5.2K out

  Today:     $0.12  → clawprobe cost for full breakdown

  🟡  Context window at 44% capacity
       → Consider starting a fresh session or manually compacting now

Issue:

• “Compacts: 2” may refer to the number of compactions, but it’s unclear whether it’s for the current session or globally.
• Suggestion: change to “Compactions (this session): 2” for clarity.

---

3. Real‑time Monitoring Commands

clawprobe top

Command:

clawprobe top

Observations:

• Default 2‑second refresh rate works; terminal usage is reasonable.
• Occasional flickering during refresh, likely due to screen clearing and redrawing.
• When the session ends or the daemon is interrupted, top continues trying and eventually errors:

  Error: No active session to monitor
  

  Instead of staying in the refresh loop, it should exit gracefully.

Suggestions:

• Detect daemon state and exit if the session disappears, with a clear message.
• Add an option like --exit-on-session-end to control behavior.

clawprobe live

Command:

clawprobe live

Observations:

• Correctly displays tool calls, results, and durations.
• However, when OpenClaw output lacks a stopReason field, error information for tool calls may be missing.
• During testing, an entry showed └─ error 1.2s exit 1 but the actual tool execution succeeded – likely a parsing issue.

Issues:

• Some tool‑call log formats don’t match what ClawProbe expects, leading to incorrect result statuses.
• --density compact hides successful result lines, but sometimes users still need a brief preview of the result.

Suggestions:

• Add a --density minimal that shows only the tool name and duration, without result preview.
• Improve parsing robustness by defaulting missing fields.

---

4. Cost and Session Analysis

clawprobe cost

Command:

clawprobe cost --week

Output (example):

💰  Weekly Cost  2026-03-18 – 2026-03-24
──────────────────────────────────────────────────
  Total:     $0.67
  Daily avg: $0.096
  Month est: $2.87

  2026-03-19  ████████████░░  $0.15
  ...

Issue:

• For models not built‑in (e.g., a self‑hosted LLaMA), cost errors:

  Error: Unknown model "llama3-70b". Add to ~/.clawprobe/config.json customPrices.
  

  The documentation doesn’t describe the configuration format, leaving users unsure how to add it.
• Even after configuring, the daemon needs to be restarted – this isn’t mentioned.

Suggestions:

• Provide a clawprobe config add-price subcommand to simplify configuration.
• Include a configuration template in the error message.

clawprobe session

Command:

clawprobe session

Observations:

• For incomplete tasks, the “→” arrow is used, but it doesn’t distinguish “in progress” from “pending”.
• If the session has no todo list or sub‑agents, these sections are empty but still take up screen space.

Suggestions:

• Add a --compact option to hide empty sections, keeping the output concise.

---

5. Context and Compaction Analysis

clawprobe context

Command:

clawprobe context

Output:

🔍  Context Window  agent: main
──────────────────────────────────────────────────
  Used:    87.3K / 200.0K tokens  ███████░░░  44%

  Workspace overhead:  ~4.2K tokens  (7 injected files)
  Conversation est:    ~83.1K tokens  (messages + system prompt + tools)

  ⚠ TOOLS.md: 31% truncated — model never sees this content
    Increase bootstrapMaxChars in openclaw.json to fix this

  Remaining:  112.7K tokens (56%)

Issue:

• “Workspace overhead” count may be inaccurate because the number of injected files is dynamic and files may be truncated.
• Truncation detection relies on file sizes, but actual truncation happens inside OpenClaw; ClawProbe can only infer it by checking recent messages – false positives are possible.

Suggestions:

• Explain the truncation detection mechanism (e.g., checking if TOOLS.md is fully included in recent messages).
• Add a --verbose flag to show the list of truncated files.

clawprobe compacts

Command:

clawprobe compacts

Output:

📦  Compact Events  last 5
──────────────────────────────────────────────────

  #3  Today 16:22  [agent:main…]  3 messages

    👤  "Can you add retry logic to the upload handler?"
    🤖  "Done — added exponential backoff with 3 retries. The key change is in…"

    → Archive: clawprobe compacts --save 3
    → Archive to custom path: clawprobe compacts --save 3 --file notes/compact-log.md

Issue:

• --save only saves the message text, not the full context at the time of compaction (e.g., tool call details). This may not be sufficient for full restoration.
• The default saved filename is compact-<id>.md, which lacks a timestamp and can be overwritten.

Suggestions:

• Provide --save-json to save the complete structured data for programmatic recovery.
• Include a timestamp in the default filename (e.g., compact-2026-03-24T16-22-00.md).

---

6. Smart Suggestions and JSON Output

clawprobe suggest

Command:

clawprobe suggest

Output (example):

Suggestions:
  [warning] context-headroom: Context window at 44% capacity
  [warning] tools-truncation: TOOLS.md 31% truncated

Issue:

• The high-compact-freq rule didn’t trigger even though the session compacted twice within 30 minutes. Possibly the threshold calculation is off.
• Dismissing a rule with --dismiss does not persist after daemon restart; the rule reappears because it’s not written to the config file.

Suggestions:

• When dismissing, also update ~/.clawprobe/config.json under rules.disabled.
• Add suggest --list-rules to show all rules and their trigger conditions.

JSON Output Testing

Command:

clawprobe status --json

Output (with session):

{
  "ok": true,
  "agent": "main",
  "daemonRunning": true,
  "sessionKey": "agent:main:workspace:direct:xxx",
  "model": "moonshot/kimi-k2.5",
  "sessionTokens": 87340,
  "windowSize": 200000,
  "utilizationPct": 44,
  "todayUsd": 0.12,
  "suggestions": [
    {
      "severity": "warning",
      "ruleId": "context-headroom",
      "title": "Context window at 44% capacity",
      "detail": "...",
      "action": "..."
    }
  ]
}

Issue:

• When there’s no active session, the output is:

  { "ok": false, "error": "no_active_session" }

  but the exit code is 0, making it hard for scripts to detect errors. A non‑zero exit code should be used.

Suggestions:

• Return a non‑zero exit code (e.g., 1) for all error cases.
• Document the JSON error structure and exit codes clearly.

---

7. Configuration and Extensibility

Test: Created ~/.clawprobe/config.json:

{
  "timezone": "Asia/Shanghai",
  "openclaw": {
    "dir": "/custom/path/.openclaw"
  }
}

Result:

• After daemon restart, the default path was still used; configuration didn’t take effect.

Issue:

• Configuration loading logic may be buggy, or openclaw.dir is being ignored.
• Documentation doesn’t specify precedence (command‑line arguments > environment variables > config file > auto‑detection).

Suggestions:

• Fix config loading, and add a --debug flag to print the final resolved configuration.
• Provide clawprobe config show to display the current effective configuration.

---

8. Edge Cases and Compatibility

• Windows WSL2: Not tested, though documentation claims support. Add WSL2 test cases to ensure file path resolution works correctly.
• Node.js version: Requires ≥22 but no check is performed. If an older version is used, a clear error message should be shown.
• OpenClaw version changes: ClawProbe relies on parsing OpenClaw’s JSONL files; if OpenClaw updates its log format, functionality may break. Add version compatibility detection.

---

Summary

ClawProbe is a very useful observability tool for OpenClaw, and its core features already meet daily monitoring needs. However, there are several areas for improvement:

1. Auto‑detection robustness: OpenClaw path and session recognition need to be more reliable.
2. Error handling: When no session exists, provide a friendly message and use proper exit codes.
3. Configuration flexibility: Custom model pricing and path configuration need to be fixed and enhanced.
4. Output formatting: Terminal output could be more compact; JSON output should have correct exit codes.
5. Documentation: Add command examples, configuration templates, and troubleshooting guides.

It is recommended to prioritize fixing session detection during daemon startup, as that is the first hurdle users encounter. Additionally, improving the usability of --json mode will facilitate agent integration.

---

Record of Test Commands

# Installation
npm install -g clawprobe

# Start daemon
clawprobe start

# Status checks
clawprobe status
clawprobe status --json

# Real‑time monitoring
clawprobe top
clawprobe top --interval 3
clawprobe live
clawprobe live --history --density compact

# Cost analysis
clawprobe cost --week
clawprobe cost --json

# Session analysis
clawprobe session
clawprobe session --list
clawprobe session --no-todos --no-turns

# Context & compaction
clawprobe context
clawprobe compacts
clawprobe compacts --save 3 --file test.md

# Suggestions
clawprobe suggest
clawprobe suggest --dismiss tools-truncation --json

# Configuration
cat ~/.clawprobe/config.json
clawprobe config show   # expected, but not available