Fix #675: Secret auto-redaction in learning capture pipeline#856
Fix #675: Secret auto-redaction in learning capture pipeline#856AlexMikhalev wants to merge 2280 commits intomainfrom
Conversation
…lo -> terraphim -- Refs adf-fleet#40 Co-Authored-By: Terraphim AI <noreply@terraphim.cloud>
…728) from task/roc-v1-step-l-rollout-runbook into main
Updates the adf.rs provider registration from Kimi K2.5 to K2.6 so the ProviderHealthMap probe + KeywordRouter fallback target the current model. KG routing tables (planning_tier.md) and conf.d/digital-twins.toml already reference kimi-for-coding/k2p6; this aligns register_providers. Tests unchanged (test fixtures use synthetic k2p5 strings that remain valid — provider_key_for_model is generic and C1 allow-list checks the kimi-for-coding prefix, not the specific model string).
…tion to k2p6' (#734) from task/kimi-k2p6-provider into main
…rand - Patch rustls-webpki from 0.103.10 to 0.103.12 (fixes name constraints bypass) - Add RUSTSEC-2026-0098/0099 to deny.toml (serenity/discord-feature-only) - Replace rand with WASM-compatible fastrand in terraphim_multi_agent and terraphim_kg_agents - Remove two direct rand dependencies from workspace crates Refs #630
…on (#818) * feat(cli): add evaluate subcommand for automata ground-truth evaluation Wire the existing evaluate() function to a CLI subcommand in terraphim_cli. Changes: - Add Evaluate command with --ground-truth and --thesaurus flags - Add handle_evaluate() function using terraphim_automata::evaluate() - Add 4 integration tests for evaluate command - Wire Evaluate match arm in command dispatcher The core evaluation logic was already implemented in terraphim_automata::evaluation (~613 lines, 13 unit tests). This adds CLI integration for automation use. Example usage: terraphim-cli evaluate --ground-truth gt.json --thesaurus th.json Part of: Gitea #576 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(clippy): replace sort_by with sort_by_key for Rust 1.95 compatibility Rust 1.95 promotes clippy::unnecessary_sort_by to hard error under -D warnings. Convert all sort_by calls to sort_by_key across 3 crates: - terraphim-markdown-parser: 1 change (descending sort with Reverse) - terraphim_router: 1 change (descending sort with Reverse) - terraphim-session-analyzer: 13 changes (ascending + descending) Line 548 in reporter.rs retains sort_by with #[allow] due to fallible string parsing in the key function. Refs #576 * fix(clippy): fix remaining sort_by in session-analyzer main.rs and submodules Missed in previous commit: session-analyzer has duplicated logic in main.rs (binary target) and submodules (kg/search, patterns/loader) that also use sort_by. Convert to sort_by_key where possible, add #[allow] for float comparisons using partial_cmp. Refs #576 * fix(clippy): workspace-wide sort_by -> sort_by_key for Rust 1.95 compatibility Convert all remaining sort_by calls across 40 files to either sort_by_key or #[allow(clippy::unnecessary_sort_by)] for cases with non-Copy types, multi-line closures, or partial_cmp on floats. Covers: terraphim_agent, terraphim_automata, terraphim_orchestrator, terraphim_service, terraphim_persistence, terraphim_update, terraphim_usage, terraphim_sessions, terraphim_cli, terraphim_mcp_server, terraphim_types, terraphim_symphony, terraphim_tinyclaw, terraphim_multi_agent, terraphim_agent_evolution, terraphim_agent_registry, terraphim_goal_alignment Refs #576 * fix(clippy): fix Rust 1.95 lints in task_decomposition and rolegraph examples - Remove unnecessary .into_iter() in extend() call (useless_conversion lint) - Collapse if guards into match arms (collapsible_match lint) - Allow explicit_counter_loop in rolegraph examples Refs #576 * fix(clippy): allow collapsible_match lint in middleware and agent_evolution Rust 1.95 clippy promotes collapsible_match to hard error under -D warnings. Add #![allow] at file level for ripgrep.rs, orchestrator_workers.rs, and parallelization.rs where collapsing the match arms would reduce readability. Refs #576 * fix(clippy): allow collapsible_match in goal_alignment/goals.rs Refs #576 * fix(ci): pin Rust toolchain to 1.94.0 to match local dev dtolnay/rust-toolchain@stable installs latest (1.95.0) which has new clippy lints (collapsible_match, unnecessary_sort_by, useless_conversion) not present in 1.94. Pin all ci-pr.yml jobs to 1.94.0 and update rust-toolchain.toml accordingly. Refs #576 --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Cherry-pick from local main (0115f06f). Refs #612 Co-authored-by: Test User <test@test.com>
Pre-build at script line 98 ran cargo build --workspace --all-targets without --features zlob. fff-search build.rs panics under CI when zlob isn't enabled (intentional gate). Clippy step at line 112 already had the flag; pre-build needed it too. Unblocks lint-and-format CI for PR #818 and any future PR. Refs #818 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
openai/* via the OpenAI Plus/Pro/Team plan is subscription-gated, not pay-per-use. PR #631 (the P0 C1 fix) removed openai from the allow-list under the mistaken belief that all openai/* was pay-per-use; this commit restores it. Session-token exhaustion (the temporary state that motivated #631 in the first place) is already handled correctly by the orchestrator's existing primitives: - ProviderHealthMap probes every probe_ttl_secs (1800s / 30 min) and marks exhausted providers unhealthy. Routing falls through to the next KG route (anthropic/opus -> kimi-for-coding/k2p6 -> openai/gpt-5.4 -> zai-coding-plan/glm-5). - CircuitBreaker opens on 5 consecutive failures, half-opens after cooldown, closes on successful probe. - ProviderBudgetTracker hour+day tumbling UTC windows; force_exhaust() on throttle classification pushes past cap so routing drops until the next window rolls. - error_signatures.rs classifies stderr as Throttle / Flake / Unknown per provider with configurable regex. Changes: - config.rs: add "openai" to ALLOWED_PROVIDER_PREFIXES. - config.rs tests: include openai/gpt-5.4 and openai/gpt-5.3-codex in test_is_allowed_provider_c1_allowed_prefixes. - provider_gate_tests.rs: rename probe_gate_rejects_openai_provider -> probe_gate_allows_openai_subscription_provider; invert assertion and document the session-token-tracking rationale. - docs/taxonomy/routing_scenarios/adf/planning_tier.md: retain anthropic/opus as primary route, add kimi-for-coding/k2p6 and openai/gpt-5.4 as alternate routes; remove the previously-present legacy openai/gpt-5.4 entry that had been stripped as a C1 violation. Tests: all 497 unit tests + 16 integration tests pass. Clippy + fmt clean.
…st + KG routing' (#737) from task/kg-planning-tier-k2p6 into main
Replace the in-memory DeviceStorage backend with MarkdownLearningStore for durable persistence of shared learnings across process restarts. Changes: - Expand markdown frontmatter to serialize full SharedLearning state - Add backwards-compatible parsing for old sparse frontmatter files - Replace DeviceStorage with MarkdownLearningStore in SharedLearningStore - Implement real load_all() that hydrates index from persisted markdown - Add startup deduplication by learning.id (canonical over shared copies) - Update StoreConfig to include MarkdownStoreConfig - Align default paths with ProjectDirs recommendation - Add integration tests for restart durability and deduplication - Fix unreachable match arm for SharedLearningSub::Inject Refs #6
Handle missing learnings root as an empty store on first run and make startup deduplication deterministic by preferring canonical agent copies over shared replicas when IDs collide. Refs #6
…ack loop Step 7: Knowledge Graph Integration - Add DocumentType::Learning variant (behind kg-integration feature) - Add learning_id field to Document (behind kg-integration feature) - Add index_learning_document() and get_learning_documents() to RoleGraph - Create learning_indexer module in terraphim_middleware - Convert SharedLearnings to IndexedDocuments with keyword-to-node resolution - Index learnings in RoleGraph with trust-level filtering (L2 minimum by default) Step 8: Cross-Agent Feedback Loop - Create feedback_loop module in terraphim_middleware - Add record_graph_touch() to SharedLearningStore - Propagate learning quality to document ranks - Update learning metrics on graph query usage Other changes: - Move SharedLearning types to terraphim_types for cross-crate access - Add kg-integration and feedback-loop features to terraphim_middleware - Create integration tests for learning indexing and querying - Step 7 (CLI) cancelled due to circular dependency risk Refs: Steps 7-8 of Learning & KG Evolution plan
…llback (Fixes adf-fleet#44)
Problem
-------
When the orchestrator polled the repo-wide Gitea comments endpoint, every
PR comment came through with issue_number=0. That zero propagated:
webhook/poll -> AdfCommand::SpawnAgent { issue_number: 0, .. }
-> DispatchTask::MentionDriven { issue_number: 0, .. }
-> mention_def.gitea_issue = Some(0)
-> OutputPoster::post_agent_output_for_project(..., 0, ...)
-> POST /api/v1/repos/{owner}/{repo}/issues/0/comments
-> 500 {"message": "issue does not exist [id: 0, repo_id: 0, index: 0]"}
so the agent output was silently lost (it ran, exited 0, but never reached
Gitea). Observed live on 2026-04-21:
15:02:08 webhook received webhook event ... issue=28 comment_id=9880
15:02:16 orch dispatching mention-driven agent ... issue=0 comment_id=9880
15:04:16 orch agent exit classified ... exit_class=success
15:04:16 ERROR failed to post output for reviewer in project digital-twins:
Gitea post_comment error 500 on issue 0
Root cause
----------
`GET /api/v1/repos/{owner}/{repo}/issues/comments` returns TWO mutually-
exclusive URL fields per comment:
- issue_url -- set for issue comments, empty string for PR comments
- pull_request_url -- set for PR comments, empty string for issue comments
`impl From<RepoComment> for IssueComment` at gitea.rs:1132 only read
issue_url. For PR comments it fed an empty string to `rsplit('/').next()`,
`.parse()` failed, and `.unwrap_or(0)` kicked in.
Fix
---
- Deserialise `pull_request_url` alongside `issue_url` on `RepoComment`.
- Try issue_url first, fall back to pull_request_url; skip empty strings.
- PRs share the issue numeric namespace in Gitea, so the same `rsplit('/')`
trailing-segment trick works for both.
Tests
-----
- New: test_repo_comments_pr_comment_extracts_pr_number_from_pull_request_url
asserts both PR comments (via pull_request_url) and issue comments (via
issue_url) resolve to the correct number in a single response.
- Existing: test `assert_eq!(comments[0].issue_number, 0)` for "no URLs at all"
still passes; behaviour preserved when both URLs are missing/empty.
Verification
------------
- `cargo test -p terraphim_tracker --all-targets` — 40/40 PASS
- `cargo test -p terraphim_orchestrator --all-targets` — all suites PASS
- `cargo clippy -p terraphim_tracker --all-targets -- -D warnings` — clean
- `cargo fmt --check` — clean
… pull_request_url fallback (Fixes adf-fleet#44)' (#738) from task/fix-outputposter-issue-zero into main
…, tracker, output, quickwit - Refs terraphim/adf-fleet#4 Threads project context through the orchestrator runtime so one process can serve multiple projects: - DispatchTask variants carry project: String; Dispatcher adds per-project fairness (round-robin within same priority) - Restart cooldown + concurrency caps keyed on (project, agent); ConcurrencyConfig gains per_project caps - Agent spawn resolves agent.project -> Project and builds SpawnContext with working_dir + ADF_PROJECT_ID / ADF_WORKING_DIR / GITEA_OWNER / GITEA_REPO env - dual_mode runs one Tracker per project (RunningTrackers map), with __global__ fallback for legacy single-project mode - output_poster routes comments to the per-project Gitea repo - quickwit events tagged with project_id; index_id resolved per project - Legacy single-project configs keep working unchanged
Documents the full pipeline from DispatchContext through RoutingDecisionEngine (KG + keyword + static merge, C1/C3 filter, budget filter, scoring, telemetry adjustment) through spawn_with_fallback, model_args, the tokio subprocess, exit classification and OutputPoster write-back. Cross-references the code (routing.rs, kg_router.rs, provider_probe.rs, provider_budget.rs, error_signatures.rs, spawner) and the log lines so an operator can read a journal trace alongside it. Includes a worked example from a real security-sentinel dispatch. Also documents the adf-fleet#44 fix (PR #738): pull_request_url fallback for PR-comment issue_number extraction.
…nce' (#739) from task/docs-adf-model-selection into main
Before this change, an agent's own `gtr comment` / API calls inside its task shell used `$GITEA_TOKEN` from `source ~/.profile` — the shared root token — so the agent posted as `root` even though OutputPoster was using the agent's own token for the wrapped completion comment. After: build_spawn_context_for_agent resolves the per-agent token via OutputPoster::agent_token(project, name) (reading agent_tokens.json) and injects it as a `GITEA_TOKEN` env override, which beats the ~/.profile value. Every agent now posts to Gitea under its own login for both the wrapped completion comment AND any direct `gtr` call in its task script. Changes: - output_poster.rs: add ProjectTrackers.agent_tokens (HashMap<String, String>) parallel to agent_trackers; rewrite build_project_trackers to return both from one pass; add pub OutputPoster::agent_token getter. - lib.rs: extend build_spawn_context_for_agent to accept Option<&OutputPoster>; inject GITEA_TOKEN when agent_token returns Some. Update the two callsites in spawn_agent and handle_review_pr. Tests: - agent_token_returns_configured_value_when_tokens_file_loaded: loads a tempdir agent_tokens.json, asserts agent_token() returns the configured value; unknown agent/project returns None. - agent_token_returns_none_when_no_tokens_file: no file configured → always None; agent defaults to project root token. Verification: - cargo test -p terraphim_orchestrator --all-targets — all pass (including the 2 new unit tests + existing 497 suite tests). - cargo clippy -p terraphim_orchestrator --all-targets -- -D warnings — clean. - cargo fmt --check — clean.
…into spawn env' (#741) from task/inject-per-agent-gitea-token into main
…port from cherry-pick
… #1012 - Intercept args before clap parsing to apply typo correction, alias expansion, and case-insensitive matching - Add build_cli_forgiving_parser() with actual CLI subcommands - Add apply_forgiving_parsing() heuristic for subcommand detection - Print correction notifications to stderr - Add 3 integration tests: auto-correction, alias expansion, case-insensitive matching
…PI Refs #1011 - Add Robot variant to Command enum with subcommands: capabilities, schemas, examples - Add RobotFormat enum (json, table, minimal) for --format flag - Add handle_robot_command() and format_robot_output() helpers - Wire Robot command in main() before catch-all server/offline dispatch - Add unreachable! arms in run_offline_command and run_server_command to satisfy exhaustive match (Robot is handled in main) - Add 4 integration tests: capabilities JSON, schemas JSON, examples text, all formats honoured - Fix pre-existing compilation errors in terraphim_orchestrator (config clone + concurrency_permit field)
- Add [Unreleased] section with commits since 1.17.0 - Generate doc gap report: 771 undocumented items across 46 crates - Generate API reference snippets from documented public APIs Refs #1046
- Change ContentBlock::ToolResult from is_error: bool to exit_code: i32 with backward-compatible deserialization for old JSON containing is_error - Add --include-pinned flag to search subcommand in terraphim_agent - Add --pinned flag to graph subcommand as kg list --pinned equivalent - Fix learn query to call query_all_entries when semantic=false, matching spec - Add pinned_node_ids to RoleGraphResponseDto for server/agent parity - Add get_role_graph_pinned to TuiService for offline pinned retrieval
- Add spec gaps fix entry to CHANGELOG (Refs #1040) - Generate documentation gap report for workspace crates - Generate API reference snippets for key public items - Update existing Gitea issue #1046 with recurrence findings Part of: #1046
…Refs #451 - Add hook_manager field to TerraphimAgent with HookManager initialized in ::new() - Add execute_llm_with_hooks() helper that wraps LLM calls with run_pre_llm/run_post_llm - Wire hooks into 5 agent command handlers: generate, answer, analyze, create, review - Wire hooks into specialized agents: ChatAgent and SummarizationAgent - Add unit tests verifying hook invocation (pre_llm, post_llm, block, modify decisions) - All existing tests pass, clippy clean, fmt clean Implements requirement from spec-validator remediation #442 to wire LLM hooks that were previously defined but not connected.
- Update [Unreleased] with 20 entries from 2026-04-27 to 2026-04-28 - Scan 58 crates: ~2,647 pub items missing documentation - Generate gap report at .docs/reports/documentation-gap-report-20260428.md - Recurrence comment on Gitea issue #1046 Refs #1046
Clippy fixes (blocking ADF build-runner on all PRs): - Remove needless borrows in exit_codes_integration_test.rs - Replace useless format! with .to_string() in offline_mode_tests.rs - Remove unused LearningStore import in learning.rs Test helper fixes (pre-existing failures): - Update review_pr_config_with_spec_fanout to push pr-spec-validator into pr_dispatch_per_project['alpha'] (per-project config takes precedence over top-level pr_dispatch) - Same fix for review_pr_config_with_security_checklist_fanout and review_pr_config_with_test_fanout Refs #238
…d handlers' (#1056) from task/451-llm-hooks-wiring into main
… test helpers' (#1058) from task/fix-clippy-and-test-helpers into main
…handler runs Add server: bool to Command::Listen so clap accepts --server. Change identity from required String to Option<String> so clap parses listen --server without failing on missing --identity. Move identity validation into application logic after the --server check. - Listen handler checks local server flag instead of cli.server - Missing identity now exits with code 2 after --server check - All exit_codes_integration_test tests pass - cargo clippy --workspace -- -D warnings passes - cargo fmt --all -- --check passes Refs #1052
- Updated CHANGELOG.md with commits since v1.17.0 - Generated documentation gap report for 2026-04-29 - Generated API reference snippets for key crates - 923 missing docs identified across 11 crates - 53 rustdoc warnings catalogued Refs #1046
…tion Refs #1075 Phase A: Add LlmUsage/LlmResult types to terraphim_types, extend LlmClient trait with chat_completion_with_usage(), create GenAiClient adapter using the terraphim/rust-genai fork for automatic usage extraction from all providers. Phase B (partial): Add ExecutionRecord::from_llm_usage() to bridge terraphim_types::LlmUsage into terraphim_usage persistent storage. Also includes research and design docs for both #1075 (LLM cost tracking) and #871 (token budget CLI flags, deferred pending user direction).
Refs #1066 Corrects previous PASS assessment. Trigger index now live, procedure.rs un-gated, shared learning CLI present. Remaining: hook success capture, agent evolution unwired, sandbox guard tier, CorrectionEvent export, integration tests.
Phase 4+5 evidence: UBS scan (0 real critical), 128 tests passing, clippy clean, full workspace build, requirements traceability matrix with 12/12 implemented ACs verified, 8 ACs deferred to Phases C+D.
… 3 fixed 5 remaining' (#1076) from spec-validation-20260429 into main
- Add LLM cost tracking, spec validation, and --server flag entries - Generate doc gap report for 2026-04-29 Refs #1046
… update' (#1077) from sync/docs-update-20260429 into main
- Add documentation gap report entry to CHANGELOG - Generate doc gap report for 2026-04-29 Refs #1046
Phase B - Persistence: - Add configurable pricing module (pricing.rs) with 22 model defaults, TOML config support, and longest-prefix glob matching - Add terraphim_usage dep to terraphim_multi_agent - Add TokenUsageTracker::drain_records() for record extraction - Add TerraphimAgent::flush_usage() to persist records to UsageStore - Wire production usage recording in agent.rs after each LLM call (pricing lookup + TokenUsageRecord creation) Phase C - Provider completion: - Implement Claude provider via ccusage (7-day usage aggregation) - Implement OpenCode Go provider via sqlite3 CLI (cost aggregation) - Create CcusageProvider adapter wrapping terraphim_ccusage - Add terraphim_ccusage as dependency of terraphim_usage - Register all 5 providers in CLI handle_usage() - Kimi remains stub (undocumented API) 37 tests pass, clippy clean, fmt clean. Refs #1075
…line Refs #675 - Add redact(&str) -> Cow<str> with built-in patterns for AWS, GCP, Azure, API keys - Replace old [AWS_KEY_REDACTED] format with [REDACTED:<pattern-name>] - Add RedactionConfig with custom_patterns support via TOML - Add comprehensive tests for all built-in patterns + custom config + property test - Update capture.rs and hook.rs tests to match new format - Maintain backward compatibility via redact_secrets() alias
Security Audit SummaryPR #856 introduces six Claude Code hooks and associated test scripts for the learning capture pipeline, git safety guarding, npm-to-bun enforcement, and LLM pre/post validation. There are no Rust source changes, no Cargo dependency changes, and no Cargo.lock modifications. The entire security surface is confined to the bash hook layer. Risk: mediumVerdict: concernsFindings[MEDIUM] JSON injection via unescaped shell variables in git_safety_guard.sh (lines 50-58) The cat <<EOF
{
"hookSpecificOutput": {
"permissionDecisionReason": "BLOCKED ...\n\nReason: $REASON\n\nCommand: $COMMAND\n..."
}
}
EOFA command containing a double-quote (e.g., Remediation: use [MEDIUM] Secret redaction is a trust boundary delegated to an unreviewed binary (learning-capture.sh line 100) The hook comment states "Redacts secrets automatically before storage" but the hook itself passes raw The PR title claims this PR fixes secret auto-redaction, but the hook code does not implement any redaction itself. The security guarantee is unverifiable from the hook diff alone. Remediation: either (a) implement minimal regex-based redaction in the shell hook as a defence-in-depth layer before passing to the binary, or (b) document explicitly in the hook that redaction is guaranteed by [LOW] Inconsistent shell safety flags across hooks
A hook without [LOW] Argument-prefix injection via If [INFORMATIONAL] on-file-write.sh output uses emoji Lines 7 and 9 emit emoji characters. This is a cosmetic violation of the project's no-emoji convention (CLAUDE.md) and may cause rendering issues in terminals without full Unicode support. Dependency ChangesNone. No Cargo.toml or Cargo.lock modifications. No new crates introduced. SummaryThe hooks follow generally sound patterns: fail-open semantics, stdin-based data passing ( Last security-audited commit: 1d0b60e |
Summary
Implements secret auto-redaction in the learning capture pipeline as specified in issue #675.
Changes
New
redact(&str) -> Cow<str>function with built-in patterns covering:AKIA...) and secretssk-..., Slack tokens, GitHub tokens)Redaction format changed from
[AWS_KEY_REDACTED]to[REDACTED:<pattern-name>]so downstream readers can see that something was removed.RedactionConfigstruct withenabledflag andcustom_patternsvector for user-defined regex patterns loaded from.terraphim/learning-capture.toml.Comprehensive test coverage:
redact(input).len() <= input.len() * 2redact_secrets()aliasWiring: Already integrated in
capture.rs(capture_failed_command,capture_correction) andhook.rs(process_*functions).Verification
cargo test -p terraphim_agent --bin terraphim-agent redaction cargo clippy -p terraphim_agent -- -D warnings cargo fmt --all -- --checkAll 442 tests pass, zero clippy warnings.
Refs #675