feat: add meta mode Phase 3 — classify and contribute upstream#376
Open
Maxusmusti wants to merge 3 commits into
Open
feat: add meta mode Phase 3 — classify and contribute upstream#376Maxusmusti wants to merge 3 commits into
Maxusmusti wants to merge 3 commits into
Conversation
Add the ability for meta mode to distinguish general improvements from project-specific ones, and contribute the general items back upstream as PRs. This closes the self-improvement loop: the more the factory is used across diverse projects, the better its default playbooks become. New CLI command `factory contribute` with three modes: - `--classify`: scores evolved playbook items on a general-vs-specific spectrum using four weighted signals (cross-project prevalence 40%, domain independence 25%, evidence strength 20%, category signal 15%) - `--submit`: creates a PR against the factory repo with approved items - `--status`: shows pending contribution candidates The classification engine uses cross-project experiment data to identify items that appear across 3+ unrelated projects as "general" (upstream candidates), single-project items as "specific" (local only), and everything in between as "uncertain" (needs more data). The CEO prompt is updated with Phase 3 (M4/M5/M6) which runs after ACE evolution, presents the user with a terminal summary showing the distinction, and lets them opt in to contributing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #376 +/- ##
==========================================
+ Coverage 87.54% 87.56% +0.02%
==========================================
Files 60 62 +2
Lines 9170 9734 +564
==========================================
+ Hits 8028 8524 +496
- Misses 1142 1210 +68 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add 26 tests covering uncovered paths: classify_evolved_playbooks pipeline, package_evidence, prepare_contribution, execute_contribution (mocked subprocess), explain_specificity/uncertainty branches, load_candidates edge cases, and cmd_contribute CLI handler. Fix lint: remove unused imports, rename ambiguous variable, drop unused locals. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
factory contributeCLI command with--classify,--submit, and--statussubcommandsMotivation
Currently, meta mode evolves playbooks locally via ACE — all learnings stay in
~/.factory/playbooks/and never flow back upstream. This means every user independently re-discovers the same improvements. With this change, the factory can identify patterns that are universally useful across diverse projects and contribute them back to the default playbooks, closing the self-improvement loop: the more the factory is used, the better it becomes for everyone.How it works
Classification engine (
factory/ace/contributor.py)Each evolved playbook item is scored on a general-vs-specific spectrum using four weighted signals:
Items scoring ≥ 0.65 are classified as general, ≤ 0.35 as specific, and between as uncertain.
User experience
At the end of a meta mode run, users see a terminal summary:
Users can then run
factory contribute --submitto create a PR, or skip — contribution is always opt-in.CLI commands
CEO prompt changes
Phase 3 (steps M4/M5/M6) is added after Phase 2 (ACE). The CEO:
factory contribute --classifyto score evolved itemsFiles changed
factory/ace/contributor.pyfactory/cli.pyfactory contributecommand with--classify/--submit/--statussubcommandsfactory/agents/prompts/ceo.mdtests/test_contributor.pyDesign decisions
ClassifiedItemwrappingPlaybookItem(sincePlaybookItemhasextra="forbid")Playbook.from_markdown(),classify_hypothesis(),discover_projects(),load_all_histories(),DEFAULTS_DIR,user_playbooks_dir()prepare_contribution()returns specs without executing git — keeps the module testable;execute_contribution()handles the actual git/gh commands separatelyTest plan
pytest tests/test_contributor.py)factory contribute --helpshows correct usagefactory contribute --classifyon a project with evolved playbooksfactory contribute --submit --dry-runto verify PR spec generation🤖 Generated with Claude Code