Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
333 lines (288 loc) · 17.1 KB

ARCHITECTURE.md

File metadata and controls

333 lines (288 loc) · 17.1 KB

Architecture Overview

PortOS is a monorepo application with a React frontend and Express.js backend, managed by PM2.

System Diagram

┌─────────────────────────────────────────────────────────────────────────────────────┐
│                                      PortOS                                         │
├─────────────────────────────────────────────────────────────────────────────────────┤
│                                                                                     │
│  ┌────────────────────┐             ┌──────────────────────────────────────┐        │
│  │  React Client      │             │        Express Server                │        │
│  │  (port 5554)       │    HTTP     │        (port 5555)                   │        │
│  │                    │ <---------> │                                      │        │
│  │  ┌──────────────┐  │             │  ┌──────────┐  ┌──────────────────┐  │        │
│  │  │    Pages     │  │             │  │  Routes  │──│    Services      │  │        │
│  │  └──────────────┘  │             │  └──────────┘  └──────────────────┘  │        │
│  │       |            │  Socket.IO  │       |              |               │        │
│  │  ┌──────────────┐  │ <---------- │       |         ┌────v─────────┐     │        │
│  │  │  Components  │  │             │       |         │   PM2 API    │     │        │
│  │  └──────────────┘  │             │       |         └──────────────┘     │        │
│  │       |            │             │       |              |               │        │
│  │  ┌──────────────┐  │             │       |         ┌────v─────────┐     │        │
│  │  │  api.js      │  │             │       |         │  JSON Files  │     │        │
│  │  │  socket.js   │  │             │       |         │   (data/)    │     │        │
│  │  └──────────────┘  │             │       |         └──────────────┘     │        │
│  └────────────────────┘             └──────────────────────────────────────┘        │
│                                                                                     │
│  ┌───────────────────────────────────────────────────────────────────────────────┐  │
│  │                    PM2-Managed Satellite Services                             │  │
│  │                                                                               │  │
│  │  ┌─────────────────────┐  ┌────────────────────┐  ┌────────────────────────┐  │  │
│  │  │ Chief of Staff      │  │ portos-browser     │  │ portos-autofixer       │  │  │
│  │  │ portos-cos :5558    │  │ CDP :5556          │  │ daemon :5559           │  │  │
│  │  │                     │  │ health :5557       │  │ UI :5560               │  │  │
│  │  │ Task Watcher        │  │                    │  │                        │  │  │
│  │  │ CoS Evaluation      │  │ Persistent         │  │ PM2 crash monitor      │  │  │
│  │  │ Sub-Agent Spawner   │  │ Chromium instance  │  │ (polls every 15m)      │  │  │
│  │  │ (Claude CLI) -------|->│ CDP WebSocket for  │  │ Claude CLI auto-fix    │  │  │
│  │  │                     │  │ web automation     │  │ Reads apps.json        │  │  │
│  │  └─────────────────────┘  └────────────────────┘  │ Session history        │  │  │
│  │                                                   └────────────────────────┘  │  │
│  └───────────────────────────────────────────────────────────────────────────────┘  │
│                                                                                     │
└─────────────────────────────────────────────────────────────────────────────────────┘

Communication paths:
  Client <--HTTP/Socket.IO--> Server --PM2 API--> all satellite processes
  Server --browserService--> portos-browser (CDP :5556, health :5557)
  Server --apps.json--> portos-autofixer reads registered apps to monitor
  CoS agents --CDP WebSocket--> portos-browser for web automation tasks
  portos-autofixer --pm2 jlist--> detects crashed processes --Claude CLI--> auto-fix
  portos-autofixer-ui --reads--> data/autofixer/sessions/ for fix history

Directory Structure

PortOS/
├── client/                    # React + Vite frontend
│   └── src/
│       ├── components/        # Reusable UI components
│       │   ├── cos/           # Chief of Staff components
│       │   └── Layout.jsx     # Main app layout
│       ├── hooks/             # Custom React hooks
│       ├── pages/             # Route-based page components
│       │   └── Browser.jsx    # Browser management dashboard
│       └── services/          # API client (api.js, socket.js)
│
├── server/                    # Express.js backend
│   ├── routes/                # HTTP endpoint handlers
│   │   └── browser.js         # /api/browser/* endpoints
│   ├── services/              # Business logic
│   │   ├── cos.js             # Chief of Staff core
│   │   ├── subAgentSpawner.js # Claude CLI integration
│   │   ├── pm2.js             # PM2 process management
│   │   ├── runner.js          # AI execution engine
│   │   ├── memory.js          # Memory system
│   │   └── browserService.js  # Browser CDP/health/PM2 control
│   ├── lib/                   # Shared utilities
│   │   ├── errorHandler.js    # Error normalization
│   │   ├── validation.js      # Zod schemas
│   │   └── taskParser.js      # TASKS.md parser
│   └── cos-runner/            # Isolated agent runner
│       └── index.js           # Standalone Express server
│
├── browser/                   # portos-browser service
│   ├── server.js              # Launches Chromium with CDP, runs health server
│   └── package.json           # Playwright dependency
│
├── autofixer/                 # portos-autofixer service
│   ├── server.js              # Crash detection daemon (polls PM2 every 15min)
│   └── ui.js                  # Standalone Express UI with SSE log streaming
│
├── data/                      # Runtime data (gitignored)
│   ├── apps.json              # Registered apps (read by autofixer)
│   ├── providers.json         # AI provider configs
│   ├── history.json           # Action history
│   ├── browser-config.json    # Browser CDP/health configuration
│   ├── TASKS.md               # User task file
│   ├── COS-TASKS.md           # System task file
│   ├── COS-GOALS.md           # Mission and goals
│   ├── cos/                   # CoS state and agents
│   │   ├── state.json         # Daemon state
│   │   ├── agents/            # Agent outputs
│   │   └── memory/            # Memory storage
│   ├── autofixer/             # Autofixer session history
│   │   ├── index.json         # Fix session index (max 100 entries)
│   │   └── sessions/          # Per-session prompt, output, metadata
│   ├── brain/                 # Brain second-brain data
│   │   ├── meta.json          # Settings
│   │   ├── inbox_log.jsonl    # Captured thoughts
│   │   ├── people.jsonl       # People records
│   │   ├── projects.jsonl     # Project records
│   │   ├── ideas.jsonl        # Ideas
│   │   ├── admin.jsonl        # Admin tasks
│   │   ├── links.json         # Saved links
│   │   └── digests.jsonl      # Daily/weekly digests
│   ├── digital-twin/          # Digital twin identity documents
│   │   ├── meta.json          # Settings and state
│   │   └── documents/         # Markdown identity documents
│   ├── uploads/               # Generic file uploads
│   ├── repos/                 # Cloned GitHub repositories
│   └── agent-personalities/   # Agent personality configs
│
├── docs/                      # Documentation
├── .github/workflows/         # CI/CD
└── ecosystem.config.cjs       # PM2 configuration

Data Flow

HTTP Request Flow

Browser → React Page → api.js → Express Route → Service → Response
                                     │
                                     ├── Zod Validation
                                     ├── Service Logic
                                     └── JSON File / PM2 API

WebSocket Event Flow

Server Event → Socket.IO → socket.js → React Component State Update
     │
     └── Real-time: logs, CoS status, errors, memory changes

Chief of Staff Flow

1. Task Watcher monitors TASKS.md for changes
2. CoS Service evaluates tasks on interval
3. For each pending task:
   a. Select appropriate AI model based on task complexity
   b. Build prompt with context and memory injection
   c. Spawn Claude CLI via Sub-Agent Spawner
4. Agent executes task, output captured
5. On completion:
   a. Mark task as completed
   b. Extract memories from output
   c. Update usage metrics

Browser Automation Flow

1. portos-browser launches persistent Chromium with CDP on :5556
2. Health server on :5557 reports connection status
3. Express Server proxies browser management via browserService.js:
   - Client UI (Browser.jsx) → /api/browser/* → browserService → CDP/PM2
4. CoS agents connect directly to CDP WebSocket for web automation
5. Configuration persisted to data/browser-config.json

Autofixer Flow

1. portos-autofixer daemon starts, reads registered apps from data/apps.json
2. Every 15 minutes, polls PM2 (pm2 jlist) for crashed processes
3. For each errored process (with 30min cooldown):
   a. Fetch last 100 lines of error logs + 50 lines of output logs
   b. Build prompt with crash context and app info
   c. Spawn Claude CLI in app's repo directory to diagnose and fix
   d. Save session (prompt.txt, output.txt, metadata.json) to data/autofixer/sessions/
4. portos-autofixer-ui (:5560) serves standalone dashboard:
   - SSE endpoint for real-time log streaming
   - Fix history viewer with success/failure status
   - Process status indicators

Key Services

Apps Service (server/services/apps.js)

  • CRUD operations for registered apps
  • Persists to data/apps.json

PM2 Service (server/services/pm2.js)

  • Start/stop/restart processes
  • Status monitoring
  • Log retrieval

Runner Service (server/services/runner.js)

  • AI provider execution
  • CLI and API-based providers
  • Output streaming and capture

CoS Service (server/services/cos.js)

  • Task evaluation and prioritization
  • Agent orchestration
  • Health monitoring
  • Self-improvement task generation

Sub-Agent Spawner (server/services/subAgentSpawner.js)

  • Claude CLI process spawning
  • Model selection based on task complexity
  • MCP server integration
  • Usage tracking

Memory Service (server/services/memory.js)

  • Semantic memory storage
  • Vector embeddings via LM Studio
  • Memory retrieval for context injection

Task Learning Service (server/services/taskLearning.js)

  • Completion tracking and success rates
  • Duration estimates by task type
  • Model tier effectiveness analysis
  • Actionable recommendations

Script Runner Service (server/services/scriptRunner.js)

  • Cron-based script scheduling
  • Command allowlist enforcement
  • Agent trigger integration
  • Run history tracking

Brain Service (server/services/brain.js)

  • Thought capture and AI classification
  • CRUD for People, Projects, Ideas, Admin
  • Daily digest and weekly review generation
  • Classification correction workflow
  • Link capture with GitHub auto-clone

Digital Twin Service (server/services/digital-twin.js)

  • Identity scaffold document management
  • Personality trait extraction (Big Five, values hierarchy)
  • Behavioral test generation and execution
  • External data import (Goodreads, Spotify, Letterboxd, iCal)
  • Confidence scoring and gap recommendations

Agent Personalities (server/services/agentPersonalities.js)

  • Agent personality CRUD and AI generation
  • Custom communication styles, tones, and quirks

Browser Service (server/services/browserService.js)

  • Manages portos-browser lifecycle via PM2 (launch/stop/restart)
  • Proxies CDP queries (open pages, version info) via HTTP to :5556
  • Health checks against :5557
  • Configuration CRUD persisted to data/browser-config.json
  • CDP host restricted to localhost to prevent SSRF

Autofixer (autofixer/server.js + autofixer/ui.js)

  • Daemon (:5559): Polls PM2 every 15 minutes for errored processes
  • Reads data/apps.json to know which processes to monitor
  • 30-minute cooldown per process to prevent fix loops
  • Spawns Claude CLI with crash context (error logs + app info) to auto-repair
  • Stores fix sessions in data/autofixer/sessions/ (prompt, output, metadata)
  • UI (:5560): Standalone Express server with SSE real-time log streaming
  • Fix history viewer, process status dashboard

Shell Service (server/services/shell.js)

  • PTY-based web terminal via node-pty
  • Session management with WebSocket I/O
  • Terminal resize handling

Error Handling

All routes use asyncHandler wrapper from server/lib/errorHandler.js:

// Routes automatically catch errors and:
// 1. Log to console with emoji prefix
// 2. Emit Socket.IO event for UI notification
// 3. Return structured JSON error response

Error severity levels:

  • warning: Non-critical, logged only
  • error: Server error, shown to user
  • critical: System-threatening, triggers auto-fix

Security Model

  1. Network Security: Relies on Tailscale for access control
  2. Command Allowlist: Shell execution restricted to approved commands
  3. No Shell Interpolation: Uses spawn() with argument arrays
  4. Zod Validation: All API inputs validated
  5. Path Traversal Prevention: Filename sanitization on uploads

PM2 Process Map

Process Port Script Purpose
portos-client 5554 client/ (Vite) React frontend dev server
portos-server 5555 server/index.js Main Express API server
portos-browser 5556 (CDP), 5557 (health) browser/server.js Persistent Chromium with CDP for web automation
portos-cos 5558 server/cos-runner/index.js Isolated CoS agent runner
portos-autofixer 5559 autofixer/server.js Autonomous crash detection and Claude CLI repair
portos-autofixer-ui 5560 autofixer/ui.js Standalone fix history dashboard with SSE logs

Extension Points

Adding a New Page

  1. Create component in client/src/pages/
  2. Add route in client/src/App.jsx
  3. Add navigation link in client/src/components/Layout.jsx

Adding an API Endpoint

  1. Create route file in server/routes/
  2. Register in server/index.js
  3. Add Zod schema if needed in server/lib/validation.js

Adding a Service

  1. Create service file in server/services/
  2. Export functions (not classes)
  3. Import in routes as needed

Adding CoS Task Types

  1. Update SELF_IMPROVEMENT_TYPES in server/services/cos.js
  2. Add prompt template in generateSelfImprovementTask()