Reduce TaskSpawner GitHub API traffic by decoupling polling from Task events and lazily fetching details

[gjkim42]

Summary

TaskSpawner currently generates more GitHub API traffic than necessary. The main causes are:

1. Task status/deletion events can trigger a full GitHub rediscovery outside the normal poll interval
2. GitHub issue/PR detail endpoints are fetched even when the active source config and templates do not need them

This issue proposes reducing GitHub traffic without changing the public TaskSpawner API.

Goals

• Keep GitHub discovery interval-driven only
• Stop Task activity from causing full GitHub rediscovery
• Fetch GitHub comments/reviews only when required by source config or templates
• Preserve existing public CRD/API shape

Proposed Plan

1. Split Task events from source polling

• Remove the Task watch from the poll controller so GitHub discovery only runs on:
  • TaskSpawner create
  • TaskSpawner spec generation changes
  • scheduled RequeueAfter poll intervals
• Add a lightweight Task activity reconcile path/controller inside the spawner process for the single managed TaskSpawner
• On Task phase/deletion changes, the Task activity controller should:
  • recompute and update status.activeTasks
  • run per-Task GitHub reporting when reporting is enabled
  • never call source discovery
• Keep refill semantics simple:
  • when maxConcurrency frees up because a Task finishes, the next GitHub item is considered on the next scheduled poll only
• Continue updating status.totalDiscovered, status.totalTasksCreated, and status.lastDiscoveryTime only during poll cycles

2. Make GitHub detail fetching lazy

• Compute internal fetch requirements in buildSource from:
  • comment policy / legacy trigger and exclude comments
  • reviewState
  • fields referenced by taskTemplate.promptTemplate or the default prompt
  • fields referenced by taskTemplate.branch
• Detect referenced template fields by parsing Go templates and walking the AST
• Keep the Source interface unchanged
• Extend internal GitHub source structs with fetch-requirement booleans

GitHub Issues rules

• Fetch issue comments only when:
  • comment policy is configured, or
  • a template references .Comments
• Otherwise leave WorkItem.Comments empty and skip issue comment requests

GitHub Pull Requests rules

• Fetch reviews only when:
  • reviewState != any, or
  • comment policy is configured, or
  • a template references .ReviewState
• Fetch conversation comments only when:
  • comment policy is configured, or
  • a template references .Comments
• Fetch review comments only when:
  • comment policy is configured, or
  • a template references .ReviewComments
• Preserve current comment-policy semantics by still loading review bodies and review comments whenever comment policy is active

Public API / Compatibility

• No CRD changes
• No new user-facing fields
• No change to the existing Source interface

Acceptance Criteria

• Task phase/deletion changes no longer trigger GitHub rediscovery
• Task phase/deletion changes still update activeTasks
• Task phase/deletion changes still perform GitHub reporting when enabled
• TaskSpawner spec changes still trigger immediate discovery
• GitHub issues source skips comment requests when .Comments is not needed
• GitHub pull request source only fetches reviews/comments required by config/template usage
• Default prompt behavior remains compatible with current output

Test Plan

Controller tests

• Task phase update/delete does not trigger GitHub discovery
• Task phase update still updates activeTasks
• Task phase update still performs GitHub reporting for that Task when enabled
• TaskSpawner spec changes still trigger immediate discovery

GitHub source tests

• Issues source with prompt/branch templates not referencing .Comments performs no comment requests
• Default prompt still causes issue comment fetches
• PR source with no comment/review fields and reviewState=any only lists PRs
• Referencing .ReviewState fetches reviews only
• Referencing .ReviewComments fetches review comments only
• Comment policy enabled fetches reviews, conversation comments, and review comments

Template analyzer tests

• Detects fields in conditionals and nested template nodes
• Treats empty prompt as the current default prompt

Notes

This issue intentionally does not include:

• webhook support
• persistent backlog state
• cross-cycle authorization caching
• public API additions

[kelos-bot]

🤖 Kelos Task Status

Task kelos-triage-794 has succeeded. ✅

[kelos-bot]

🤖 Kelos Agent @gjkim42

Kelos Triage Report

Kind: kind/feature
Priority: priority/important-longterm — Valuable optimization reducing unnecessary GitHub API traffic, but not blocking adoption or correctness
Actor: actor/kelos — Scope is well-defined with concrete acceptance criteria, no CRD changes, and objectively verifiable tests

Status: STILL VALID

Evidence:

• Task events (phase changes, deletions) currently trigger full GitHub rediscovery via the spawner reconciler's Task watch at cmd/kelos-spawner/reconciler.go:150-177 — requestsForTask() maps Task events back to TaskSpawner reconciliation which runs src.Discover(ctx)
• All GitHub data (comments, reviews, review comments) is fetched unconditionally on every discovery cycle in internal/source/github.go and internal/source/github_pr.go — no template analysis or lazy fetching exists
• No merged PRs or recent commits reference this issue
• The codebase confirms both problems described: Task events cause unnecessary rediscovery, and GitHub detail endpoints are called regardless of template/config needs

Duplicates found: #418 (partial overlap — both aim to reduce GitHub API traffic, but #418 proposes a cache/informer pattern with ETag-based delta sync, while #794 proposes decoupling Task events from polling and lazy-fetching details based on template analysis; the approaches are complementary, not duplicative)

Recommendation: KEEP OPEN — The two proposed changes (decoupling Task events from source polling, and lazy detail fetching based on template/config needs) are well-scoped, immediately actionable, and independent from the longer-term informer approach in #418.

[gjkim42]

I think we should land this in two phases inside #794, and treat event polling as a follow-up only if traffic is still too high after measurement.

Proposed plan

Phase 1: Decouple Task activity from source discovery

• Remove the current Task watch path that enqueues the main discovery reconcile.
• Add a lightweight Task activity path inside the spawner process for the single managed TaskSpawner.
• On Task phase change, deletion timestamp change, or delete:
  • recompute and update status.activeTasks
  • run per-Task GitHub reporting when reporting is enabled and the Task object still exists
  • never call source discovery
• Keep refill semantics simple:
  • if maxConcurrency frees up because a Task finishes, the next GitHub item is only considered on the next scheduled poll
• Keep status.totalDiscovered, status.totalTasksCreated, and status.lastDiscoveryTime as poll-cycle-only fields

Phase 2: Lazy-fetch GitHub details only when required

• Compute internal fetch requirements in buildSource() from:
  • comment policy / legacy trigger + exclude comments
  • reviewState
  • fields referenced by taskTemplate.promptTemplate
  • fields referenced by taskTemplate.branch
  • the default prompt when promptTemplate is empty
• Detect template field usage by parsing Go templates and walking the AST
• Keep the public Source interface unchanged
• Extend internal GitHub source structs with fetch booleans

GitHub issues

• Fetch issue comments only when:
  • comment policy is configured, or
  • a template references .Comments
• Otherwise leave WorkItem.Comments empty and skip issue comment requests

GitHub pull requests

• Fetch reviews only when:
  • reviewState != any, or
  • comment policy is configured, or
  • a template references .ReviewState
• Fetch conversation comments only when:
  • comment policy is configured, or
  • a template references .Comments
• Fetch review comments only when:
  • comment policy is configured, or
  • a template references .ReviewComments
• Preserve current comment-policy semantics by still loading review bodies and review comments whenever comment policy is active

Expected API call savings

These numbers are reductions in raw GitHub HTTP requests. Actual rate-limit savings may be somewhat lower because unchanged GETs can still be satisfied through ETag/304 handling.

Phase 1

This removes 100% of GitHub requests caused by Task-driven rediscovery.

Savings formula:

task_activity_events_per_hour x current_discovery_calls_per_cycle

Current cycle cost is roughly:

• issues source: 1 issue-list call + 1 comment call per matched issue
• PR source: 1 PR-list call + 1 reviews call + 1 conversation-comments call + 1 review-comments call per matched PR

Examples:

• 20 matched PRs:
  • current cycle cost is about 1 + (20 x 3) = 61 requests
  • if Task churn causes 40 extra rediscovery runs/hour, phase 1 saves about 40 x 61 = 2440 requests/hour
• 50 matched issues:
  • current cycle cost is about 1 + 50 = 51 requests
  • if Task churn causes 30 extra rediscovery runs/hour, phase 1 saves about 30 x 51 = 1530 requests/hour

Phase 2

This reduces the cost of every remaining scheduled poll.

Savings formula:

scheduled_polls_per_hour x detail_calls_removed_per_poll

Examples:

• issues source, comments not needed:
  • save about 1 request per matched issue per poll
  • 50 matched issues at 1m poll interval => about 60 x 50 = 3000 requests/hour saved
• PR source, minimal config:
  • reviewState=any
  • no comment policy
  • no .Comments
  • no .ReviewState
  • no .ReviewComments
  • save about 3 requests per matched PR per poll
  • 50 matched PRs at 1m poll interval => about 60 x (50 x 3) = 9000 requests/hour saved
• PR source, only .ReviewState needed:
  • save about 2 requests per matched PR per poll
  • 50 matched PRs at 1m poll interval => about 60 x (50 x 2) = 6000 requests/hour saved

Follow-up only if needed after measurement

If traffic is still too high after phases 1 and 2, the next step should be a separate issue for hybrid event polling:

• poll GitHub repository events only as a dirty signal
• refetch canonical issue/PR/review/comment data only for touched items
• keep periodic broad reconciliation as a safety net

I would not make events the authoritative discovery source for this issue.

Test plan

• Task phase update/delete no longer triggers GitHub discovery
• Task activity still updates activeTasks
• Task activity still performs GitHub reporting when enabled
• TaskSpawner spec changes still trigger immediate discovery
• issues source skips comment requests when .Comments is not needed
• default prompt still causes issue comment fetches
• PR source only fetches the review/comment endpoints actually required by config/template usage
• add request-count assertions so tests verify the reduction in HTTP calls, not just the output behavior