Skip to content

SEP-022: validate_plan_quality.sh citation check is satisfiable with fake paths #12

@samudzi

Description

@samudzi

Summary

if ! echo "$JUST_CONTENT" | grep -qE '(docs/|scripts/|tools/|...CLAUDE\.md|README|...)'; then

Any plan that mentions "CLAUDE.md" in the Justification section passes this check, regardless of whether it actually explains what CLAUDE.md says about the change. The causal reasoning check (because, therefore, etc.) helps but can also be trivially satisfied.

Motivation

This is probably acceptable — the quality gate is a filter, not a proof. But the known bypass should be documented so future SEPs don't try to make it airtight at the cost of false positives.

Proposed Change

Document the known bypass in the script comments and/or ARCHITECTURE.md. Consider whether a stronger check (e.g., requiring a quote from the cited file) is worth the false-positive risk.

Acceptance Criteria

  • Known limitation is documented in script comments
  • Decision recorded on whether stronger checking is warranted
  • No increase in false positives for legitimate plans

Metadata

Metadata

Assignees

No one assigned

    Labels

    SEPSoftware Evolution ProposalenhancementNew feature or requestminorMinor issues

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions