Add in comparator, negated modifier, and plain-JSON nested-path filters

[kriszyp]

Summary

Tier-1 Resource API query additions (RESTful + structured shapes), covering items from RESOURCE-API-TIER1-PROPOSAL.md:

• in comparator — native, with REST (v1,v2,v3) value-list syntax (?status=in=(active,pending)). Phase 1 routes through filter; index-merge optimization is a Phase 2 follow-up.
• negated modifier — both via a negated: boolean flag on DirectCondition and via not_X prefix on comparator names (not_in, not_starts_with, not_contains, not_between, not_ends_with). The parser strips the prefix and sets the flag; the execution layer normalizes structured queries the same way via resolveComparator.
• Plain-JSON nested-path filters — attribute: ['address', 'city'] (or REST ?address.city=…) is no longer rejected when the first segment isn't a declared relationship. Array intermediates use some semantics, mirroring relationship traversal.
• SubSelect array projection fix — select: [{ name: 'addresses', select: ['city'] }] now correctly maps per-element when the value is an array of plain JSON objects (previously collapsed to one undefined entry).
• Multi-character FIQL operators — relaxed the historical 2-character cap so =in=, =between=, =not_starts_with= etc. parse.
• Filter-side range cases — gele/gtlt/gtle/gelt (the chained-condition collapse output) are now handled in filterByType, fixing a latent bug where chained gt/lt on non-indexed/full-scan paths threw "Unknown query comparator".

Why

The SQL engine translates WHERE x IN (...), WHERE address.city = …, and WHERE NOT (...) shapes into Resource API conditions via whereToConditions.ts. Today these become residual filters (extra rows pulled, JS-filtered) because the Resource API rejects nested non-relationship paths and has no native in/not. This change moves them to first-class condition shapes, so the SQL engine can emit them directly and the optimizer can target indexable plans.

What to look at first

• resources/search.ts — the bulk of the logic. Specific spots:
  • resolveComparator and buildCondition — alias resolution + not_ stripping + (v1,v2) list-value detection at parse time.
  • searchByIndex — non-relationship array attributes now fall through to filter (line ~206); the negated branch overrides start/end to a full scan and skipIndex forces primary-store iteration so null-valued records still match a negated filter (semantic correctness for nulls outside the secondary index).
  • filterByType — 'in' case (cached Set); negated wrapping at the end; new plain-JSON nested-path branch using a synthetic NESTED_PATH_TABLE to recurse cleanly through the comparator switch.
• resources/Table.ts — prepareConditions normalizes not_X from structured queries, and allows nested paths whose root attribute exists but whose leaf is not in the schema (skips type coercion in that case to avoid coercing string values as the root's object type). SubSelect array fix is in transformEntryForSelect.
• unitTests/resources/query-tier1.test.js — 39 tests, including parsing, execution, nested paths, multi-value association (independent vs chained), and edge cases (null values, triple-nested array intermediates, backwards-compat).

Notes for review

• Backwards compat: value=gt=(4) still produces the literal string "(4)" because gt isn't a list-value comparator. The (...) list parsing only kicks in for in and between. Confirmed by an explicit regression test.
• Negation semantics on nulls: a record with status: null matches not_in=(active) because null is not in the list. This required iterating the primary store (not the secondary index) for negated queries on indexed attributes — the secondary index doesn't include null entries unless indexNulls is set.
• Multi-value association: independent conditions (child.age=gt=10&child.age=lt=15) match when ANY child satisfies each independently (different children OK); chained (child.age=gt=10&=lt=15) collapses to a single range filter that requires the SAME child to satisfy both. Tests confirm both shapes; the chained-condition execution path in Table.ts:prepareConditions is unchanged (still limited to gt/lt → range).
• Cross-model review: ran git diff | gemini -p … — flagged null-record behavior (now tested + fixed), unused getNestedValue (kept as a public utility — it's tested and may be useful to callers like the SQL engine), and recommended triple-nested path tests (added).
• Pre-existing failures unrelated: unitTests/security/keys.test.js (cert handling) and unitTests/resources/transaction.test.js (test ordering, passes alone) fail on this branch the same way they fail on main.

Test plan

• npx mocha unitTests/resources/query-tier1.test.js — 39 passing
• npx mocha unitTests/resources/query.test.js — 102 passing (no regressions)
• npx mocha unitTests/resources/query-parse.test.js — 21 passing (no regressions)
• npm run test:unit:resources — 379 passing (1 pre-existing flaky transaction test)
• npm run lint:required — clean
• npm run format:write — clean

🤖 Generated with Claude Code

[claude]

not_X comparators missing from the Comparator union

not_in, not_starts_with, not_between, not_contains, and not_ends_with are valid runtime inputs to the structured query API — prepareConditions normalizes them, and the test suite has a test explicitly titled "accepts not_in directly without going through parser". They're not in the Comparator union, so TypeScript callers get a compile error when using these strings.

not_equal (the existing ne alias) is already in the type and sets the precedent. Either add the new not_X entries here, or clarify in the test description that these are internal aliases not intended as typed public inputs (in which case { comparator: 'in', negated: true } is the only typed path).

Suggested change

	| 'in'
	+ | 'in'

(Extend to add 'not_in' | 'not_starts_with' | 'not_between' | 'not_contains' | 'not_ends_with' after 'in' if these are intended as public API strings.)

[claude]

One finding:

1. not_X comparators missing from the Comparator union type

File: resources/ResourceInterface.ts:143
What: not_in, not_starts_with, not_between, not_contains, not_ends_with are valid structured-query inputs — prepareConditions normalizes them and one test is explicitly titled "accepts not_in directly without going through parser" — but they're absent from the Comparator union. TypeScript callers get a compile error. not_equal (the existing ne alias) is already in the type.
Why it matters: Harper core doesn't use tsc || true, so type correctness is a real bar. External callers using the typed API can't express these forms without a cast, even though the runtime accepts them.
Suggested fix: Add 'not_in' | 'not_starts_with' | 'not_between' | 'not_contains' | 'not_ends_with' to the Comparator union — or, if the intent is that { comparator: 'in', negated: true } is the only typed path, update the test description to say so and the runtime behavior is fine as-is.