Skip to content

feat: add parameterized query output support#34

Merged
h22rana merged 69 commits intomasterfrom
feat/parameterized-query-output
Mar 30, 2026
Merged

feat: add parameterized query output support#34
h22rana merged 69 commits intomasterfrom
feat/parameterized-query-output

Conversation

@h22rana
Copy link
Copy Markdown
Owner

@h22rana h22rana commented Mar 26, 2026

Summary

  • Add a fully isolated, non-breaking parameterized query pipeline that returns SQL with bind placeholders (@p1, $1) plus a separate parameter list, instead of inlined literals
  • Dialect-aware placeholder styles: named (@p1) for BigQuery/Spanner/ClickHouse, positional ($1) for PostgreSQL/DuckDB
  • Schema coercion applied before parameter binding (e.g., string "50000" -> int64(50000) for integer fields)
  • New E350 ErrUnreferencedPlaceholder error code catches custom operators that drop arguments
  • Full API symmetry: 6 Transpiler methods + 6 package-level convenience functions mirroring existing inline API
  • REPL: :params toggle command switches between inline and parameterized output
  • WASM: transpileParameterized, transpileConditionParameterized, quickTranspileParameterized JS functions + playground UI checkbox
  • Large integer precision: JSON decoding uses json.Decoder.UseNumber() to preserve exact integer values. Unquoted integers like 9223372036854775808 now produce exact SQL instead of lossy float64 representations.

What gets parameterized

Value Parameterized? Reason
Strings, numbers Yes User-supplied data
NULL, TRUE, FALSE No Structural SQL tokens
Column names (var) No Identifiers, not data

Example

sql, params, _ := jsonlogic2sql.TranspileParameterized(
    jsonlogic2sql.DialectBigQuery,
    `{"and": [{"==": [{"var": "status"}, "active"]}, {">": [{"var": "amount"}, 1000]}]}`,
)
// sql    = "WHERE (status = @p1 AND amount > @p2)"
// params = [{Name: "p1", Value: "active"}, {Name: "p2", Value: 1000}]

REPL

[BigQuery] jsonlogic> :params
Parameterized mode: ON (output uses bind placeholders)

[BigQuery] jsonlogic> {"==": [{"var": "status"}, "active"]}
SQL:    WHERE status = @p1
Params: [{p1: "active"}]

Architecture

All existing functions remain untouched. New parallel *Param methods are created throughout the stack:

  • internal/params/ParamCollector, PlaceholderStyle, ValidatePlaceholderRefs, FindQuotedPlaceholderRef, ValueForPlaceholder
  • internal/operators/*.goToSQLParam() / valueToSQLParam() for all operator types
  • internal/parser/parser.goParseParameterized pipeline with ParamExpressionParser callback + quoted-placeholder guard for custom operators
  • parameterized.go — public QueryParam type and all 12 public functions
  • internal/errors/errors.goE350 ErrUnreferencedPlaceholder
  • cmd/repl/main.go:params toggle, printParams helper, parameterized-aware custom operators
  • demo/wasm/main.go — 3 new JS bridge functions
  • demo/wasm/index.html — Parameterized checkbox with bind params display

Post-review fixes

The following issues were identified during code review and fixed in follow-up commits:

Large integer precision (json.Decoder.UseNumber)

  • Root cause: json.Unmarshal coerces all JSON numbers to float64, losing precision for integers exceeding 2^53. For example, 9223372036854775808 and 9223372036854775809 produced identical SQL.
  • Fix: Switched to json.Decoder.UseNumber() which preserves numbers as json.Number (string-backed). Added json.Number handling to all operator paths (isPrimitive, isNumber, valueToSQL, valueToSQLParam, getNumber, coerceValueForComparison, handleIn/handleInParam, primitiveToSQLParam).
  • Parameterized behavior: JS-safe integers (within ±2^53−1) bind as float64 for backward compatibility; overflow integers bind as string.

Out-of-range float preservation

  • Root cause: jsonNumberParamValue errored on float64 overflow (e.g., 1e309) and silently bound underflow values (e.g., 1e-400) as 0, while inline transpilation preserved these literals verbatim.
  • Fix: Changed jsonNumberParamValue to fall back to the original numeric string when ParseFloat produces infinity or underflow-to-zero. Added the same underflow guard to parser.primitiveToSQLParam (separate conversion path used by custom operators). Both paths now match inline behavior.

Quoted-placeholder guard for custom operators

  • Root cause: Custom operators in parameterized mode receive placeholders (e.g., @p1) as plain string arguments. A naive operator could wrap them in SQL string literals (e.g., '@p1'), which passes validation but silently breaks bind semantics at runtime.
  • Fix: Added FindQuotedPlaceholderRef in internal/params/ that scans SQL string literals for placeholder tokens with proper boundary checking (avoids false positives like 'name@p1.example'). The parser now runs this guard after custom operator SQL generation in parameterized mode, failing with E102 ErrCustomOperatorFailed if a placeholder is found inside quotes. Valid expressions like CONCAT(@p1, '%') continue to work.

Parameterized in operator

  • String containment with custom-op SQL literals: ProcessedValue{IsSQL:true} from custom operators (e.g., 'foo') was misrouted to array membership (IN UNNEST) instead of string containment (STRPOS). Fixed by mirroring the non-param quoting heuristic.
  • String containment with custom-op placeholders: Custom operators returning bare placeholders (e.g., @p1) lost type information. Added ParamCollector.ValueForPlaceholder() to look up the Go type of the stored parameter and route string values to containment.
  • Dialect-aware strposFunc routing: Both handleIn and handleInParam now use dialect-aware strposFunc for literal string/number containment (e.g., STRPOS on BigQuery, POSITION(x IN y) on PostgreSQL).
  • Schema coercion: handleInParam now defers placeholder generation until after schema-based coercion is applied.

REPL custom LIKE operators

  • Placeholder embedding: Custom LIKE operators (startsWith, endsWith, contains and negations) were embedding placeholders inside SQL string literals ('@p1%'). Refactored with buildLikeSQL helper to use CONCAT(@p1, '%').
  • Wildcard escaping: Parameterized LIKE patterns now escape %, _, \ at runtime via SQL REPLACE functions.
  • CAST for numeric patterns: Numeric bind values are cast to string (CAST(@p1 AS STRING) / CAST($1 AS TEXT)) before REPLACE to prevent type errors.
  • Reversed contains args: Added isBoundValue() helper for robust column-vs-pattern detection in parameterized mode.
  • Array-pattern contains: Single-element array patterns (["x"]) in parameterized mode are now formatted as strings in primitiveToSQLParam so custom operators can detect and unwrap them consistently.

BigQuery regexpContains

  • Raw string prefix: BigQuery's r'...' prefix is only valid for inline SQL literals. In parameterized mode, the pattern is now emitted without the r prefix to avoid invalid SQL and E350 errors.

Security

  • WASM XSS prevention: Replaced innerHTML with textContent + DOM element creation for rendering SQL output in the WASM playground.
  • json.Number SQL injection via map/interface APIs: TranspileFromMap/TranspileFromInterface accepted crafted json.Number values (e.g., json.Number("1 OR 1=1")) and inlined them verbatim, enabling SQL injection. Defense-in-depth at two layers: (1) validJSONNumberLiteral regex in data.valueToSQL, numeric.valueToSQL, and their parameterized counterparts rejects invalid literals at the operator level; (2) the validator's isNumber() now validates json.Number against the JSON number grammar, catching invalid values at the earliest point before any operator (built-in or custom) sees them. The Transpile(string) path was not affected since json.Decoder only produces valid numeric tokens.

Typed-nil schema fix

  • SetSchema(nil) and NewTranspilerWithConfig with a nil *Schema pointer now correctly return a true nil interface, ensuring no-schema identifier validation (regex whitelist) is not bypassed.

Code quality and optimizations

  • JSON decoding allocation: Switched bytes.NewBufferString to strings.NewReader in decodeJSONLogic to avoid unnecessary byte slice copy.
  • In-place array mutation: handleIn and handleInParam now copy arrays before schema coercion to prevent mutating caller-provided slices.
  • Placeholder validation perf: Replaced per-parameter regexp.Compile in ValidatePlaceholderRefs with boundary-aware strings.Index scanning.
  • WASM error handling: All json.Marshal calls in demo/wasm/main.go now handle errors instead of swallowing them.

Documentation

  • Updated QueryParam.Value type documentation to cover unquoted large JSON integers, quoted string inputs, and out-of-range floats, with UseNumber explanation.
  • Clarified parameterized LIKE placeholder patterns (valid vs invalid forms).
  • Documented quoted-placeholder rejection rule for custom operators.

Test plan

  • All existing tests pass (no regressions)
  • Unit tests for internal/params package (collector, styles, validator, ValueForPlaceholder, FindQuotedPlaceholderRef)
  • Unit tests for all operator ToSQLParam methods (data, comparison, numeric, logical, string, array)
  • Parser ParseParameterized tests across BigQuery and PostgreSQL dialects
  • Integration tests in parameterized_test.go covering all operators, all 5 dialects, schema coercion, custom operators, nested expressions
  • Large integer precision tests: distinct values produce distinct SQL and exact parameterized strings
  • Out-of-range float tests: 1e309 overflow and 1e-400 underflow preserve original string in both operator and custom operator paths
  • Quoted-placeholder guard: rejection of '@p1' patterns and acceptance of CONCAT(@p1, '%') expressions
  • json.Number injection rejection: TranspileFromMap/TranspileFromInterface reject crafted non-numeric json.Number values in both built-in and custom operator paths
  • Error parity tests ensuring identical error codes between inline and parameterized pipelines
  • E350 detection test for custom operators that drop arguments
  • REPL tests: printParams formatting, params mode toggle, parameterized transpile, custom LIKE operators, reversed args, regexpContains BigQuery, array-pattern contains
  • Typed-nil schema tests (NewTranspilerWithConfig nil pointer, SetSchema(nil) restore)
  • Dialect compliance test for in string containment and numeric boundaries across all 5 dialects
  • make lint — 0 issues
  • make test — all 8 packages pass
  • make bench — benchmarks pass
  • make build / make build/wasm — successful

h22rana added 30 commits March 26, 2026 17:32
Add a fully isolated, non-breaking parameterized query pipeline that
returns SQL with bind placeholders (@p1/$1) plus parameter values
instead of inlined literals. All existing functions remain untouched;
new parallel *Param methods are created throughout the stack.

- internal/params: ParamCollector, PlaceholderStyle, ValidatePlaceholderRefs
- operators: ToSQLParam/valueToSQLParam for all operator types
- parser: ParseParameterized pipeline with ParamExpressionParser callback
- parameterized.go: public API (6 Transpiler methods + 6 package functions)
- errors: E350 ErrUnreferencedPlaceholder for dropped-arg detection

Made-with: Cursor
Unit and integration tests covering all parameterized methods across
operators, parser, params package, and the public API. Includes tests
for all 5 dialects, schema coercion, custom operators, nested expressions,
error parity, and E350 unreferenced placeholder detection.

Made-with: Cursor
- New docs/parameterized-queries.md with full guide (placeholder styles,
  schema coercion, database driver examples, custom operator contract)
- Update README.md with parameterized query feature and example
- Update api-reference.md with new methods, QueryParam type
- Update getting-started.md, examples.md, dialects.md with param sections
- Update error-handling.md with E350 entry

Made-with: Cursor
REPL:
- Add :params toggle command to switch between inline and parameterized output
- printParams helper formats bind parameters for display
- Both main input loop and :file command respect params mode

WASM:
- Add transpileParameterized, transpileConditionParameterized,
  quickTranspileParameterized JS bridge functions
- Add Parameterized checkbox to playground UI with bind params display
- Multi-dialect comparison also shows params when enabled

Made-with: Cursor
- TestPrintParams: verifies formatted output for empty, single, multiple params
- TestParamsModeToggle: verifies toggle state flip
- TestTranspileParameterized_BigQuery: 6 cases (equality, numeric, AND, IN, null, bool)
- TestTranspileParameterized_PostgreSQL: positional $1 placeholders
- TestTranspileParameterized_CustomOperator: custom operators in parameterized mode
- TestTranspileParameterized_Error: invalid JSON and unknown operator paths

Made-with: Cursor
- README: add :params example to Interactive REPL section
- docs/repl.md: add :params command and Parameterized Output section
- docs/wasm-playground.md: add parameterized JS API examples and
  playground feature entry
- Rebuild REPL and WASM binaries

Made-with: Cursor
…gers

- REPL LIKE operators (startsWith, endsWith, contains) now use CONCAT
  for bind placeholders instead of embedding them inside SQL string
  literals, which rendered placeholders non-functional
- Large integer strings exceeding int64 range are preserved as strings
  in parameterized numeric path instead of being rounded via float64

Made-with: Cursor
…ecision

- REPL parameterized LIKE tests for all 6 operators (startsWith,
  !startsWith, endsWith, !endsWith, contains, !contains)
- Non-parameterized LIKE regression tests to ensure backward compat
- Unit tests for large int64 overflow preservation in numeric operator
- Integration test for large integer parameterized transpilation
- Updated parseContainsArgs tests for raw SQL pattern return value

Made-with: Cursor
- NewParser: update comment to reflect nil config fallback behavior
- PlaceholderQuestion: document per-parameter validation limitation

Made-with: Cursor
Document why LIKE patterns must concatenate bind placeholders (e.g., CONCAT(@p1, '%'))
instead of embedding placeholder text in string literals, and add REPL-based revalidation notes.

Made-with: Cursor
Add explicit guards for parameterized LIKE placeholder handling to ensure bind tokens
are not quoted in pattern literals across dialects, and add a large-integer preservation
case for numeric parameterized output.

Made-with: Cursor
Document the newly added test coverage and validation commands so the runtime findings
remain traceable in local PR review artifacts.

Made-with: Cursor
Drop parameterized-query-output-pr-review.md from the branch so local review notes
are not included in repository commits or PR diff.

Made-with: Cursor
…n bugs

- Defer leftSQL placeholder generation in handleInParam so the
  placeholder is created only after coercion decisions are made,
  preventing orphaned parameters that triggered false E350 errors
- Use the Go type of the original left value (string vs non-string)
  instead of SQL quoting to distinguish string containment from array
  membership, fixing misclassification when placeholders replaced
  quoted literals

Made-with: Cursor
- Update handleInParam unit tests for refactored signature
- Add string-containment-without-schema and schema-coercion unit tests
- Add integration tests for string containment across dialects
- Add integration tests for numeric-to-string coercion with schema

Made-with: Cursor
…pe LIKE wildcards in parameterized mode

Route literal string/number right-hand sides in the 'in' operator through
strposFunc instead of hardcoded POSITION(), ensuring dialect-correct SQL
(STRPOS for BigQuery/Spanner/DuckDB, POSITION for PostgreSQL, position()
for ClickHouse). Wrap LIKE placeholders in SQL REPLACE calls to escape
%, _, and \ metacharacters at query time, matching non-parameterized
escaping semantics.

Made-with: Cursor
…aping

Update in-operator test expectations to reflect dialect-aware function
names. Add dialect compliance test for literal-on-right string
containment across all 5 dialects. Update REPL LIKE test expectations
to include REPLACE-based wildcard escaping in parameterized mode.

Made-with: Cursor
…rameterized mode

Replace innerHTML with textContent + DOM element creation in the WASM
playground's parameterized output path to prevent script injection from
crafted JSONLogic string literals.

Add placeholder-aware argument detection (isPlaceholder/isBoundValue) to
parseContainsArgs so reversed contains/!contains arguments like
{"contains": ["foo", {"var": "name"}]} are correctly identified in
parameterized mode where literals become @p1/$1 placeholders.

Made-with: Cursor
Add parseContainsArgs test cases for reversed placeholder arguments
(@p1, $1). Add integration tests verifying contains/!contains with
reversed args produce correct column LIKE pattern SQL in both
non-parameterized and parameterized modes.

Made-with: Cursor
…rized mode

When a numeric string exceeds int64 range, store it as *big.Int in the
QueryParam value instead of a plain string. Database drivers (BigQuery,
PostgreSQL, Spanner) bind *big.Int as a numeric type, avoiding runtime
type errors when the parameter appears in arithmetic or comparison
contexts (e.g. @p1 * @p2, field >= $1).

Made-with: Cursor
Update unit and integration test expectations to verify overflow integers
are stored as *big.Int. Add big.Int-aware comparison helpers
(numericParamValuesEqual, assertParams via reflect.DeepEqual) for correct
pointer-value equality checks.

Made-with: Cursor
Split buildLikeSQL into three branches: SQL string literals (inline),
bind placeholders (CONCAT+REPLACE), and unquoted primitives like numeric
or boolean values (inline as literal). Previously, unquoted primitives
fell into the placeholder branch, producing invalid SQL like
REPLACE(1000, ...) where REPLACE requires text arguments.

Made-with: Cursor
…ed mode

Verify that contains and startsWith with numeric JSON values (1000, 404)
produce valid inlined LIKE SQL rather than invalid REPLACE wrapping.

Made-with: Cursor
h22rana added 28 commits March 30, 2026 12:28
…idation

Add schemaProvider() helper that returns a true nil interface when
*Schema is nil, avoiding Go's typed-nil interface pitfall. Without
this, passing a nil *Schema pointer through config or SetSchema
creates a non-nil SchemaProvider interface, causing convertVarName
to skip the identifier regex whitelist and accept invalid names
like "bad field".

Made-with: Cursor
Switch JSON decoding from json.Unmarshal (which coerces all numbers
to float64) to json.Decoder.UseNumber, preserving numbers as
json.Number throughout the pipeline. This fixes precision loss for
integers exceeding 2^53 (e.g., 9223372036854775808 and
9223372036854775809 previously produced identical SQL).

Add json.Number handling to all operator paths: isPrimitive, isNumber,
valueToSQL, valueToSQLParam, getNumber, coerceValueForComparison,
handleIn/handleInParam, and primitiveToSQLParam.

Parameterized behavior: JS-safe integers (within ±2^53-1) bind as
float64 for backward compatibility; overflow integers bind as string
to preserve precision.

Made-with: Cursor
Add tests verifying distinct large integers produce distinct SQL and
correct parameterized string values. Update existing expectations for
json.Number rendering: 9999999999999 (no longer scientific notation),
1e10 (original JSON form), 1.0 (preserved decimal).

Made-with: Cursor
In primitiveToSQLParam, format []interface{} values using fmt.Sprintf
instead of parameterizing them as bind values. This matches the
non-param primitiveToSQL behavior and ensures custom operators (e.g.
REPL contains) receive the same "[x]" string form in both modes,
allowing parseContainsArgs to detect and unwrap array patterns.

Previously, arrays were stored as []interface{} bind values (e.g.
@p1 = [x]), causing type errors when CAST/REPLACE was applied in
parameterized LIKE expressions.

Made-with: Cursor
jsonNumberParamValue now falls back to the original numeric string
instead of erroring when float64 conversion overflows (1e309) or
underflows (1e-400). Also adds underflow detection in both
data.jsonNumberParamValue and parser.primitiveToSQLParam to prevent
silently binding non-zero values as 0. This aligns parameterized
output with inline transpilation which preserves these literals
verbatim.

Made-with: Cursor
Covers both operator path (jsonNumberParamValue) and custom operator
path (primitiveToSQLParam) with 1e309 overflow and 1e-400 underflow
cases, asserting the original string is preserved as the param value.

Made-with: Cursor
Add rows for overflow (1e309) and underflow (1e-400) floats to the
QueryParam.Value type table. Update the UseNumber note to cover
float range limits alongside integer precision.

Made-with: Cursor
…om operators

Add FindQuotedPlaceholderRef guard that detects when a custom operator
wraps a placeholder inside a SQL string literal (e.g. '@p1'), which
silently breaks bind semantics. The guard runs after custom operator
SQL generation in parameterized mode and fails with E102
ErrCustomOperatorFailed. Valid expressions like CONCAT(@p1, '%')
continue to work.

Made-with: Cursor
Unit tests for FindQuotedPlaceholderRef covering named/positional
placeholders, partial-match rejection, escaped quotes, and clean SQL.
Integration tests for parameterized custom operators: rejection of
quoted placeholders and acceptance of expression-based usage.

Made-with: Cursor
TranspileFromMap/TranspileFromInterface accepted crafted json.Number
values (e.g. json.Number("1 OR 1=1")) and inlined them verbatim,
enabling SQL injection. Add validJSONNumberLiteral regex that enforces
the JSON number grammar before emission in both data.valueToSQL and
numeric.valueToSQL, plus their parameterized counterparts. The
Transpile(string) path is not affected since json.Decoder only
produces valid numeric tokens.

Made-with: Cursor
Unit tests in data_test.go for valueToSQL and valueToSQLParam with
valid and invalid json.Number inputs. Integration tests in
transpiler_test.go for TranspileFromMap/TranspileFromInterface
rejection. All-dialect parameterized custom operator tests in
parameterized_test.go for quoted-placeholder detection.

Made-with: Cursor
… injection path

The validator's isNumber() accepted any json.Number, allowing crafted
values like json.Number("1 OR 1=1") through TranspileFromMap to reach
custom operators unvalidated. Now validates against the JSON number
grammar before accepting, catching the issue at the earliest defense
point for all code paths (built-in and custom operators).

Made-with: Cursor
…tor path

Validator tests for valid/invalid json.Number primitives and custom
operator arguments. Integration test confirming TranspileFromMap with
a custom operator rejects crafted json.Number literals.

Made-with: Cursor
…param regex

- Switch bytes.NewBufferString to strings.NewReader in decodeJSONLogic
  to avoid unnecessary byte slice allocation.
- Copy arrays before schema coercion in handleIn/handleInParam to
  prevent in-place mutation of caller-provided slices.
- Replace per-parameter regex compilation in ValidatePlaceholderRefs
  with boundary-aware string scanning, reusing hasPlaceholderBoundaries.

Made-with: Cursor
@h22rana h22rana merged commit 427edfd into master Mar 30, 2026
@h22rana h22rana deleted the feat/parameterized-query-output branch March 30, 2026 12:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant