feat(operators): add error return scenarios for custom operator null fallback by aepfli · Pull Request #342 · open-feature/flagd-testbed

aepfli · 2026-03-12T15:15:36Z

Summary

Adds @operator-errors tagged scenarios to targeting.feature covering error return behavior from flagd#1874.

New Tag

@operator-errors — SDK implementations can opt in to verify their custom operators return null (not false) on error, which triggers the default variant fallback rather than the false branch of a conditional.

New Scenarios

All three scenarios use flags where the targeting rule is a bare operator call (not wrapped in if), so a null return triggers the default variant directly.

sem_ver with unparseable version (`semver-invalid-version-flag`)

Context passes not-a-version — sem_ver can't parse it, must return null → default variant fallback.

sem_ver with unknown operator (`semver-invalid-operator-flag`)

Rule uses === which is not a valid sem_ver operator — must return null → default variant fallback.

fractional with missing bucket key (`fractional-null-bucket-key-flag`)

Context is empty — bucket key variable resolves to null — must return null → default variant fallback.

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the testing suite by adding comprehensive scenarios for custom operator error handling. The primary goal is to ensure that when custom operators encounter invalid input or conditions, they correctly return a null value, which then triggers the intended default variant fallback. This improves the robustness and predictability of flag evaluation in error conditions.

Highlights

New Error Handling Scenarios: Introduced new Gherkin scenarios tagged with @operator-errors to specifically test the error return behavior of custom operators.
Null Fallback Verification: These scenarios verify that SDK implementations correctly return null (instead of false) when custom operators encounter errors, ensuring the default variant fallback mechanism is triggered.
Specific Test Cases: Added test cases for sem_ver operator with an unparseable version, sem_ver with an invalid operator, and fractional operator with a missing bucket key, all designed to trigger null returns and default variant fallbacks.

Changelog

flags/edge-case-flags.json
- Added semver-invalid-version-flag to test sem_ver operator with unparseable version context.
- Added semver-invalid-operator-flag to test sem_ver operator with an invalid operator.
- Added fractional-null-bucket-key-flag to test fractional operator with a missing bucket key.
gherkin/targeting.feature
- Added a new @operator-errors tagged scenario outline for custom operator error returns and default variant fallback.
- Added a new @operator-errors tagged scenario for fractional operator with a missing bucket key falling back to the default variant.

Activity

No activity has been recorded for this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request adds test scenarios to verify that custom operators correctly return null on error, leading to the default variant fallback. The changes include new flag definitions in flags/edge-case-flags.json for sem_ver and fractional operator error conditions, and corresponding test scenarios in gherkin/targeting.feature. The implementation looks correct and covers the intended error cases. I've added one suggestion to improve the conciseness of a Gherkin Scenario Outline.

gherkin/targeting.feature

aepfli · 2026-03-12T21:08:35Z

relates too open-feature/flagd#1874

toddbaert · 2026-03-30T13:32:49Z

Nice work, the scenarios here look correct and well-structured. A few things from the original issue (#1874) that aren't covered yet:

starts_with / ends_with error paths (notably Go's inconsistency between the two)
fractional "no bucket matched" scenario (weight misconfiguration; distinct from the null bucket key case covered here)
fractional negative bucket weights to "clamp" to zero.
Edge cases like wrong argument count or invalid argument types

Happy to merge this as-is and add those in a follow-up.

…rn behaviour Covers the remaining cases from open-feature/flagd#1874 identified in the review of #342: - starts_with / ends_with: non-string first argument → must return null - starts_with / ends_with: wrong argument count → must return null - sem_ver: wrong argument count (missing target version) → must return null - fractional: all-zero bucket weights (no bucket matched) → must return null - fractional: negative bucket weight clamped to zero → "one" gets effective weight 0, "two" gets 100% of the weight All scenarios use a bare operator as the targeting expression (not wrapped in 'if') so a null result selects the defaultVariant directly. Scenarios are tagged @operator-errors and mirrored in both the SDK-level gherkin (gherkin/targeting.feature) and the evaluator-level gherkin files (evaluator/gherkin/{string,semver,fractional}.feature) with matching flag definitions added to both flags/edge-case-flags.json and evaluator/flags/testkit-flags.json. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Simon Schrottner <simon.schrottner@dynatrace.com>

aepfli · 2026-03-30T14:10:27Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces new test flags and Gherkin scenarios to verify error handling for semantic versioning and fractional operators, specifically covering invalid versions, invalid operators, and missing bucket keys. A review comment suggests simplifying a scenario outline in gherkin/targeting.feature by hardcoding the expected fallback value for better consistency with other tests.

gherkin/targeting.feature

aepfli · 2026-03-30T14:27:09Z

i created a follow up pr, to address your points @toddbaert

…l operators Adds @operator-errors tagged scenarios verifying that custom operator errors return null (triggering default variant fallback) rather than false (which would take the false branch of a conditional): - sem_ver with an unparseable version string - sem_ver with an unknown comparison operator - fractional with a missing/null bucket key Addresses: open-feature/flagd#1874 Signed-off-by: Simon Schrottner <simon.schrottner@dynatrace.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

… suite Adds the sem_ver invalid-version, invalid-operator and fractional null-bucket-key error scenarios from the SDK-level gherkin to the evaluator suite, with matching flag definitions in evaluator/flags/testkit-flags.json. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Simon Schrottner <simon.schrottner@dynatrace.com>

…column Addresses Gemini review feedback: since the expected value is always 'fallback', hardcode it in the Then step and remove the value column from the Examples table for clarity and consistency with the evaluator scenarios. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Simon Schrottner <simon.schrottner@dynatrace.com>

…rn behaviour Covers the remaining cases from open-feature/flagd#1874 identified in the review of #342: - starts_with / ends_with: non-string first argument → must return null - starts_with / ends_with: wrong argument count → must return null - sem_ver: wrong argument count (missing target version) → must return null - fractional: all-zero bucket weights (no bucket matched) → must return null - fractional: negative bucket weight clamped to zero → "one" gets effective weight 0, "two" gets 100% of the weight All scenarios use a bare operator as the targeting expression (not wrapped in 'if') so a null result selects the defaultVariant directly. Scenarios are tagged @operator-errors and mirrored in both the SDK-level gherkin (gherkin/targeting.feature) and the evaluator-level gherkin files (evaluator/gherkin/{string,semver,fractional}.feature) with matching flag definitions added to both flags/edge-case-flags.json and evaluator/flags/testkit-flags.json. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Simon Schrottner <simon.schrottner@dynatrace.com>

aepfli requested a review from a team as a code owner March 12, 2026 15:15

gemini-code-assist bot reviewed Mar 12, 2026

View reviewed changes

gherkin/targeting.feature Outdated Show resolved Hide resolved

aepfli force-pushed the test/operator-error-returns branch from 72a8839 to a6a7234 Compare March 12, 2026 15:19

aepfli mentioned this pull request Mar 18, 2026

feat(evaluator): port new test scenarios from PRs #341-345 into evaluator testkit #348

Closed

toddbaert approved these changes Mar 30, 2026

View reviewed changes

aepfli mentioned this pull request Mar 30, 2026

feat(operators): add follow-up error scenarios for operator null-return behaviour #356

Open

gemini-code-assist bot reviewed Mar 30, 2026

View reviewed changes

gherkin/targeting.feature Outdated Show resolved Hide resolved

aepfli changed the title ~~test(operators): add error return scenarios for custom operator null fallback~~ feat(operators): add error return scenarios for custom operator null fallback Mar 30, 2026

aepfli and others added 3 commits March 30, 2026 16:48

aepfli force-pushed the test/operator-error-returns branch from 6edba4e to e882ffb Compare March 30, 2026 14:49

aepfli merged commit eb3fa72 into main Mar 30, 2026
3 checks passed

openfeaturebot mentioned this pull request Mar 30, 2026

chore(main): release 3.5.0 #358

Open

aepfli deleted the test/operator-error-returns branch March 30, 2026 18:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(operators): add error return scenarios for custom operator null fallback#342

feat(operators): add error return scenarios for custom operator null fallback#342
aepfli merged 3 commits intomainfrom
test/operator-error-returns

aepfli commented Mar 12, 2026

Uh oh!

gemini-code-assist bot commented Mar 12, 2026

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

aepfli commented Mar 12, 2026

Uh oh!

toddbaert commented Mar 30, 2026 •

edited

Loading

Uh oh!

aepfli commented Mar 30, 2026

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

aepfli commented Mar 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

aepfli commented Mar 12, 2026

Summary

New Tag

New Scenarios

sem_ver with unparseable version (semver-invalid-version-flag)

sem_ver with unknown operator (semver-invalid-operator-flag)

fractional with missing bucket key (fractional-null-bucket-key-flag)

Related

Uh oh!

gemini-code-assist bot commented Mar 12, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

aepfli commented Mar 12, 2026

Uh oh!

toddbaert commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aepfli commented Mar 30, 2026

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

aepfli commented Mar 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sem_ver with unparseable version (`semver-invalid-version-flag`)

sem_ver with unknown operator (`semver-invalid-operator-flag`)

fractional with missing bucket key (`fractional-null-bucket-key-flag`)

toddbaert commented Mar 30, 2026 •

edited

Loading