Skip to content

feat: Support Experiments (A/B tests) on Swetrix CE#528

Merged
Blaumaus merged 8 commits intomainfrom
feature/swetrix-ce-experiments
May 6, 2026
Merged

feat: Support Experiments (A/B tests) on Swetrix CE#528
Blaumaus merged 8 commits intomainfrom
feature/swetrix-ce-experiments

Conversation

@Blaumaus
Copy link
Copy Markdown
Member

@Blaumaus Blaumaus commented May 5, 2026

Changes

If applicable, please describe what changes were made in this pull request.

Community Edition support

  • Your feature is implemented for the Swetrix Community Edition
  • This PR only updates the Cloud (Enterprise) Edition code (e.g. Paddle webhooks, blog, payouts, etc.)

Database migrations

  • Clickhouse / MySQL migrations added for this PR
  • No table schemas changed in this PR

Documentation

  • You have updated the documentation according to your PR
  • This PR did not change any publicly documented endpoints

Summary by CodeRabbit

  • New Features

    • Experiments now available in all editions with full management (create, update, start/pause/complete, delete).
    • Experiment results include Bayesian win probabilities, confidence flags, and optional time-bucket charts.
    • Experiments can be linked to feature flags and surfaced in flag evaluations with variant assignment and exposure tracking.
  • Documentation

    • README feature list clarified: "Experiments" entry and clearer Cloud vs Community wording.
  • Chores

    • Database schema and migrations added to store experiments and exposures.

@Blaumaus Blaumaus self-assigned this May 5, 2026
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 5, 2026

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 5fe201a6-82ce-43fc-866a-52bfeb951782

📥 Commits

Reviewing files that changed from the base of the PR and between 8053fad and 3252e93.

⛔ Files ignored due to path filters (1)
  • docs/package-lock.json is excluded by !**/package-lock.json
📒 Files selected for processing (9)
  • README.md
  • backend/apps/community/src/experiment/bayesian.ts
  • backend/apps/community/src/experiment/dto/experiment.dto.ts
  • backend/apps/community/src/experiment/experiment.controller.ts
  • backend/apps/community/src/experiment/experiment.service.ts
  • backend/apps/community/src/feature-flag/dto/feature-flag.dto.ts
  • backend/apps/community/src/feature-flag/feature-flag.controller.ts
  • docs/package.json
  • web/app/pages/Project/View/ViewProject.tsx

📝 Walkthrough

Walkthrough

Adds a full A/B experimentation feature: data model and ClickHouse tables, REST APIs and DTOs, deterministic variant assignment, Bayesian result computation, feature-flag integration, exposure tracking for evaluations and custom events, module wiring, migrations, and UI tab visibility for Experiments.

Changes

A/B Experimentation System

Layer / File(s) Summary
Data Shape & Schema
backend/apps/community/src/experiment/entity/*, backend/apps/community/src/feature-flag/entity/*, backend/migrations/clickhouse/*, web/app/api/api.server.ts
New Experiment/ExperimentVariant types and enums; FeatureFlag gains experimentId; ClickHouse DDL adds experiment, experiment_variant, experiment_exposures tables and experimentId column on feature_flag; web API types updated.
API Contracts & Validation
backend/apps/community/src/experiment/dto/experiment.dto.ts
New DTOs: CreateExperimentDto, UpdateExperimentDto, ExperimentDto, VariantResultDto, ExperimentResultsDto with class-validator rules and Swagger decorators.
Core Service & Persistence
backend/apps/community/src/experiment/experiment.service.ts
ExperimentService added: CRUD, pagination/search, formatting between ClickHouse and domain objects, bulk variant hydration, running-experiment queries, and variant insert/update helpers.
Controller & Lifecycle
backend/apps/community/src/experiment/experiment.controller.ts
ExperimentController endpoints for list/get/create/update/delete, start/pause/complete lifecycle, results endpoint that queries ClickHouse and computes Bayesian results, chart generation and helper methods for attribution and validation.
Statistical Analysis
backend/apps/community/src/experiment/bayesian.ts
calculateBayesianProbabilities added: deterministic, seeded Monte Carlo Beta sampling to compute per-variant win probabilities, with edge-case handling and PRNG seeding.
Variant Assignment & Evaluation
backend/apps/community/src/feature-flag/evaluation.ts, backend/apps/community/src/feature-flag/feature-flag.controller.ts, backend/apps/community/src/feature-flag/feature-flag.service.ts
getExperimentVariant added for deterministic variant selection (SHA-256 + rollout bucketing). Feature flag evaluation fetches running experiments, assigns variants, extends evaluation response with experiments and experimentsByFlag, and persists experiment_exposures; service supports experimentId read/write.
Analytics / Exposure Tracking
backend/apps/community/src/analytics/analytics.controller.ts, backend/apps/community/src/analytics/analytics.module.ts
AnalyticsController.logCustom now asynchronously tracks exposures for custom-event experiments via trackCustomEventExperimentExposures; AnalyticsModule imports ExperimentModule.
Module Wiring
backend/apps/community/src/experiment/experiment.module.ts, backend/apps/community/src/app.module.ts, backend/apps/community/src/feature-flag/feature-flag.module.ts, backend/apps/community/src/goal/goal.module.ts
New ExperimentModule registered and exported; modules wired with forwardRef to resolve circular deps; ExperimentModule added to AppModule.
Frontend / Docs
web/app/pages/Project/View/ViewProject.tsx, README.md, docs/package.json
Project view tab logic updated to always include Experiments tab; README feature list updated to surface "Experiments" and clarify CE column; docs package.json deps updated.

Sequence Diagram

sequenceDiagram
    participant Client
    participant ExperimentController
    participant ExperimentService
    participant ClickHouse as ClickHouse (events, experiment, experiment_exposures)
    participant Bayesian as calculateBayesianProbabilities

    Client->>ExperimentController: GET /experiment/:id/results
    ExperimentController->>ExperimentService: findOneWithRelations(id)
    ExperimentService->>ClickHouse: SELECT experiment, variants
    ClickHouse-->>ExperimentService: experiment + variants
    ExperimentService-->>ExperimentController: Experiment{variants}

    ExperimentController->>ClickHouse: Query exposures (experiment_exposures)
    ClickHouse-->>ExperimentController: exposures per variant

    alt goal exists
        ExperimentController->>ClickHouse: Query conversions (events)
        ClickHouse-->>ExperimentController: conversions per variant
    end

    ExperimentController->>Bayesian: calculateBayesianProbabilities(variants)
    Bayesian->>Bayesian: seeded Monte Carlo Beta sampling
    Bayesian-->>ExperimentController: variant win probabilities

    ExperimentController-->>Client: ExperimentResultsDto (variants, winner, chart?)
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

  • Swetrix/swetrix#528: Overlapping experiment backend changes (controllers, service, migrations, evaluation/exposure tracking).
  • Swetrix/swetrix#527: Adds analytics → experiment wiring and exposure tracking for custom events.
  • Swetrix/swetrix#439: Prior experiments feature work touching Bayesian logic, ClickHouse tables, and feature-flag integration.

Poem

🐇 I hopped through code and dug a burrow deep,

Seeding variants while the logs did keep,
Bayesian dreams in Monte Carlo light,
Flags decide which path each profile might,
Exposures tracked, results in sight—hip hop, delight!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely summarizes the main feature addition: supporting A/B tests (Experiments) for the Community Edition, which aligns with the comprehensive changeset.
Description check ✅ Passed The PR description follows the template structure with all key sections addressed: changes mentioned, Community Edition support confirmed, ClickHouse migrations added, and documentation updated.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feature/swetrix-ce-experiments

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🧹 Nitpick comments (9)
backend/apps/community/src/experiment/bayesian.ts (1)

40-72: 💤 Low value

Optional: dead branch for shape < 1.

Given that calculateBayesianProbabilities always calls sampleGamma with alpha = conversions + 1 ≥ 1 and beta = Math.max(1, ...) ≥ 1, the recursive boost branch on lines 69–71 is unreachable in practice. You can either drop it for clarity or keep it as defensive future-proofing in case the function is reused with shape < 1 callers.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/apps/community/src/experiment/bayesian.ts` around lines 40 - 72, The
recursive branch handling shape < 1 in sampleGamma is effectively dead because
calculateBayesianProbabilities only calls sampleGamma with alpha and beta >= 1;
remove the lower-than-1 branch (the recursive call and subsequent acceptance
using Math.pow(u, 1 / shape)) and either add a short guard/assert at the top of
sampleGamma (e.g., throw or console.assert if shape < 1) or a comment stating
the function expects shape >= 1; keep references to sampleGamma and
calculateBayesianProbabilities so readers can see the invariant.
backend/apps/community/src/experiment/experiment.module.ts (1)

10-22: ⚖️ Poor tradeoff

LGTM, with a note on the cycle density.

Four of five imports use forwardRef, indicating the experiment/analytics/feature-flag/goal/project quartet has dense bidirectional dependencies. This works, but if the dependency graph keeps growing it may be worth extracting a small "shared types/utilities" module (e.g., ExperimentVariantSelectionService) to break some cycles. Not blocking for this PR.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/apps/community/src/experiment/experiment.module.ts` around lines 10 -
22, The module currently relies on forwardRef for ProjectModule,
AnalyticsModule, GoalModule and FeatureFlagModule indicating cyclic
dependencies; break the cycle by extracting the shared logic into a small module
(e.g., create ExperimentVariantSelectionService inside a new
ExperimentSharedModule), move the variant selection/util functions from
ExperimentService into that service, export ExperimentVariantSelectionService
from ExperimentSharedModule and have ExperimentModule, AnalyticsModule,
GoalModule, FeatureFlagModule and ProjectModule import ExperimentSharedModule
instead of referencing each other via forwardRef; update ExperimentModule to
remove forwardRef imports where possible and inject
ExperimentVariantSelectionService into ExperimentService (and any other modules
that need it) to eliminate the dense bidirectional dependency graph.
backend/apps/community/src/feature-flag/feature-flag.controller.ts (3)

316-348: 💤 Low value

Variant assignment flow looks correct, minor consistency note.

Variants from experimentService are already ordered by key ASC in ClickHouse (in getVariantsForExperiments), and you re-sort here with localeCompare. For ASCII-only variant keys (the typical case) both orderings match, but localeCompare is locale-sensitive while ClickHouse uses byte order. Since getExperimentVariant is deterministic for any fixed input order, this is fine — but it'd be slightly cleaner to rely on the service-side ordering and drop the re-sort, or lock the ordering down to one canonical form so future refactors don't shift assignments.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/apps/community/src/feature-flag/feature-flag.controller.ts` around
lines 316 - 348, The code re-sorts experiment.variants with localeCompare before
calling getExperimentVariant which duplicates and can introduce locale-dependent
ordering differences versus the service-side (getVariantsForExperiments)
byte-ordering; remove the re-sort and pass the variants from experiments
directly (or explicitly sort using a byte/ASCII comparator if you prefer) so the
ordering is canonical and deterministic—update the block around
experiment.variants, sortedVariants, and the getExperimentVariant call in
feature-flag.controller.ts (references: experimentService.findRunningByIds,
experiment.variants, getExperimentVariant).

408-439: ⚡ Quick win

Consider async_insert: 1 for both inserts on this hot path.

POST /feature-flag/evaluate is a public, high-throughput endpoint, and trackEvaluations performs two sequential awaited ClickHouse inserts (feature_flag_evaluations then experiment_exposures). Other ingest paths in this codebase (e.g., analytics.controller.ts experiment_exposures insert at line 1243-1255 and the events insert at line 1167-1173) use clickhouse_settings: { async_insert: 1 } for similar fire-and-forget ingestion. Aligning here would reduce p99 on the evaluate path and keep ingestion behavior consistent.

♻️ Suggested change
       await clickhouse.insert({
         table: 'feature_flag_evaluations',
         values,
         format: 'JSONEachRow',
+        clickhouse_settings: { async_insert: 1 },
       })
@@
         await clickhouse.insert({
           table: 'experiment_exposures',
           values: experimentExposures,
           format: 'JSONEachRow',
+          clickhouse_settings: { async_insert: 1 },
         })

Based on learnings from past feedback that clickhouse_settings: { async_insert: 1 } is the accepted pattern for high-throughput, fire-and-forget ingestion in this codebase.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/apps/community/src/feature-flag/feature-flag.controller.ts` around
lines 408 - 439, Add ClickHouse async insert settings to both insert calls to
make these fire-and-forget like other ingest paths: when calling
clickhouse.insert for table 'feature_flag_evaluations' (using variable values)
and for table 'experiment_exposures' (using experimentExposures), pass
clickhouse_settings: { async_insert: 1 } in the options object (keep the
existing try/catch logging behavior unchanged).

360-385: 💤 Low value

Consider not mixing two key namespaces in one map.

experimentsByIdOrFlagKey indexes the same value by both experiment id and feature-flag key. This works in practice (flag keys are unlikely to collide with UUIDs), but it makes the response shape harder to type, harder to evolve, and forces SDK consumers to know which lookup convention to use. Splitting into two records (e.g., experiments keyed by id, experimentsByFlag keyed by flag key) would be unambiguous and easier to document.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/apps/community/src/feature-flag/feature-flag.controller.ts` around
lines 360 - 385, Currently the code mixes experiment IDs and flag keys in one
map (experimentsByIdOrFlagKey) which is confusing; change the response shape to
have two separate optional records (e.g., response.experiments:
Record<string,string> keyed by experimentId and response.experimentsByFlag:
Record<string,string> keyed by flag key), update the response type declaration
to include both, then in the loop over experimentVariants assign
experiments[experimentId] = variantKey and for each linkedFlag assign
experimentsByFlag[linkedFlag.key] = variantKey (instead of writing both into one
map), and finally set response.experiments and response.experimentsByFlag before
returning.
backend/apps/community/src/experiment/experiment.controller.ts (3)

416-491: 💤 Low value

Resuming a paused experiment overwrites the original startedAt.

The start handler unconditionally sets startedAt: dayString() (line 486), so resuming a PAUSED experiment loses the original start timestamp. Since results queries use groupFrom/groupTo from the request rather than startedAt, the data side is fine, but downstream views/reporting that show "experiment running since X" will report the resume time instead. Consider preserving startedAt if it's already set:

♻️ Suggested change
     const updatedExperiment = await this.experimentService.update(id, {
       status: ExperimentStatus.RUNNING,
-      startedAt: dayString(),
+      ...(experiment.startedAt ? {} : { startedAt: dayString() }),
       featureFlagId,
     })
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/apps/community/src/experiment/experiment.controller.ts` around lines
416 - 491, The handler currently always sets startedAt: dayString() when calling
this.experimentService.update (see variable updatedExperiment and the update
call in experiment.controller.ts), which overwrites the original start time when
resuming a PAUSED experiment; change the payload to preserve the existing
experiment.startedAt if present (e.g. set startedAt to experiment.startedAt ??
dayString()) so startedAt is only set to now for truly new starts and not when
resuming.

1063-1073: ⚡ Quick win

Dead branch in variantSelector — both enum values lead to the same expression.

MultipleVariantHandling only has FIRST_EXPOSURE and EXCLUDE, so the ternary's else branch ('any(variantKey)') is unreachable. This is misleading and brittle: if a third enum value is added, it would silently fall through to a different aggregation. Either simplify to always use argMin (current effective behavior), or — if the original intent was to use any for EXCLUDE (since HAVING uniqExact(variantKey) = 1 guarantees a single variant per profile, making any cheaper) — restore that explicitly.

♻️ Suggested change (simplification)
-    const variantSelector =
-      experiment.multipleVariantHandling ===
-        MultipleVariantHandling.FIRST_EXPOSURE ||
-      experiment.multipleVariantHandling === MultipleVariantHandling.EXCLUDE
-        ? 'argMin(variantKey, tuple(created, variantKey))'
-        : 'any(variantKey)'
+    const variantSelector =
+      experiment.multipleVariantHandling === MultipleVariantHandling.FIRST_EXPOSURE
+        ? 'argMin(variantKey, tuple(created, variantKey))'
+        : 'any(variantKey)'
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/apps/community/src/experiment/experiment.controller.ts` around lines
1063 - 1073, The ternary in getExposureAttributionSubquery creates a dead
else-branch because MultipleVariantHandling only has FIRST_EXPOSURE and EXCLUDE
and both currently resolve to argMin; simplify variantSelector to always use
'argMin(variantKey, tuple(created, variantKey))' (remove the unreachable
'any(variantKey)' branch) while leaving multiVariantFilter behavior for
MultipleVariantHandling.EXCLUDE ('HAVING uniqExact(variantKey) = 1') intact so
the function's logic (variantSelector and multiVariantFilter) is clear and not
brittle; update references in getExposureAttributionSubquery and remove the
unnecessary enum branch checks.

865-872: 💤 Low value

Strict float equality on rollout sum can reject legitimate inputs.

totalPercentage !== 100 will reject any decimal split that incurs floating-point drift (e.g., 33.3 + 33.3 + 33.4 works, but combinations summing through values like 0.1 + 0.2 won't). If the API accepts non-integer rollouts, prefer an epsilon-based comparison; if the contract is integers-only, validate that explicitly per-variant.

♻️ Suggested change
-    const totalPercentage = _sum(
-      variants.map((variant) => variant.rolloutPercentage),
-    )
-    if (totalPercentage !== 100) {
+    const totalPercentage = _sum(
+      variants.map((variant) => variant.rolloutPercentage),
+    )
+    if (Math.abs(totalPercentage - 100) > 0.01) {
       throw new BadRequestException(
         'Variant rollout percentages must sum to 100',
       )
     }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/apps/community/src/experiment/experiment.controller.ts` around lines
865 - 872, The current strict equality check using _sum over variants'
rolloutPercentage (variants.map(...)) can fail due to floating-point drift;
update the validation in the block that computes totalPercentage to either (a)
enforce per-variant integer rollouts by validating each
variant.rolloutPercentage is an integer before summing, or (b) use an
epsilon-based comparison (e.g., Math.abs(totalPercentage - 100) <= EPSILON)
where EPSILON is a small constant (e.g., 1e-6) and include that in the
BadRequestException path; reference the existing identifiers _sum, variants,
rolloutPercentage and the thrown BadRequestException when making the change.
backend/apps/community/src/experiment/experiment.service.ts (1)

301-329: ⚡ Quick win

Consider scoping findRunningByIds by projectId for defense in depth.

The query trusts that experimentIds is already project-scoped (true today: callers derive IDs from project-scoped flags). However, adding a projectId filter would make the method safe regardless of caller, and prevents cross-tenant data exposure if a future caller passes mixed IDs. Same recommendation could apply to findOne/findOneWithRelations for view/manage paths, though those are followed by an allowedToView/allowedToManage check.

♻️ Suggested signature change
   async findRunningByIds(
     experimentIds: Array<string | null>,
     exposureTrigger: ExposureTrigger,
+    projectId?: string,
   ): Promise<Experiment[]> {
     const ids = experimentIds.filter((id): id is string => Boolean(id))

     if (_isEmpty(ids)) {
       return []
     }

     const { data } = await clickhouse
       .query({
         query: `
           SELECT *
           FROM experiment
           WHERE id IN {ids:Array(String)}
             AND status = {status:String}
             AND exposureTrigger = {exposureTrigger:String}
+            ${projectId ? 'AND projectId = {projectId:FixedString(12)}' : ''}
         `,
         query_params: {
           ids,
           status: ExperimentStatus.RUNNING,
           exposureTrigger,
+          ...(projectId ? { projectId } : {}),
         },
       })
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/apps/community/src/experiment/experiment.service.ts` around lines 301
- 329, Add a projectId filter to findRunningByIds to prevent cross-tenant
exposure: update the method signature (findRunningByIds) to accept a projectId
(string), add a WHERE clause AND projectId = {projectId:String} to the
ClickHouse query and include projectId in query_params, and ensure callers pass
the caller's projectId (or derive it) when invoking findRunningByIds; similarly
consider adding optional projectId filtering to findOne/findOneWithRelations
(and update their callers) so those methods can be scoped by projectId as an
additional defense-in-depth measure.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@backend/apps/community/src/experiment/dto/experiment.dto.ts`:
- Around line 133-141: Add a minimum-size validator to the variants arrays so
DTO validation rejects too-small experiments: add `@ArrayMinSize`(2) (or
`@ArrayMinSize`(1) if rollout-only experiments are allowed) above the variants
property in ExperimentDto (the variants: ExperimentVariantDto[] declaration) and
likewise add the same `@ArrayMinSize`(...) to UpdateExperimentDto. Also import
ArrayMinSize from class-validator and keep the existing `@ArrayMaxSize`(20),
`@IsArray`(), `@ValidateNested`({ each: true }) and `@Type`(() =>
ExperimentVariantDto) decorators.

In `@backend/apps/community/src/experiment/experiment.controller.ts`:
- Around line 678-693: In the conversionsQuery (and the other query occurrence
that uses assumeNotNull(c.profileId)), replace the unsafe
assumeNotNull(c.profileId) usage in the JOIN with an explicit null check and a
proper equality comparison: change the JOIN clause so it requires c.profileId IS
NOT NULL and then compares e.profileId = c.profileId (instead of e.profileId =
assumeNotNull(c.profileId)); update the same pattern wherever
assumeNotNull(c.profileId) appears (e.g., the other query around the
exposureAttributionSubquery usage) so nullable profileId is handled safely.

In `@backend/migrations/clickhouse/initialise_selfhosted.js`:
- Around line 97-132: The self-hosted initializer is missing the
experiment_exposures table so fresh installs fail on exposure inserts; in
initialise_selfhosted.js add a CREATE TABLE IF NOT EXISTS
${dbName}.experiment_exposures entry (matching the schema used in
initialise_database.js/cloud initializer) alongside the existing experiment and
experiment_variant CREATE statements so the experiment_exposures table is
created during clickhouse:initialise; ensure the new table uses the same
columns, ENGINE, ORDER BY and PARTITION BY as the cloud version to keep schemas
consistent.

In `@README.md`:
- Line 60: The README has inconsistent availability for Experiments: the feature
bullet "- **Experiments**: run A/B tests and experiments to optimize your site."
is written as generally available, while the Cloud vs CE matrix row for
"Experiments" marks CE as not included; pick the correct truth and make both
places match. Update either the bullet line (the "- **Experiments**" sentence)
to indicate "Cloud only" or similar if CE does not include it, or update the
Cloud vs CE matrix row to mark CE as included if Experiments is now available in
CE; ensure you change both the bullet text and the matrix cell for "Experiments"
so they are identical.

In `@web/app/pages/Project/View/ViewProject.tsx`:
- Around line 778-797: PROJECT_TABS.experiments can be undefined in self-hosted
builds, causing experimentsTab to have id: undefined and break tab logic; update
the code so the experiments tab is only added when a valid id exists or ensure
SELFHOSTED_PROJECT_TABS defines experiments. Concretely, either add an
experiments entry to SELFHOSTED_PROJECT_TABS in web/app/lib/constants/index.ts
or change ViewProject.tsx where experimentsTab is created/added (symbols:
experimentsTab, PROJECT_TABS.experiments, isSelfhosted, newTabs) to guard the
tab creation/append by checking PROJECT_TABS.experiments !== undefined (or
filter out tabs with falsy id) before building newTabs so no tab with id:
undefined is produced.

---

Nitpick comments:
In `@backend/apps/community/src/experiment/bayesian.ts`:
- Around line 40-72: The recursive branch handling shape < 1 in sampleGamma is
effectively dead because calculateBayesianProbabilities only calls sampleGamma
with alpha and beta >= 1; remove the lower-than-1 branch (the recursive call and
subsequent acceptance using Math.pow(u, 1 / shape)) and either add a short
guard/assert at the top of sampleGamma (e.g., throw or console.assert if shape <
1) or a comment stating the function expects shape >= 1; keep references to
sampleGamma and calculateBayesianProbabilities so readers can see the invariant.

In `@backend/apps/community/src/experiment/experiment.controller.ts`:
- Around line 416-491: The handler currently always sets startedAt: dayString()
when calling this.experimentService.update (see variable updatedExperiment and
the update call in experiment.controller.ts), which overwrites the original
start time when resuming a PAUSED experiment; change the payload to preserve the
existing experiment.startedAt if present (e.g. set startedAt to
experiment.startedAt ?? dayString()) so startedAt is only set to now for truly
new starts and not when resuming.
- Around line 1063-1073: The ternary in getExposureAttributionSubquery creates a
dead else-branch because MultipleVariantHandling only has FIRST_EXPOSURE and
EXCLUDE and both currently resolve to argMin; simplify variantSelector to always
use 'argMin(variantKey, tuple(created, variantKey))' (remove the unreachable
'any(variantKey)' branch) while leaving multiVariantFilter behavior for
MultipleVariantHandling.EXCLUDE ('HAVING uniqExact(variantKey) = 1') intact so
the function's logic (variantSelector and multiVariantFilter) is clear and not
brittle; update references in getExposureAttributionSubquery and remove the
unnecessary enum branch checks.
- Around line 865-872: The current strict equality check using _sum over
variants' rolloutPercentage (variants.map(...)) can fail due to floating-point
drift; update the validation in the block that computes totalPercentage to
either (a) enforce per-variant integer rollouts by validating each
variant.rolloutPercentage is an integer before summing, or (b) use an
epsilon-based comparison (e.g., Math.abs(totalPercentage - 100) <= EPSILON)
where EPSILON is a small constant (e.g., 1e-6) and include that in the
BadRequestException path; reference the existing identifiers _sum, variants,
rolloutPercentage and the thrown BadRequestException when making the change.

In `@backend/apps/community/src/experiment/experiment.module.ts`:
- Around line 10-22: The module currently relies on forwardRef for
ProjectModule, AnalyticsModule, GoalModule and FeatureFlagModule indicating
cyclic dependencies; break the cycle by extracting the shared logic into a small
module (e.g., create ExperimentVariantSelectionService inside a new
ExperimentSharedModule), move the variant selection/util functions from
ExperimentService into that service, export ExperimentVariantSelectionService
from ExperimentSharedModule and have ExperimentModule, AnalyticsModule,
GoalModule, FeatureFlagModule and ProjectModule import ExperimentSharedModule
instead of referencing each other via forwardRef; update ExperimentModule to
remove forwardRef imports where possible and inject
ExperimentVariantSelectionService into ExperimentService (and any other modules
that need it) to eliminate the dense bidirectional dependency graph.

In `@backend/apps/community/src/experiment/experiment.service.ts`:
- Around line 301-329: Add a projectId filter to findRunningByIds to prevent
cross-tenant exposure: update the method signature (findRunningByIds) to accept
a projectId (string), add a WHERE clause AND projectId = {projectId:String} to
the ClickHouse query and include projectId in query_params, and ensure callers
pass the caller's projectId (or derive it) when invoking findRunningByIds;
similarly consider adding optional projectId filtering to
findOne/findOneWithRelations (and update their callers) so those methods can be
scoped by projectId as an additional defense-in-depth measure.

In `@backend/apps/community/src/feature-flag/feature-flag.controller.ts`:
- Around line 316-348: The code re-sorts experiment.variants with localeCompare
before calling getExperimentVariant which duplicates and can introduce
locale-dependent ordering differences versus the service-side
(getVariantsForExperiments) byte-ordering; remove the re-sort and pass the
variants from experiments directly (or explicitly sort using a byte/ASCII
comparator if you prefer) so the ordering is canonical and deterministic—update
the block around experiment.variants, sortedVariants, and the
getExperimentVariant call in feature-flag.controller.ts (references:
experimentService.findRunningByIds, experiment.variants, getExperimentVariant).
- Around line 408-439: Add ClickHouse async insert settings to both insert calls
to make these fire-and-forget like other ingest paths: when calling
clickhouse.insert for table 'feature_flag_evaluations' (using variable values)
and for table 'experiment_exposures' (using experimentExposures), pass
clickhouse_settings: { async_insert: 1 } in the options object (keep the
existing try/catch logging behavior unchanged).
- Around line 360-385: Currently the code mixes experiment IDs and flag keys in
one map (experimentsByIdOrFlagKey) which is confusing; change the response shape
to have two separate optional records (e.g., response.experiments:
Record<string,string> keyed by experimentId and response.experimentsByFlag:
Record<string,string> keyed by flag key), update the response type declaration
to include both, then in the loop over experimentVariants assign
experiments[experimentId] = variantKey and for each linkedFlag assign
experimentsByFlag[linkedFlag.key] = variantKey (instead of writing both into one
map), and finally set response.experiments and response.experimentsByFlag before
returning.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: bfebf7ef-921e-42c9-8cc7-b5ad19a9e560

📥 Commits

Reviewing files that changed from the base of the PR and between 6b212e7 and 8053fad.

📒 Files selected for processing (22)
  • README.md
  • backend/apps/community/src/analytics/analytics.controller.ts
  • backend/apps/community/src/analytics/analytics.module.ts
  • backend/apps/community/src/app.module.ts
  • backend/apps/community/src/experiment/bayesian.ts
  • backend/apps/community/src/experiment/dto/experiment.dto.ts
  • backend/apps/community/src/experiment/entity/experiment-variant.entity.ts
  • backend/apps/community/src/experiment/entity/experiment.entity.ts
  • backend/apps/community/src/experiment/experiment.controller.ts
  • backend/apps/community/src/experiment/experiment.module.ts
  • backend/apps/community/src/experiment/experiment.service.ts
  • backend/apps/community/src/feature-flag/dto/feature-flag.dto.ts
  • backend/apps/community/src/feature-flag/entity/feature-flag.entity.ts
  • backend/apps/community/src/feature-flag/evaluation.ts
  • backend/apps/community/src/feature-flag/feature-flag.controller.ts
  • backend/apps/community/src/feature-flag/feature-flag.module.ts
  • backend/apps/community/src/feature-flag/feature-flag.service.ts
  • backend/apps/community/src/goal/goal.module.ts
  • backend/migrations/clickhouse/initialise_selfhosted.js
  • backend/migrations/clickhouse/selfhosted_2026_05_05_experiments.js
  • web/app/api/api.server.ts
  • web/app/pages/Project/View/ViewProject.tsx

Comment thread backend/apps/community/src/experiment/dto/experiment.dto.ts
Comment thread backend/apps/community/src/experiment/experiment.controller.ts
Comment thread backend/migrations/clickhouse/initialise_selfhosted.js
Comment thread README.md Outdated
Comment thread web/app/pages/Project/View/ViewProject.tsx Outdated
@Blaumaus Blaumaus merged commit 11d0564 into main May 6, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant