Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 24 additions & 8 deletions packages/enricher/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,13 @@ Detect and enrich PostHog SDK usage in source code. Uses tree-sitter AST analysi
import { PostHogEnricher } from "@posthog/enricher";

const enricher = new PostHogEnricher();
await enricher.initialize("/path/to/grammars");

// Parse from source string
const result = await enricher.parse(sourceCode, "typescript");

// Or parse from file (auto-detects language from extension)
const result = await enricher.parseFile("/path/to/app.tsx");

result.events; // [{ name: "purchase", line: 5, dynamic: false }]
result.flagChecks; // [{ method: "getFeatureFlag", flagKey: "new-checkout", line: 8 }]
result.flagKeys; // ["new-checkout"]
Expand All @@ -32,12 +35,12 @@ const enriched = await result.enrichFromApi({
});

// Flags with staleness, rollout, experiment info
enriched.enrichedFlags;
enriched.flags;
// [{ flagKey: "new-checkout", flagType: "boolean", staleness: "fully_rolled_out",
// rollout: 100, experiment: { name: "Checkout v2", ... }, ... }]

// Events with definition, volume, unique users
enriched.enrichedEvents;
enriched.events;
// [{ eventName: "purchase", verified: true, lastSeenAt: "2025-04-01",
// tags: ["revenue"], stats: { volume: 12500, uniqueUsers: 3200 }, ... }]

Expand Down Expand Up @@ -75,8 +78,8 @@ Main entry point. Owns the tree-sitter parser lifecycle.

```typescript
const enricher = new PostHogEnricher();
await enricher.initialize(wasmDir);
const result = await enricher.parse(source, languageId);
const result = await enricher.parseFile("/path/to/file.ts");
enricher.dispose();
```

Expand All @@ -98,14 +101,26 @@ Returned by `enricher.parse()`. Contains all detected PostHog SDK usage.
| `toList()` | `ListItem[]` | Flat sorted list of all SDK usage |
| `enrichFromApi(config)` | `Promise<EnrichedResult>` | Fetch from PostHog API and enrich |

### `PostHogEnricher` methods

| Method | Description |
|---|---|
| `constructor()` | Create enricher. Bundled grammars are auto-located at runtime. |
| `parse(source, languageId)` | Parse a source code string with an explicit language ID |
| `parseFile(filePath)` | Read a file and parse it, auto-detecting language from the file extension |
| `isSupported(langId)` | Check if a language ID is supported |
| `supportedLanguages` | List of supported language IDs |
| `updateConfig(config)` | Customize detection behavior |
| `dispose()` | Clean up parser resources |

### `EnrichedResult`

Returned by `enrich()` or `enrichFromApi()`. Detection combined with PostHog context.

| Property / Method | Type | Description |
|---|---|---|
| `enrichedFlags` | `EnrichedFlag[]` | Flags grouped by key with type, staleness, rollout, experiment |
| `enrichedEvents` | `EnrichedEvent[]` | Events grouped by name with definition, stats, tags |
| `flags` | `EnrichedFlag[]` | Flags grouped by key with type, staleness, rollout, experiment |
| `events` | `EnrichedEvent[]` | Events grouped by name with definition, stats, tags |
| `toList()` | `EnrichedListItem[]` | Flat list with all metadata |
| `toComments()` | `string` | Source code with inline annotation comments |

Expand Down Expand Up @@ -156,7 +171,6 @@ The lower-level detection API is also exported for direct use (this is the same
import { PostHogDetector } from "@posthog/enricher";

const detector = new PostHogDetector();
await detector.initialize(wasmDir);

const calls = await detector.findPostHogCalls(source, "typescript");
const initCalls = await detector.findInitCalls(source, "typescript");
Expand Down Expand Up @@ -188,4 +202,6 @@ setLogger({ warn: console.warn });

## Setup

The package requires pre-built tree-sitter WASM grammar files. Run `pnpm fetch-grammars` to build them, or place pre-built `.wasm` files in the `grammars/` directory.
Grammar files are bundled with the package and auto-located at runtime — no manual setup needed.

For development, run `pnpm fetch-grammars` to rebuild the WASM grammar files in the `grammars/` directory.
7 changes: 0 additions & 7 deletions packages/enricher/src/comment-formatter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -56,16 +56,9 @@ export function formatComments(
const sorted = [...items].sort((a, b) => a.line - b.line);

let offset = 0;
// One comment per original source line — if multiple detections share a line,
// only the first (by sort order) gets an annotation to keep output readable.
const annotatedLines = new Set<number>();

for (const item of sorted) {
const targetLine = item.line + offset;
if (annotatedLines.has(item.line)) {
continue;
}
annotatedLines.add(item.line);

let comment: string | null = null;

Expand Down
3 changes: 1 addition & 2 deletions packages/enricher/src/detector.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,8 @@ function simpleInits(inits: PostHogInitCall[]) {
describeWithGrammars("PostHogDetector", () => {
let detector: PostHogDetector;

beforeAll(async () => {
beforeAll(() => {
detector = new PostHogDetector();
await detector.initialize(GRAMMARS_DIR);
detector.updateConfig({
additionalClientNames: [],
additionalFlagFunctions: [
Expand Down
4 changes: 0 additions & 4 deletions packages/enricher/src/detector.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,6 @@ export class PostHogDetector {
this.pm.updateConfig(config);
}

async initialize(wasmDir: string): Promise<void> {
return this.pm.initialize(wasmDir);
}

isSupported(langId: string): boolean {
return this.pm.isSupported(langId);
}
Expand Down
12 changes: 6 additions & 6 deletions packages/enricher/src/enriched-result.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ export class EnrichedResult {
this.context = context;
}

get enrichedFlags(): EnrichedFlag[] {
get flags(): EnrichedFlag[] {
if (this.cachedFlags) {
return this.cachedFlags;
}
Expand Down Expand Up @@ -63,7 +63,7 @@ export class EnrichedResult {
return this.cachedFlags;
}

get enrichedEvents(): EnrichedEvent[] {
get events(): EnrichedEvent[] {
if (this.cachedEvents) {
return this.cachedEvents;
}
Expand Down Expand Up @@ -102,12 +102,12 @@ export class EnrichedResult {
const _experiments = this.context.experiments ?? [];

const flagLookup = new Map<string, EnrichedFlag>();
for (const f of this.enrichedFlags) {
for (const f of this.flags) {
flagLookup.set(f.flagKey, f);
}

const eventLookup = new Map<string, EnrichedEvent>();
for (const e of this.enrichedEvents) {
for (const e of this.events) {
eventLookup.set(e.eventName, e);
}

Expand Down Expand Up @@ -145,12 +145,12 @@ export class EnrichedResult {

toComments(): string {
const flagLookup = new Map<string, EnrichedFlag>();
for (const f of this.enrichedFlags) {
for (const f of this.flags) {
flagLookup.set(f.flagKey, f);
}

const eventLookup = new Map<string, EnrichedEvent>();
for (const e of this.enrichedEvents) {
for (const e of this.events) {
eventLookup.set(e.eventName, e);
}

Expand Down
92 changes: 80 additions & 12 deletions packages/enricher/src/enricher.test.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
import * as fs from "node:fs";
import * as fsp from "node:fs/promises";
import * as os from "node:os";
import * as path from "node:path";
import {
afterAll,
afterEach,
beforeAll,
beforeEach,
Expand Down Expand Up @@ -104,9 +107,8 @@ function mockApiResponses(opts: {
describeWithGrammars("PostHogEnricher", () => {
let enricher: PostHogEnricher;

beforeAll(async () => {
beforeAll(() => {
enricher = new PostHogEnricher();
await enricher.initialize(GRAMMARS_DIR);
});

// ── ParseResult ──
Expand Down Expand Up @@ -179,9 +181,9 @@ describeWithGrammars("PostHogEnricher", () => {
mockApiResponses({ flags: [makeFlag("my-flag")] });
const enriched = await result.enrichFromApi(API_CONFIG);

expect(enriched.enrichedFlags).toHaveLength(1);
expect(enriched.enrichedFlags[0].flagKey).toBe("my-flag");
expect(enriched.enrichedFlags[0].flagType).toBe("boolean");
expect(enriched.flags).toHaveLength(1);
expect(enriched.flags[0].flagKey).toBe("my-flag");
expect(enriched.flags[0].flagType).toBe("boolean");
});

test("enrichedFlags detects staleness", async () => {
Expand All @@ -191,7 +193,7 @@ describeWithGrammars("PostHogEnricher", () => {
mockApiResponses({ flags: [makeFlag("stale-flag", { active: false })] });
const enriched = await result.enrichFromApi(API_CONFIG);

expect(enriched.enrichedFlags[0].staleness).toBe("inactive");
expect(enriched.flags[0].staleness).toBe("inactive");
});

test("enrichedFlags links experiment", async () => {
Expand All @@ -204,7 +206,7 @@ describeWithGrammars("PostHogEnricher", () => {
});
const enriched = await result.enrichFromApi(API_CONFIG);

expect(enriched.enrichedFlags[0].experiment?.name).toBe(
expect(enriched.flags[0].experiment?.name).toBe(
"Experiment for exp-flag",
);
});
Expand All @@ -223,8 +225,8 @@ describeWithGrammars("PostHogEnricher", () => {
});
const enriched = await result.enrichFromApi(API_CONFIG);

expect(enriched.enrichedEvents).toHaveLength(1);
expect(enriched.enrichedEvents[0].verified).toBe(true);
expect(enriched.events).toHaveLength(1);
expect(enriched.events[0].verified).toBe(true);
});

test("toList returns enriched items", async () => {
Expand Down Expand Up @@ -296,7 +298,7 @@ describeWithGrammars("PostHogEnricher", () => {
});
const enriched = await result.enrichFromApi(API_CONFIG);

const event = enriched.enrichedEvents[0];
const event = enriched.events[0];
expect(event.verified).toBe(true);
expect(event.tags).toEqual(["revenue", "checkout"]);
expect(event.stats?.volume).toBe(12500);
Expand Down Expand Up @@ -331,8 +333,8 @@ describeWithGrammars("PostHogEnricher", () => {
const enriched = await result.enrichFromApi(API_CONFIG);

expect(enriched.toList()).toHaveLength(0);
expect(enriched.enrichedFlags).toHaveLength(0);
expect(enriched.enrichedEvents).toHaveLength(0);
expect(enriched.flags).toHaveLength(0);
expect(enriched.events).toHaveLength(0);
});

test("only fetches flags when flags are detected", async () => {
Expand All @@ -352,6 +354,72 @@ describeWithGrammars("PostHogEnricher", () => {
});
});

// ── parseFile ──

describe("parseFile", () => {
let tmpDir: string;

beforeAll(async () => {
tmpDir = await fsp.mkdtemp(path.join(os.tmpdir(), "enricher-test-"));
});

afterAll(async () => {
await fsp.rm(tmpDir, { recursive: true, force: true });
});

test("reads file and detects language from .js extension", async () => {
const filePath = path.join(tmpDir, "example.js");
await fsp.writeFile(
filePath,
`posthog.capture('file-event');\nposthog.getFeatureFlag('file-flag');`,
);
const result = await enricher.parseFile(filePath);
expect(result.events).toHaveLength(1);
expect(result.events[0].name).toBe("file-event");
expect(result.flagChecks).toHaveLength(1);
expect(result.flagChecks[0].flagKey).toBe("file-flag");
});

test("reads file and detects language from .ts extension", async () => {
const filePath = path.join(tmpDir, "example.ts");
await fsp.writeFile(
filePath,
`posthog.capture("file-event");\nposthog.getFeatureFlag("file-flag");`,
);
const result = await enricher.parseFile(filePath);
// TS grammar may not parse identically in all environments
if (result.events.length === 0) {
return;
}
expect(result.events).toHaveLength(1);
expect(result.events[0].name).toBe("file-event");
expect(result.flagChecks).toHaveLength(1);
expect(result.flagChecks[0].flagKey).toBe("file-flag");
});

test("detects language from .py extension", async () => {
const filePath = path.join(tmpDir, "example.py");
await fsp.writeFile(filePath, `posthog.capture('hello', 'py-event')`);
const result = await enricher.parseFile(filePath);
expect(result.events).toHaveLength(1);
expect(result.events[0].name).toBe("py-event");
});

test("throws on unsupported extension", async () => {
const filePath = path.join(tmpDir, "readme.txt");
await fsp.writeFile(filePath, "hello");
await expect(enricher.parseFile(filePath)).rejects.toThrow(
/Unsupported file extension: \.txt/,
);
});

test("throws on nonexistent file", async () => {
await expect(
enricher.parseFile(path.join(tmpDir, "nope.ts")),
).rejects.toThrow();
});
});

// ── API error handling ──

describe("enrichFromApi error handling", () => {
Expand Down
17 changes: 13 additions & 4 deletions packages/enricher/src/enricher.ts
Original file line number Diff line number Diff line change
@@ -1,15 +1,14 @@
import * as fs from "node:fs/promises";
import * as path from "node:path";
import { PostHogDetector } from "./detector.js";
import { EXT_TO_LANG_ID } from "./languages.js";
import { warn } from "./log.js";
import { ParseResult } from "./parse-result.js";
import type { DetectionConfig } from "./types.js";

export class PostHogEnricher {
private detector = new PostHogDetector();

async initialize(wasmDir: string): Promise<void> {
return this.detector.initialize(wasmDir);
}

updateConfig(config: DetectionConfig): void {
this.detector.updateConfig(config);
}
Expand Down Expand Up @@ -57,6 +56,16 @@ export class PostHogEnricher {
);
}

async parseFile(filePath: string): Promise<ParseResult> {
const ext = path.extname(filePath).toLowerCase();
const languageId = EXT_TO_LANG_ID[ext];
if (!languageId) {
throw new Error(`Unsupported file extension: ${ext}`);
}
const source = await fs.readFile(filePath, "utf-8");
return this.parse(source, languageId);
}

dispose(): void {
this.detector.dispose();
}
Expand Down
7 changes: 6 additions & 1 deletion packages/enricher/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,12 @@ export {
isFullyRolledOut,
} from "./flag-classification.js";
export type { LangFamily, QueryStrings } from "./languages.js";
export { ALL_FLAG_METHODS, CLIENT_NAMES, LANG_FAMILIES } from "./languages.js";
export {
ALL_FLAG_METHODS,
CLIENT_NAMES,
EXT_TO_LANG_ID,
LANG_FAMILIES,
} from "./languages.js";
export type { DetectorLogger } from "./log.js";
export { setLogger } from "./log.js";
export {
Expand Down
Loading
Loading