feat(dialtone-docs): DLT-3109 add markdown-to-JSON generator for AI docs#1178
feat(dialtone-docs): DLT-3109 add markdown-to-JSON generator for AI docs#1178belumontoya wants to merge 3 commits intostagingfrom
Conversation
belumontoya
requested review from
Brad Paugh (braddialpad),
Francis Rupert (francisrupert),
Ignacio Ropolo (iropolo) and
Nina Repetto (ninarepetto)
as code owners
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 208c3b30bd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
Brad Paugh (braddialpad)
left a comment
Brad Paugh (braddialpad)
left a comment
There was a problem hiding this comment.
Looks really good, perhaps my only concern would be the large amount of confusing regex to parse the markdown, however it is necessary for this change and also easier to understand in the age of AI.
Couple of small comments, nothing major.
| for (const doc of docs) { | ||
| expect(doc.type, `"${doc.id}" type is null`).not.toBeNull(); | ||
| expect(ALLOWED_TYPES, `"${doc.id}" invalid type "${doc.type}"`).toContain(doc.type); | ||
| } |
There was a problem hiding this comment.
For all tests where we are looping through arrays like this, we should be using test.each instead of a for loop.
A single failure will mask all subsequent ones when doing it the current way.
There was a problem hiding this comment.
Agree, but one thing worth noting. docs is built in beforeAll, so test.each can't get it at test definition time. I could build synchronously at module scope or keep the loop, but use soft assertions. Any preference? Or a better idea?
There was a problem hiding this comment.
I guess we can just leave docs alone, and do something like this at least:
test.each(REQUIRED_FIELDS)('all entries have field "%s"', (field) => {
for (const doc of docs) {
expect(doc).toHaveProperty(field);
}
});
Moving the file reads outside of test execution has worse side effects than the current method IMO
- Fix non-deterministic date serialization: String(Date) is
timezone-dependent, use toISOString().split('T')[0] for stable
YYYY-MM-DD output
- Remove redundant null assertion in type field test
# Conflicts: # packages/dialtone-docs/src/generators/build-ai-docs.mjs # packages/dialtone-docs/tests/helpers/markdownParser.js
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository YAML (base), Central YAML (inherited), Organization UI (inherited) Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
WalkthroughThe dialtone-docs package build and test configuration was updated: the test target now depends on the build target, the Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
2 participants
feat(dialtone-docs): DLT-3109 add markdown-to-JSON generator for AI docs
Obligatory GIF (super important!)
🛠️ Type Of Change
📖 Jira Ticket
https://dialpad.atlassian.net/browse/DLT-3109
📖 Description
Adds the markdown-to-JSON build pipeline for the
dialtone-docspackage:src/generators/build-ai-docs.mjs— Reads all markdown files undersrc/content/, parses YAML frontmatter (type, category, keywords, ai_summary), strips markdown syntax, and compiles everything intodist/ai-docs.json— a flat JSON array of document entries for AI consumption.src/utils/strip-markdown.mjs— Utility that strips frontmatter, code blocks, HTML, links, headings, emphasis, and other markdown syntax to produce searchable plain text.package.json/project.json— Addedbuildscript and NX target sopnpm nx run dialtone-docs:buildtriggers the generator.tests/tests/build-output.test.js— 11 tests validating the JSON output schema (required fields, types, no markdown artifacts in content, file path integrity, no duplicate IDs).tests/tests/strip-markdown.test.js— Unit tests for the strip-markdown utility (headings, code blocks, links, emphasis, frontmatter removal).tests/helpers/markdownParser.js— Refactored to importstripMarkdown/stripFrontmatterfrom the new utility instead of bundling its own copy.💡 Context
The
dialtone-docspackage provides AI-discoverable documentation for the Dialtone monorepo. This PR adds Milestone 3: the build step that compiles markdown content into a structured JSON file (ai-docs.json). This JSON output will serve as the data source for MCP server and CLI search tools, enabling AI agents to search the entire documentation site programmatically.Each JSON entry includes:
id,title,type,category,keywords,summary,content(plain text),filePath,lastUpdated, andrelatedPackages.📝 Checklist
🔮 Next Steps
ai-docs.jsoninto the MCP server and CLI as a search data sourceUpdated date serialization in the markdown-to-JSON generator to use deterministic ISO format. Refactored test setup to check for pre-built dist/ai-docs.json instead of executing builds synchronously. Added build target dependency to ensure compilation before tests.