From 4c4cbceb3a499d04bfaa2e022942eb68b90dc79c Mon Sep 17 00:00:00 2001 From: Inline Pizza <249805557+InlinePizza@users.noreply.github.com> Date: Thu, 12 Mar 2026 10:05:34 -0700 Subject: [PATCH 1/6] Add Doc Detective + Promptless blog post Announce Doc Detective support in Promptless as a product update. --- .../blog/product-updates/doc-detective.mdx | 100 ++++++++++++++++++ 1 file changed, 100 insertions(+) create mode 100644 src/content/blog/product-updates/doc-detective.mdx diff --git a/src/content/blog/product-updates/doc-detective.mdx b/src/content/blog/product-updates/doc-detective.mdx new file mode 100644 index 00000000..e5afb47d --- /dev/null +++ b/src/content/blog/product-updates/doc-detective.mdx @@ -0,0 +1,100 @@ +--- +title: 'Doc Detective + Promptless: Test your docs like you test your code' +subtitle: Published March 2026 +description: >- + Promptless now supports Doc Detective, the documentation testing framework. + Apply zero trust to your documentation: love your product team, but verify + that docs and product are in sync. +date: '2026-03-11T00:00:00.000Z' +section: Product Updates +hidden: false +--- + +Manny Silva, Doc Detective creator, borrows a concept from security: apply zero trust to your documentation. Love your product team, but don't assume things are in sync -- verify it. + +Back in August 2024, Manny Silva was one of the first technical writers from whom we ever got feedback about Promptless. Back then, he'd already been working on Doc Detective for years, and it seemed like Promptless and Doc Detective would work perfectly together. + +**Today, we released Doc Detective support in Promptless**, meaning that Promptless can create/update Doc Detective tests automatically and fix documentation or escalate software issues whenever Doc Detective tests break! + +## Why should you test your docs? + +Good documentation is expensive. Not because writing is hard (though it is), but because the thinking behind it is hard. You have to understand the problems your users face, figure out how to structure information so that someone with a specific problem can find the right page, the right section, and get unstuck. You have to think about information architecture, about progressive disclosure, about the right level of detail for each audience. That work takes real effort and real expertise. + +And once you have done all that thinking and planning, you end up with documentation that is tightly coupled to your product. The docs describe specific screens, specific API responses, specific command outputs. That coupling is the whole point. Your docs are valuable precisely because they reflect what the product actually does. + +But coupling has a cost. Anyone on the team can change the product. A developer renames an API field. A designer moves a button. A backend change alters a response format. The product changes, the docs don't, and now your carefully structured documentation is lying to your users. All that time you spent thinking about how to guide people through their problems? Wasted, because step 3 no longer matches what they see on screen. + +This is the problem Doc Detective solves. It tests the coupling between your documentation and your product, so you know when it breaks. + +## What is Doc Detective? + +Doc Detective is a documentation testing framework. You write test specifications (in JSON, YAML, or inline in your Markdown files) that describe the procedures in your docs, and Doc Detective executes them against the real product. It opens a browser, navigates pages, clicks elements, fills in forms, takes screenshots, makes API calls, runs shell commands. If what happens doesn't match what your docs say should happen, the test fails. + +Think of it as end-to-end testing, but the source of truth is your documentation rather than a product spec. + +## Use case 1: Your product has a UI, and your docs walk users through it + +Your docs say: go to this page, click this button, fill in this field, see this result. Doc Detective does exactly that. It navigates to a URL with `goTo`, finds elements with `find` (by CSS selector, text content, ARIA label, test ID, whatever matches your situation), clicks them, types into input fields, and takes screenshots. + +It handles the messiness of real web applications too. You can configure wait conditions that monitor network activity and DOM mutations, so the test only proceeds once the page has actually finished loading. Useful when your single-page app is still fetching data three seconds after navigation. + +The screenshot capability goes further than just capturing images. Doc Detective compares screenshots against reference images pixel by pixel. You set a variation threshold (say 5%), and if the current state of the UI has drifted beyond that, the test flags it. So when someone changes the product and the docs still show the old screenshots, you find out from your test suite, not from a confused user filing a support ticket. + +## Use case 2: Your product is API-driven + +API documentation is full of example requests and expected responses. These are promises to your users: "Send this, get that back." When someone changes the API and forgets the docs, those promises break silently. + +Doc Detective's `httpRequest` action lets you define requests and validate responses: status codes, headers, body fields. If you maintain an OpenAPI specification, it can pull examples directly from the spec, validate request bodies against your schemas, and verify that responses match what the spec defines. That gives you a feedback loop between the spec, the docs, and the actual API. + +## Use case 3: Your docs include shell commands or code snippets + +Every tutorial that says "run this command and you should see this output" is making a testable claim. With `runShell` and `runCode`, Doc Detective executes those commands (Python, JavaScript, Bash) and checks the results: exit codes, stdout matching (literal or regex), stderr. When someone changes the CLI output format and doesn't update the tutorial, the test catches it. + +## Use case 4: Cross-browser and cross-platform verification + +Doc Detective supports Firefox, Chrome, and Safari. You define multiple contexts for a test (Chrome on macOS, Firefox on Linux) and the same specification runs across all of them. If your docs describe a flow that works in Chrome but breaks in Firefox, you will know. + +## Use case 5: Authenticated flows + +Most useful documentation covers features behind a login. Doc Detective handles this with cookie management (`saveCookie` / `loadCookie`) and has a Clerk integration for injecting test tokens. You can test documented procedures for authenticated features without manual login steps. + +## Use case 6: Visual regression for documentation screenshots + +You have dozens or hundreds of screenshots in your docs. A new release ships. Which screenshots are now wrong? Without tooling, someone has to check each one manually. With Doc Detective, every screenshot is retaken during the test run and compared against the reference. If the UI has changed beyond your threshold, the test flags it. You can even set it to auto-update reference images, so your docs stay current without manual intervention. + +## Use case 7: Recording procedures + +Doc Detective can record browser sessions as WebM video. Start a recording, run through a documented procedure, stop the recording. You get an up-to-date video artifact every time the tests run. + +## How tests are structured + +A specification contains tests, each test is a sequence of steps, and each step is an action: + +```json +{ + "tests": [ + { + "steps": [ + { "action": "goTo", "url": "https://myapp.example.com/login" }, + { "action": "find", "selector": "#email", "click": true }, + { "action": "type", "keys": "user@example.com" }, + { "action": "find", "selector": "#password", "click": true }, + { "action": "type", "keys": "correct-horse-battery-staple" }, + { "action": "find", "selector": "button[type=submit]", "click": true }, + { "action": "find", "text": "Welcome back" }, + { "action": "screenshot", "path": "dashboard.png" } + ] + } + ] +} +``` + +You can also embed tests directly in Markdown as comments, or use automatic markup detection. Doc Detective can parse instructions like "Go to [the dashboard](https://app.example.com/)" and generate the corresponding test action from the Markdown itself. + +## Where does it fit in your workflow? + +Doc Detective runs from the command line. Put it in your CI/CD pipeline. Every time someone changes the docs or the product, the doc tests run. Failures show up alongside your code test failures. The coupling between docs and product becomes something you actively monitor, not something that silently degrades. + +## Final thoughts + +The hard part of documentation is the thinking: understanding your users' problems and structuring information so they can find answers. That investment is wasted when the docs fall out of sync with the product. Doc Detective protects that investment by testing the coupling continuously. It won't write your docs for you, but it will tell you, with certainty, when they stop being true. From d64381bc7518cda435a06249704e1d946ffc5738 Mon Sep 17 00:00:00 2001 From: Inline Pizza <249805557+InlinePizza@users.noreply.github.com> Date: Tue, 17 Mar 2026 09:53:19 -0700 Subject: [PATCH 2/6] fix: update to v4 syntax --- .../blog/product-updates/doc-detective.mdx | 76 ++++++++++++++----- 1 file changed, 59 insertions(+), 17 deletions(-) diff --git a/src/content/blog/product-updates/doc-detective.mdx b/src/content/blog/product-updates/doc-detective.mdx index e5afb47d..6f1ce07b 100644 --- a/src/content/blog/product-updates/doc-detective.mdx +++ b/src/content/blog/product-updates/doc-detective.mdx @@ -10,11 +10,43 @@ section: Product Updates hidden: false --- -Manny Silva, Doc Detective creator, borrows a concept from security: apply zero trust to your documentation. Love your product team, but don't assume things are in sync -- verify it. - -Back in August 2024, Manny Silva was one of the first technical writers from whom we ever got feedback about Promptless. Back then, he'd already been working on Doc Detective for years, and it seemed like Promptless and Doc Detective would work perfectly together. - -**Today, we released Doc Detective support in Promptless**, meaning that Promptless can create/update Doc Detective tests automatically and fix documentation or escalate software issues whenever Doc Detective tests break! +Manny Silva, Doc Detective creator, borrows a concept from security: apply zero +trust to your documentation. Love your product team, but don't assume things are +in sync -- verify it. + +Back in August 2024, Manny Silva was one of the first technical writers from +whom we ever got feedback about Promptless. Back then, he'd already been working +on Doc Detective for years, and it seemed like Promptless and Doc Detective +would work perfectly together. + +**Today, we released Doc Detective support in Promptless**, meaning that +Promptless can create/update Doc Detective tests automatically and fix +documentation or escalate software issues whenever Doc Detective tests fail! + +## If you have Promptless, why do you need Doc Detective? + +Promptless and Doc Detective solve related but different problems. Promptless +reduces documentation drift by watching for product changes and proactively +updating your docs. That is valuable, but it is not the whole picture. +Promptless is an AI system, and AI systems can produce content that reads well +but is subtly wrong. Human writers can too. The source of the content does not +matter — what matters is whether the documented procedure actually works. + +Doc Detective answers that question. It takes the claims your documentation +makes ("click this button", "you should see this", "send this request," "run +this command") and executes them against the real product. If the result does +not match what the docs say, the test fails. It verifies that the documentation +that all your customers, lead, and prospectssd see continues to accurately reflect the product. + +Doc Detective and Promptless play different roles. Doc Detective gives you +precise, high-confidence verification that your docs are correct by executing +what they claim. Promptless listens for product and code changes that suggest +your docs are about to become stale, incomplete, or wrong, then helps you +respond. Promptless wields Doc Detective as a tool: suggesting new tests, +generating them, and using failures to drive targeted doc updates or escalate +product issues. In practice, Doc Detective is the rigorous foundation, and +Promptless is the eager, adaptive brick-layer on the job site that keeps +improving coverage over time. ## Why should you test your docs? @@ -24,7 +56,11 @@ And once you have done all that thinking and planning, you end up with documenta But coupling has a cost. Anyone on the team can change the product. A developer renames an API field. A designer moves a button. A backend change alters a response format. The product changes, the docs don't, and now your carefully structured documentation is lying to your users. All that time you spent thinking about how to guide people through their problems? Wasted, because step 3 no longer matches what they see on screen. -This is the problem Doc Detective solves. It tests the coupling between your documentation and your product, so you know when it breaks. +This is fundamentally a coordination problem between engineering and technical writing. Engineers are experimenting with ways to ship changes faster and faster. Technical writers are responsible for communicating to customers exactly how to solve their problems with the product. Every time a writer documents a procedure, they are making assumptions about how the product works. Without tooling, the only way to check those assumptions is to manually verify each procedure — and that does not scale. + +Doc Detective gives technical writers the ability to automatically verify every assumption they made when they wrote the docs. When something in the product changes and breaks the contract with the documentation, the doc tests fail. Writers do not have to spend their time manually checking up on engineering. They can focus on the hard, valuable work — understanding users and structuring information — and let the tests tell them when something needs attention. + +It also shifts responsibility. Put doc tests in CI, and when an engineer's PR breaks the docs, they get that feedback directly. The engineer who made the change is the one who sees the failure. The technical writer does not have to be the one communicating "hey, you broke the docs" — the test suite does that for them. It is an unbiased signal that scales across every engineer on the team without requiring the writer to be in the loop on every change. ## What is Doc Detective? @@ -42,7 +78,7 @@ The screenshot capability goes further than just capturing images. Doc Detective ## Use case 2: Your product is API-driven -API documentation is full of example requests and expected responses. These are promises to your users: "Send this, get that back." When someone changes the API and forgets the docs, those promises break silently. +API documentation is full of example requests and expected responses. These are promises to your users: "Send this, get that back." When someone changes the API and forgets the docs, those promises fail silently. Doc Detective's `httpRequest` action lets you define requests and validate responses: status codes, headers, body fields. If you maintain an OpenAPI specification, it can pull examples directly from the spec, validate request bodies against your schemas, and verify that responses match what the spec defines. That gives you a feedback loop between the spec, the docs, and the actual API. @@ -52,11 +88,17 @@ Every tutorial that says "run this command and you should see this output" is ma ## Use case 4: Cross-browser and cross-platform verification -Doc Detective supports Firefox, Chrome, and Safari. You define multiple contexts for a test (Chrome on macOS, Firefox on Linux) and the same specification runs across all of them. If your docs describe a flow that works in Chrome but breaks in Firefox, you will know. +Doc Detective supports Firefox, Chrome, and Safari. You define multiple contexts for a test (Chrome on macOS, Firefox on Linux) and the same specification runs across all of them. If your docs describe a flow that works in Chrome but fails in Firefox, you will know. ## Use case 5: Authenticated flows -Most useful documentation covers features behind a login. Doc Detective handles this with cookie management (`saveCookie` / `loadCookie`) and has a Clerk integration for injecting test tokens. You can test documented procedures for authenticated features without manual login steps. +Most useful documentation covers features behind a login. Doc Detective provides building blocks for handling auth: you can use `before` and `beforeAny` hooks to run login steps before your tests, and `saveCookie` / `loadCookie` actions to manage cookies. + +The challenge is that each Doc Detective test starts in a fresh browser instance. That means you will be logging in over and over again unless you find a way to persist authentication state between tests. How hard that is depends entirely on your auth provider. Some providers make it straightforward to save and reload session cookies. Others make it genuinely difficult. + +For example, if you use Clerk, their auth cookies have a 60-second lifetime and are refreshed by client-side JavaScript. This is how Clerk implements SSO sign-out — since they cannot push token revocation to your servers, they keep cookie lifetimes very short and rely on the browser to refresh them. That design is reasonable from Clerk's perspective, but it means you cannot simply save a cookie and reload it in a new browser instance. You will need to come up with a workaround, like re-authenticating at the start of each test or finding a way to inject a valid session outside of Clerk's normal flow. + +This is an area where you will need to invest some effort to match your specific auth setup. Doc Detective gives you the primitives, but the solution depends on your provider. ## Use case 6: Visual regression for documentation screenshots @@ -75,14 +117,14 @@ A specification contains tests, each test is a sequence of steps, and each step "tests": [ { "steps": [ - { "action": "goTo", "url": "https://myapp.example.com/login" }, - { "action": "find", "selector": "#email", "click": true }, - { "action": "type", "keys": "user@example.com" }, - { "action": "find", "selector": "#password", "click": true }, - { "action": "type", "keys": "correct-horse-battery-staple" }, - { "action": "find", "selector": "button[type=submit]", "click": true }, - { "action": "find", "text": "Welcome back" }, - { "action": "screenshot", "path": "dashboard.png" } + { "goTo": "https://myapp.example.com/login" }, + { "find": { "selector": "#email", "click": true } }, + { "type": "user@example.com" }, + { "find": { "selector": "#password", "click": true } }, + { "type": "correct-horse-battery-staple" }, + { "click": "button[type=submit]" }, + { "find": "Welcome back" }, + { "screenshot": "dashboard.png" } ] } ] From 39a71a3bb921ae84716f698fdc65cef952916290 Mon Sep 17 00:00:00 2001 From: Inline Pizza <249805557+InlinePizza@users.noreply.github.com> Date: Tue, 17 Mar 2026 12:37:30 -0700 Subject: [PATCH 3/6] fixup! fix: update to v4 syntax --- .../blog/product-updates/doc-detective.mdx | 98 ++++++++++++------- 1 file changed, 65 insertions(+), 33 deletions(-) diff --git a/src/content/blog/product-updates/doc-detective.mdx b/src/content/blog/product-updates/doc-detective.mdx index 6f1ce07b..19718618 100644 --- a/src/content/blog/product-updates/doc-detective.mdx +++ b/src/content/blog/product-updates/doc-detective.mdx @@ -25,42 +25,74 @@ documentation or escalate software issues whenever Doc Detective tests fail! ## If you have Promptless, why do you need Doc Detective? -Promptless and Doc Detective solve related but different problems. Promptless -reduces documentation drift by watching for product changes and proactively -updating your docs. That is valuable, but it is not the whole picture. -Promptless is an AI system, and AI systems can produce content that reads well -but is subtly wrong. Human writers can too. The source of the content does not -matter — what matters is whether the documented procedure actually works. - -Doc Detective answers that question. It takes the claims your documentation -makes ("click this button", "you should see this", "send this request," "run -this command") and executes them against the real product. If the result does -not match what the docs say, the test fails. It verifies that the documentation -that all your customers, lead, and prospectssd see continues to accurately reflect the product. - -Doc Detective and Promptless play different roles. Doc Detective gives you -precise, high-confidence verification that your docs are correct by executing -what they claim. Promptless listens for product and code changes that suggest -your docs are about to become stale, incomplete, or wrong, then helps you -respond. Promptless wields Doc Detective as a tool: suggesting new tests, -generating them, and using failures to drive targeted doc updates or escalate -product issues. In practice, Doc Detective is the rigorous foundation, and -Promptless is the eager, adaptive brick-layer on the job site that keeps -improving coverage over time. +Promptless fixes docs drift. When a pull request renames an API field or changes +a workflow, Promptless identifies what needs to be updated in the docs before the +code ships. It monitors pull requests, Slack, Discord, and support channels so +your writers do not have to manually track every engineering change. The most +valuable thing Promptless does is catch drift before it reaches your customers. + +But products are complex and teams move fast. Not every change that affects the +docs will be caught before it ships. Doc Detective is your safety net. It takes +the procedures your docs describe ("click this button," "send this request," +"run this command") and executes them against the real product. If the result +does not match what the docs say, the test fails. No interpretation, no +inference. It is a test suite for your documentation, closer to Cucumber than to +an AI agent, and cheap enough to run on every PR or on a schedule. + +When a Doc Detective test fails, that is definitive proof that your docs have +drifted. Promptless picks up that failure, figures out what the documentation +needs to say to match the behavior Doc Detective actually observed, and makes +the fix. Promptless also uses your product directly to write new Doc Detective +tests, so your test coverage grows as your docs grow. -## Why should you test your docs? - -Good documentation is expensive. Not because writing is hard (though it is), but because the thinking behind it is hard. You have to understand the problems your users face, figure out how to structure information so that someone with a specific problem can find the right page, the right section, and get unstuck. You have to think about information architecture, about progressive disclosure, about the right level of detail for each audience. That work takes real effort and real expertise. - -And once you have done all that thinking and planning, you end up with documentation that is tightly coupled to your product. The docs describe specific screens, specific API responses, specific command outputs. That coupling is the whole point. Your docs are valuable precisely because they reflect what the product actually does. -But coupling has a cost. Anyone on the team can change the product. A developer renames an API field. A designer moves a button. A backend change alters a response format. The product changes, the docs don't, and now your carefully structured documentation is lying to your users. All that time you spent thinking about how to guide people through their problems? Wasted, because step 3 no longer matches what they see on screen. - -This is fundamentally a coordination problem between engineering and technical writing. Engineers are experimenting with ways to ship changes faster and faster. Technical writers are responsible for communicating to customers exactly how to solve their problems with the product. Every time a writer documents a procedure, they are making assumptions about how the product works. Without tooling, the only way to check those assumptions is to manually verify each procedure — and that does not scale. - -Doc Detective gives technical writers the ability to automatically verify every assumption they made when they wrote the docs. When something in the product changes and breaks the contract with the documentation, the doc tests fail. Writers do not have to spend their time manually checking up on engineering. They can focus on the hard, valuable work — understanding users and structuring information — and let the tests tell them when something needs attention. +## Why should you test your docs? -It also shifts responsibility. Put doc tests in CI, and when an engineer's PR breaks the docs, they get that feedback directly. The engineer who made the change is the one who sees the failure. The technical writer does not have to be the one communicating "hey, you broke the docs" — the test suite does that for them. It is an unbiased signal that scales across every engineer on the team without requiring the writer to be in the loop on every change. +Good documentation is hard. Not because writing is hard (though it is), but +because the thinking behind it is hard. You have to understand the problems your +users face, figure out how to structure information so that someone with a +specific problem can find the right page, the right section, and get unstuck. +You have to think about information architecture, about progressive disclosure, +about the right level of detail for each audience. That work takes a bunch of +effort that is repaid in tiny increments as your customers, leads, and prospects +find the answers they need. + +The tricky bit is once you have done all that thinking and planning, you end up +with documentation that is tightly coupled to your product. The docs describe +specific screens, specific API responses, specific command outputs. That +coupling is the whole point. Your docs are valuable precisely because they +reflect what the product actually does. + +That coupling has a cost. Anyone on the team can change the product. A developer +fixes deficiencies in the API to make it more useful. A designer moves a button. +A backend change alters a response format. The product changes, the docs don't, +and now your carefully structured documentation is lying to your users. All that +time you spent thinking about how to guide people through their problems? +Wasted, because step 3 no longer matches what they see on screen. + +People typicall allude to a coordination problem between engineering and +technical writing. + +Engineers are experimenting with ways to ship changes faster +and faster. Technical writers are responsible for communicating to customers +exactly how to solve their problems with the product. Every time a writer +documents a procedure, they are making assumptions about how the product works. +Without tooling, the only way to check those assumptions is to manually verify +each procedure — and that does not scale. + +Doc Detective gives technical writers the ability to automatically verify every +assumption they made when they wrote the docs. When something in the product +changes and breaks the contract with the documentation, the doc tests fail. +Writers do not have to spend their time manually checking up on engineering. +They can focus on the hard, valuable work — understanding users and structuring +information — and let the tests tell them when something needs attention. + +It also shifts responsibility. Put doc tests in CI, and when an engineer's PR +breaks the docs, they get that feedback directly. The engineer who made the +change is the one who sees the failure. The technical writer does not have to be +the one communicating "hey, you broke the docs" — the test suite does that for +them. It scales across every engineer on the team. The writer +does not have to be in the loop on every change. ## What is Doc Detective? From 087cb5b96f81be9dd04d53b5d9f5adddfc3590ad Mon Sep 17 00:00:00 2001 From: Inline Pizza <249805557+InlinePizza@users.noreply.github.com> Date: Tue, 17 Mar 2026 13:17:32 -0700 Subject: [PATCH 4/6] fix: shorten article, move use case list to appendix --- .../blog/product-updates/doc-detective.mdx | 229 +++++++++--------- src/styles/custom.css | 6 + 2 files changed, 123 insertions(+), 112 deletions(-) diff --git a/src/content/blog/product-updates/doc-detective.mdx b/src/content/blog/product-updates/doc-detective.mdx index 19718618..db1e5f7b 100644 --- a/src/content/blog/product-updates/doc-detective.mdx +++ b/src/content/blog/product-updates/doc-detective.mdx @@ -23,152 +23,157 @@ would work perfectly together. Promptless can create/update Doc Detective tests automatically and fix documentation or escalate software issues whenever Doc Detective tests fail! -## If you have Promptless, why do you need Doc Detective? -Promptless fixes docs drift. When a pull request renames an API field or changes -a workflow, Promptless identifies what needs to be updated in the docs before the -code ships. It monitors pull requests, Slack, Discord, and support channels so -your writers do not have to manually track every engineering change. The most -valuable thing Promptless does is catch drift before it reaches your customers. - -But products are complex and teams move fast. Not every change that affects the -docs will be caught before it ships. Doc Detective is your safety net. It takes -the procedures your docs describe ("click this button," "send this request," -"run this command") and executes them against the real product. If the result -does not match what the docs say, the test fails. No interpretation, no -inference. It is a test suite for your documentation, closer to Cucumber than to -an AI agent, and cheap enough to run on every PR or on a schedule. +## What is Doc Detective? -When a Doc Detective test fails, that is definitive proof that your docs have -drifted. Promptless picks up that failure, figures out what the documentation -needs to say to match the behavior Doc Detective actually observed, and makes -the fix. Promptless also uses your product directly to write new Doc Detective -tests, so your test coverage grows as your docs grow. +Doc Detective is a documentation testing framework. You write test +specifications (in JSON, YAML, or inline in your Markdown files) that describe +the procedures in your docs, and Doc Detective executes them against the real +product. It opens a browser, navigates pages, clicks elements, fills in forms, +takes screenshots, makes API calls, runs shell commands. If what happens doesn't +match what your docs say should happen, the test fails. +Think of it as end-to-end testing, but the source of truth is your documentation +rather than a product spec. ## Why should you test your docs? -Good documentation is hard. Not because writing is hard (though it is), but -because the thinking behind it is hard. You have to understand the problems your -users face, figure out how to structure information so that someone with a -specific problem can find the right page, the right section, and get unstuck. -You have to think about information architecture, about progressive disclosure, -about the right level of detail for each audience. That work takes a bunch of -effort that is repaid in tiny increments as your customers, leads, and prospects -find the answers they need. - -The tricky bit is once you have done all that thinking and planning, you end up -with documentation that is tightly coupled to your product. The docs describe -specific screens, specific API responses, specific command outputs. That -coupling is the whole point. Your docs are valuable precisely because they -reflect what the product actually does. - -That coupling has a cost. Anyone on the team can change the product. A developer -fixes deficiencies in the API to make it more useful. A designer moves a button. -A backend change alters a response format. The product changes, the docs don't, -and now your carefully structured documentation is lying to your users. All that -time you spent thinking about how to guide people through their problems? -Wasted, because step 3 no longer matches what they see on screen. - -People typicall allude to a coordination problem between engineering and -technical writing. - -Engineers are experimenting with ways to ship changes faster -and faster. Technical writers are responsible for communicating to customers -exactly how to solve their problems with the product. Every time a writer -documents a procedure, they are making assumptions about how the product works. -Without tooling, the only way to check those assumptions is to manually verify -each procedure — and that does not scale. - -Doc Detective gives technical writers the ability to automatically verify every -assumption they made when they wrote the docs. When something in the product -changes and breaks the contract with the documentation, the doc tests fail. -Writers do not have to spend their time manually checking up on engineering. -They can focus on the hard, valuable work — understanding users and structuring -information — and let the tests tell them when something needs attention. - -It also shifts responsibility. Put doc tests in CI, and when an engineer's PR -breaks the docs, they get that feedback directly. The engineer who made the -change is the one who sees the failure. The technical writer does not have to be -the one communicating "hey, you broke the docs" — the test suite does that for -them. It scales across every engineer on the team. The writer -does not have to be in the loop on every change. +We see Promptless and Doc Detective as two sides of the same coin. Promptless +helps prevent and remediate docs drift, and Doc Detective _deterministically +validates_ that any content, whether written by Promptless or a human, is +faithful to the product that it documents. -## What is Doc Detective? +Companies that are the most serious about guaranteeing the accuracy of their +docs should use Promptless and Doc Detective together, and now Promptless users +can get Doc Detective tests created automatically with every Promptless +suggestion! -Doc Detective is a documentation testing framework. You write test specifications (in JSON, YAML, or inline in your Markdown files) that describe the procedures in your docs, and Doc Detective executes them against the real product. It opens a browser, navigates pages, clicks elements, fills in forms, takes screenshots, makes API calls, runs shell commands. If what happens doesn't match what your docs say should happen, the test fails. +## How tests are structured -Think of it as end-to-end testing, but the source of truth is your documentation rather than a product spec. +A specification contains tests, each test is a sequence of steps, and each step is an action: -## Use case 1: Your product has a UI, and your docs walk users through it +```json +{ + "tests": [ + { + "steps": [ + { "goTo": "https://myapp.example.com/login" }, + { "find": { "selector": "#email", "click": true } }, + { "type": "user@example.com" }, + { "find": { "selector": "#password", "click": true } }, + { "type": "correct-horse-battery-staple" }, + { "click": "button[type=submit]" }, + { "find": "Welcome back" }, + { "screenshot": "dashboard.png" } + ] + } + ] +} +``` -Your docs say: go to this page, click this button, fill in this field, see this result. Doc Detective does exactly that. It navigates to a URL with `goTo`, finds elements with `find` (by CSS selector, text content, ARIA label, test ID, whatever matches your situation), clicks them, types into input fields, and takes screenshots. +You can also embed tests directly in Markdown as comments, or use automatic +markup detection. Doc Detective can parse the natural language of your docs. For +example if you write "Go to [the dashboard](https://app.example.com/)", Doc +Detective will parse and perform the corresponding test action. -It handles the messiness of real web applications too. You can configure wait conditions that monitor network activity and DOM mutations, so the test only proceeds once the page has actually finished loading. Useful when your single-page app is still fetching data three seconds after navigation. +## Where does it fit in your workflow? -The screenshot capability goes further than just capturing images. Doc Detective compares screenshots against reference images pixel by pixel. You set a variation threshold (say 5%), and if the current state of the UI has drifted beyond that, the test flags it. So when someone changes the product and the docs still show the old screenshots, you find out from your test suite, not from a confused user filing a support ticket. +Doc Detective runs from the command line. Put it in your CI/CD pipeline. Every +time someone changes the docs or the product, the doc tests run. When you run +Doc Detective in CI, doc test failures appear in the same pipeline results as +your code test failures. The coupling between docs and product becomes something +you actively monitor, not something that silently degrades. -## Use case 2: Your product is API-driven +## My Final thoughts -API documentation is full of example requests and expected responses. These are promises to your users: "Send this, get that back." When someone changes the API and forgets the docs, those promises fail silently. +I hate it when my investment in good documentation is squandered when the docs fall +out of sync with the product. Doc Detective protects that investment by testing +the coupling continuously. It won't write your docs for you, but it will tell +you, with certainty, when they stop being true. -Doc Detective's `httpRequest` action lets you define requests and validate responses: status codes, headers, body fields. If you maintain an OpenAPI specification, it can pull examples directly from the spec, validate request bodies against your schemas, and verify that responses match what the spec defines. That gives you a feedback loop between the spec, the docs, and the actual API. +## Appendix A: Use Cases -## Use case 3: Your docs include shell commands or code snippets +I wanted you some concrete ideas of how to use Doc Detective becuase it can do +quite a lot! So don't be overwhelmed by the list below. -Every tutorial that says "run this command and you should see this output" is making a testable claim. With `runShell` and `runCode`, Doc Detective executes those commands (Python, JavaScript, Bash) and checks the results: exit codes, stdout matching (literal or regex), stderr. When someone changes the CLI output format and doesn't update the tutorial, the test catches it. +### Use case 1: Your product has a UI, and your docs walk users through it -## Use case 4: Cross-browser and cross-platform verification +Your docs say: go to this page, click this button, fill in this field, see this +result. Doc Detective does exactly that. It navigates to a URL with `goTo`, +finds elements with `find` (by CSS selector, text content, ARIA label, test ID, +whatever matches your situation), clicks them, types into input fields, and +takes screenshots. -Doc Detective supports Firefox, Chrome, and Safari. You define multiple contexts for a test (Chrome on macOS, Firefox on Linux) and the same specification runs across all of them. If your docs describe a flow that works in Chrome but fails in Firefox, you will know. +It handles the messiness of real web applications too. You can configure wait +conditions that monitor network activity and DOM mutations, so the test only +proceeds once the page has actually finished loading. Useful when your +single-page app is still fetching data three seconds after navigation. -## Use case 5: Authenticated flows +The screenshot capability goes further than just capturing images. Doc Detective +compares screenshots against reference images pixel by pixel. You set a +variation threshold (say 5%), and if the current state of the UI has drifted +beyond that, the test flags it. So when someone changes the product and the docs +still show the old screenshots, you find out from your test suite, not from a +confused user filing a support ticket. -Most useful documentation covers features behind a login. Doc Detective provides building blocks for handling auth: you can use `before` and `beforeAny` hooks to run login steps before your tests, and `saveCookie` / `loadCookie` actions to manage cookies. +### Use case 2: Your product is API-driven -The challenge is that each Doc Detective test starts in a fresh browser instance. That means you will be logging in over and over again unless you find a way to persist authentication state between tests. How hard that is depends entirely on your auth provider. Some providers make it straightforward to save and reload session cookies. Others make it genuinely difficult. +API documentation is full of example requests and expected responses. These are +promises to your users: "Send this, get that back." When someone changes the API +and forgets the docs, those promises fail silently. -For example, if you use Clerk, their auth cookies have a 60-second lifetime and are refreshed by client-side JavaScript. This is how Clerk implements SSO sign-out — since they cannot push token revocation to your servers, they keep cookie lifetimes very short and rely on the browser to refresh them. That design is reasonable from Clerk's perspective, but it means you cannot simply save a cookie and reload it in a new browser instance. You will need to come up with a workaround, like re-authenticating at the start of each test or finding a way to inject a valid session outside of Clerk's normal flow. +Doc Detective's `httpRequest` action lets you define requests and validate +responses: status codes, headers, body fields. If you maintain an OpenAPI +specification, it can pull examples directly from the spec, validate request +bodies against your schemas, and verify that responses match what the spec +defines. That gives you a feedback loop between the spec, the docs, and the +actual API. -This is an area where you will need to invest some effort to match your specific auth setup. Doc Detective gives you the primitives, but the solution depends on your provider. +## Use case 3: Your docs include shell commands or code snippets -## Use case 6: Visual regression for documentation screenshots +Every tutorial that says "run this command and you should see this output" is +making a testable claim. With `runShell` and `runCode`, Doc Detective executes +those commands (Python, JavaScript, Bash) and checks the results: exit codes, +stdout matching (literal or regex), stderr. When someone changes the CLI output +format and doesn't update the tutorial, the test catches it. -You have dozens or hundreds of screenshots in your docs. A new release ships. Which screenshots are now wrong? Without tooling, someone has to check each one manually. With Doc Detective, every screenshot is retaken during the test run and compared against the reference. If the UI has changed beyond your threshold, the test flags it. You can even set it to auto-update reference images, so your docs stay current without manual intervention. +### Use case 4: Cross-browser and cross-platform verification -## Use case 7: Recording procedures +Doc Detective supports Firefox, Chrome, and Safari. You define multiple contexts +for a test (Chrome on macOS, Firefox on Linux) and the same specification runs +across all of them. If your docs describe a flow that works in Chrome but fails +in Firefox, you will know. -Doc Detective can record browser sessions as WebM video. Start a recording, run through a documented procedure, stop the recording. You get an up-to-date video artifact every time the tests run. +### Use case 5: Authenticated flows -## How tests are structured +Most useful documentation covers features behind a login. Doc Detective provides +building blocks for handling auth: you can use `before` and `beforeAny` hooks to +run login steps before your tests, and `saveCookie` / `loadCookie` actions to +manage cookies. -A specification contains tests, each test is a sequence of steps, and each step is an action: +The challenge is that each Doc Detective test starts in a fresh browser +instance. That means you will be logging in over and over again unless you find +a way to persist authentication state between tests. How hard that is depends +entirely on your auth provider. Some providers make it straightforward to save +and reload session cookies. Others make it genuinely difficult. -```json -{ - "tests": [ - { - "steps": [ - { "goTo": "https://myapp.example.com/login" }, - { "find": { "selector": "#email", "click": true } }, - { "type": "user@example.com" }, - { "find": { "selector": "#password", "click": true } }, - { "type": "correct-horse-battery-staple" }, - { "click": "button[type=submit]" }, - { "find": "Welcome back" }, - { "screenshot": "dashboard.png" } - ] - } - ] -} -``` +For example, if you use Clerk, their auth cookies have a 60-second lifetime and +are refreshed by client-side JavaScript. This is how Clerk implements SSO +sign-out — since they cannot push token revocation to your servers, they keep +cookie lifetimes very short and rely on the browser to refresh them. That design +is reasonable from Clerk's perspective, but it means you cannot simply save a +cookie and reload it in a new browser instance. You will need to come up with a +workaround, like re-authenticating at the start of each test or finding a way to +inject a valid session outside of Clerk's normal flow. -You can also embed tests directly in Markdown as comments, or use automatic markup detection. Doc Detective can parse instructions like "Go to [the dashboard](https://app.example.com/)" and generate the corresponding test action from the Markdown itself. +This is an area where you will need to invest some effort to match your specific +auth setup. Doc Detective gives you the primitives, but the solution depends on +your provider. -## Where does it fit in your workflow? +### Use case 6: Visual regression for documentation screenshots -Doc Detective runs from the command line. Put it in your CI/CD pipeline. Every time someone changes the docs or the product, the doc tests run. Failures show up alongside your code test failures. The coupling between docs and product becomes something you actively monitor, not something that silently degrades. +You have dozens or hundreds of screenshots in your docs. A new release ships. Which screenshots are now wrong? Without tooling, someone has to check each one manually. With Doc Detective, every screenshot is retaken during the test run and compared against the reference. If the UI has changed beyond your threshold, the test flags it. You can even set it to auto-update reference images, so your docs stay current without manual intervention. -## Final thoughts +### Use case 7: Recording procedures -The hard part of documentation is the thinking: understanding your users' problems and structuring information so they can find answers. That investment is wasted when the docs fall out of sync with the product. Doc Detective protects that investment by testing the coupling continuously. It won't write your docs for you, but it will tell you, with certainty, when they stop being true. +Doc Detective can record browser sessions as WebM video. Start a recording, run through a documented procedure, stop the recording. You get an up-to-date video artifact every time the tests run. \ No newline at end of file diff --git a/src/styles/custom.css b/src/styles/custom.css index 8f3b354a..b812871b 100644 --- a/src/styles/custom.css +++ b/src/styles/custom.css @@ -465,6 +465,12 @@ starlight-theme-select { max-width: 100%; } + /* Blog posts and other pages without sidebar: limit content width for comfortable reading */ + :root:not([data-has-sidebar]) .content-panel > .sl-container { + max-width: 72ch; + margin-inline: auto; + } + :root[data-has-sidebar][data-has-toc] .right-sidebar-panel .sl-container { width: calc(var(--pl-right-sidebar-width) - 2 * var(--sl-sidebar-pad-x)); max-width: calc( From 6fc1e8678b4ada00df7a7aa80c11b3ebd898523b Mon Sep 17 00:00:00 2001 From: Inline Pizza <249805557+InlinePizza@users.noreply.github.com> Date: Tue, 17 Mar 2026 14:14:58 -0700 Subject: [PATCH 5/6] fixup! fix: shorten article, move use case list to appendix --- .../blog/product-updates/doc-detective.mdx | 37 +++++++------------ 1 file changed, 14 insertions(+), 23 deletions(-) diff --git a/src/content/blog/product-updates/doc-detective.mdx b/src/content/blog/product-updates/doc-detective.mdx index db1e5f7b..d92b5bd3 100644 --- a/src/content/blog/product-updates/doc-detective.mdx +++ b/src/content/blog/product-updates/doc-detective.mdx @@ -50,27 +50,19 @@ suggestion! ## How tests are structured -A specification contains tests, each test is a sequence of steps, and each step is an action: - -```json -{ - "tests": [ - { - "steps": [ - { "goTo": "https://myapp.example.com/login" }, - { "find": { "selector": "#email", "click": true } }, - { "type": "user@example.com" }, - { "find": { "selector": "#password", "click": true } }, - { "type": "correct-horse-battery-staple" }, - { "click": "button[type=submit]" }, - { "find": "Welcome back" }, - { "screenshot": "dashboard.png" } - ] - } - ] -} +Here's an example test from our own Slack Integration docs page: + +```mdx +{/* test {"testId": "slack-integration-content"} */} +## Installation + +{/* step {"checkLink": {"url": "https://app.gopromptless.ai/integrations"}} */} +1. Click "Connect Slack" from the [integrations page](https://app.gopromptless.ai/integrations). ``` +See the full source file in our docs repo: +[https://github.com/Promptless/docs/blob/main/src/content/docs/docs/integrations/slack-integration.mdx](https://github.com/Promptless/docs/blob/main/src/content/docs/docs/integrations/slack-integration.mdx). + You can also embed tests directly in Markdown as comments, or use automatic markup detection. Doc Detective can parse the natural language of your docs. For example if you write "Go to [the dashboard](https://app.example.com/)", Doc @@ -86,10 +78,9 @@ you actively monitor, not something that silently degrades. ## My Final thoughts -I hate it when my investment in good documentation is squandered when the docs fall -out of sync with the product. Doc Detective protects that investment by testing -the coupling continuously. It won't write your docs for you, but it will tell -you, with certainty, when they stop being true. +Doc Detective protects your investment in great documentation by testing the +coupling continuously. It won't write your docs for you, but it will tell you, +with certainty, when they stop being true. ## Appendix A: Use Cases From 5281302bf89420032459ac3999ce829268a16d41 Mon Sep 17 00:00:00 2001 From: Inline Pizza <249805557+InlinePizza@users.noreply.github.com> Date: Thu, 19 Mar 2026 10:26:41 -0700 Subject: [PATCH 6/6] fixup! fixup! fix: shorten article, move use case list to appendix --- .../blog/product-updates/doc-detective.mdx | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/src/content/blog/product-updates/doc-detective.mdx b/src/content/blog/product-updates/doc-detective.mdx index d92b5bd3..2aa350a9 100644 --- a/src/content/blog/product-updates/doc-detective.mdx +++ b/src/content/blog/product-updates/doc-detective.mdx @@ -33,8 +33,8 @@ product. It opens a browser, navigates pages, clicks elements, fills in forms, takes screenshots, makes API calls, runs shell commands. If what happens doesn't match what your docs say should happen, the test fails. -Think of it as end-to-end testing, but the source of truth is your documentation -rather than a product spec. +It's end-to-end testing, where the source of truth is your documentation +rather than what the engineers integration. ## Why should you test your docs? @@ -120,7 +120,7 @@ bodies against your schemas, and verify that responses match what the spec defines. That gives you a feedback loop between the spec, the docs, and the actual API. -## Use case 3: Your docs include shell commands or code snippets +### Use case 3: Your docs include shell commands or code snippets Every tutorial that says "run this command and you should see this output" is making a testable claim. With `runShell` and `runCode`, Doc Detective executes @@ -163,8 +163,15 @@ your provider. ### Use case 6: Visual regression for documentation screenshots -You have dozens or hundreds of screenshots in your docs. A new release ships. Which screenshots are now wrong? Without tooling, someone has to check each one manually. With Doc Detective, every screenshot is retaken during the test run and compared against the reference. If the UI has changed beyond your threshold, the test flags it. You can even set it to auto-update reference images, so your docs stay current without manual intervention. +You have dozens or hundreds of screenshots in your docs. A new release ships. +Which screenshots are now wrong? Without tooling, someone has to check each one +manually. With Doc Detective, every screenshot is retaken during the test run +and compared against the reference. If the UI has changed beyond your threshold, +the test flags it. You can even set it to auto-update reference images, so your +docs stay current without manual intervention. ### Use case 7: Recording procedures -Doc Detective can record browser sessions as WebM video. Start a recording, run through a documented procedure, stop the recording. You get an up-to-date video artifact every time the tests run. \ No newline at end of file +Doc Detective can record browser sessions as WebM video. Start a recording, run +through a documented procedure, stop the recording. You get an up-to-date video +artifact every time the tests run. \ No newline at end of file