Open
Conversation
Announce Doc Detective support in Promptless as a product update.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
hawkeyexl
suggested changes
Mar 13, 2026
hawkeyexl
left a comment
There was a problem hiding this comment.
I like it overall, but it doesn't address why you might need Doc Detective is you already have Promptless, how they're two sides of the same coin. Promptless reduces drift, but Doc Detective verified that any content, human-created or AI-generated, is accurate.
|
|
||
| It handles the messiness of real web applications too. You can configure wait conditions that monitor network activity and DOM mutations, so the test only proceeds once the page has actually finished loading. Useful when your single-page app is still fetching data three seconds after navigation. | ||
|
|
||
| The screenshot capability goes further than just capturing images. Doc Detective compares screenshots against reference images pixel by pixel. You set a variation threshold (say 5%), and if the current state of the UI has drifted beyond that, the test flags it. So when someone changes the product and the docs still show the old screenshots, you find out from your test suite, not from a confused user filing a support ticket. |
There was a problem hiding this comment.
Repeated in its own section later on.
|
|
||
| A specification contains tests, each test is a sequence of steps, and each step is an action: | ||
|
|
||
| ```json |
There was a problem hiding this comment.
This whole object is completely wrong. We haven't used the "action" key since v2.
| } | ||
| ``` | ||
|
|
||
| You can also embed tests directly in Markdown as comments, or use automatic markup detection. Doc Detective can parse instructions like "Go to [the dashboard](https://app.example.com/)" and generate the corresponding test action from the Markdown itself. |
There was a problem hiding this comment.
You're likely going to need code syntax here so the markdown doesn't render.
|
|
||
| ## Where does it fit in your workflow? | ||
|
|
||
| Doc Detective runs from the command line. Put it in your CI/CD pipeline. Every time someone changes the docs or the product, the doc tests run. Failures show up alongside your code test failures. The coupling between docs and product becomes something you actively monitor, not something that silently degrades. |
prithvi2206
reviewed
Mar 17, 2026
Comment on lines
+26
to
+46
| ## If you have Promptless, why do you need Doc Detective? | ||
|
|
||
| Promptless fixes docs drift. When a pull request renames an API field or changes | ||
| a workflow, Promptless identifies what needs to be updated in the docs before the | ||
| code ships. It monitors pull requests, Slack, Discord, and support channels so | ||
| your writers do not have to manually track every engineering change. The most | ||
| valuable thing Promptless does is catch drift before it reaches your customers. | ||
|
|
||
| But products are complex and teams move fast. Not every change that affects the | ||
| docs will be caught before it ships. Doc Detective is your safety net. It takes | ||
| the procedures your docs describe ("click this button," "send this request," | ||
| "run this command") and executes them against the real product. If the result | ||
| does not match what the docs say, the test fails. No interpretation, no | ||
| inference. It is a test suite for your documentation, closer to Cucumber than to | ||
| an AI agent, and cheap enough to run on every PR or on a schedule. | ||
|
|
||
| When a Doc Detective test fails, that is definitive proof that your docs have | ||
| drifted. Promptless picks up that failure, figures out what the documentation | ||
| needs to say to match the behavior Doc Detective actually observed, and makes | ||
| the fix. Promptless also uses your product directly to write new Doc Detective | ||
| tests, so your test coverage grows as your docs grow. |
Contributor
There was a problem hiding this comment.
Suggested change
| ## If you have Promptless, why do you need Doc Detective? | |
| Promptless fixes docs drift. When a pull request renames an API field or changes | |
| a workflow, Promptless identifies what needs to be updated in the docs before the | |
| code ships. It monitors pull requests, Slack, Discord, and support channels so | |
| your writers do not have to manually track every engineering change. The most | |
| valuable thing Promptless does is catch drift before it reaches your customers. | |
| But products are complex and teams move fast. Not every change that affects the | |
| docs will be caught before it ships. Doc Detective is your safety net. It takes | |
| the procedures your docs describe ("click this button," "send this request," | |
| "run this command") and executes them against the real product. If the result | |
| does not match what the docs say, the test fails. No interpretation, no | |
| inference. It is a test suite for your documentation, closer to Cucumber than to | |
| an AI agent, and cheap enough to run on every PR or on a schedule. | |
| When a Doc Detective test fails, that is definitive proof that your docs have | |
| drifted. Promptless picks up that failure, figures out what the documentation | |
| needs to say to match the behavior Doc Detective actually observed, and makes | |
| the fix. Promptless also uses your product directly to write new Doc Detective | |
| tests, so your test coverage grows as your docs grow. | |
| We see Promptless and Doc Detective as two sides of the same coin. Promptless helps prevent and remediate docs drift, and Doc Detective _deterministically validates_ that any content—whether written by Promptless or a human—is faithful to the product that it documents. | |
| Companies that are the most serious about guaranteeing the accuracy of their docs should use Promptless and Doc Detective together, and now Promptless users can get Doc Detective tests created automatically with every Promptless suggestion! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan