Repository navigation
Serve public site content as Markdown for AI agents #192
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| import { getSettings } from "@/lib/api"; | ||
| import { notFoundMarkdown, renderMarkdown } from "@/lib/markdown"; | ||
| import { markdownUrl } from "@/lib/markdown-negotiation"; | ||
|
|
||
| /** | ||
| * Markdown for AI agents. The proxy rewrites here for `Accept: | ||
| * text/markdown` and `<path>.md` (see lib/markdown.ts); `?md=url` marks a | ||
| * `.md` request so redirects stay on `.md` URLs. | ||
| */ | ||
| export async function GET( | ||
| request: Request, | ||
| { params }: { params: Promise<{ teamId: string; path?: string[] }> }, | ||
| ) { | ||
| const { teamId, path: segments = [] } = await params; | ||
| const path = `/${segments.join("/")}`; | ||
| const headers = { "content-type": "text/markdown; charset=utf-8" }; | ||
| const markdown = await renderMarkdown(teamId, path); | ||
| if (markdown) return new Response(markdown, { headers }); | ||
|
|
||
| // Redirects from site.json only apply where no page exists. | ||
| const settings = await getSettings(teamId); | ||
| const match = settings?.redirects?.find( | ||
| (redirect) => (redirect.from.replace(/\/+$/, "") || "/") === path, | ||
| ); | ||
| if (match) { | ||
| const viaUrl = new URL(request.url).searchParams.get("md") === "url"; | ||
| const location = | ||
| viaUrl && match.to.startsWith("/") ? markdownUrl(match.to) : match.to; | ||
| return new Response(null, { | ||
| status: match.permanent === false ? 307 : 308, | ||
| headers: { location }, | ||
| }); | ||
| } | ||
| return new Response(notFoundMarkdown(path), { status: 404, headers }); | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| import { getApiReference, getSettings, getSitemap } from "@/lib/api"; | ||
| import { buildLlmsTxt } from "@/lib/markdown"; | ||
|
|
||
| /** `/llms.txt`: an index of the site's public content for AI agents. */ | ||
| export async function GET( | ||
| _request: Request, | ||
| { params }: { params: Promise<{ teamId: string }> }, | ||
| ) { | ||
| const { teamId } = await params; | ||
| const [entries, settings, reference] = await Promise.all([ | ||
| getSitemap(teamId), | ||
| getSettings(teamId), | ||
| getApiReference(teamId), | ||
| ]); | ||
| return new Response( | ||
| buildLlmsTxt(settings, settings?.hideFromSearch ? [] : entries, Boolean(reference)), | ||
|
There was a problem hiding this comment.
When Useful? React with 👍 / 👎. |
||
| { headers: { "content-type": "text/plain; charset=utf-8" } }, | ||
| ); | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| import { describe, expect, it } from "vitest"; | ||
| import { | ||
| isNegotiablePath, | ||
| markdownSourcePath, | ||
| markdownUrl, | ||
| prefersMarkdown, | ||
| } from "./markdown-negotiation"; | ||
|
|
||
| describe("prefersMarkdown", () => { | ||
| it("serves Markdown only when asked for at least as strongly as HTML", () => { | ||
| expect(prefersMarkdown("text/markdown")).toBe(true); | ||
| expect(prefersMarkdown("text/markdown, text/html;q=0.9")).toBe(true); | ||
| expect(prefersMarkdown("text/html, text/markdown")).toBe(true); | ||
| expect(prefersMarkdown("text/html, text/markdown;q=0.5")).toBe(false); | ||
| expect(prefersMarkdown("text/markdown;q=0")).toBe(false); | ||
| }); | ||
|
|
||
| it("keeps browsers on HTML", () => { | ||
| expect( | ||
| prefersMarkdown( | ||
| "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,*/*;q=0.8", | ||
| ), | ||
| ).toBe(false); | ||
| expect(prefersMarkdown("*/*")).toBe(false); | ||
| expect(prefersMarkdown(null)).toBe(false); | ||
| }); | ||
| }); | ||
|
|
||
| describe("markdown paths", () => { | ||
| it("maps .md URLs to their page", () => { | ||
| expect(markdownSourcePath("/docs/intro.md")).toBe("/docs/intro"); | ||
| expect(markdownSourcePath("/index.md")).toBe("/"); | ||
| expect(markdownSourcePath("/docs/intro")).toBeNull(); | ||
| expect(markdownUrl("/")).toBe("/index.md"); | ||
| expect(markdownUrl("/blog/post")).toBe("/blog/post.md"); | ||
| }); | ||
|
|
||
| it("negotiates pages only", () => { | ||
| expect(isNegotiablePath("/")).toBe(true); | ||
| expect(isNegotiablePath("/docs/intro")).toBe(true); | ||
| expect(isNegotiablePath("/sitemap.xml")).toBe(false); | ||
| expect(isNegotiablePath("/blog/rss.xml")).toBe(false); | ||
| expect(isNegotiablePath("/help/tickets/t_1")).toBe(false); | ||
| expect(isNegotiablePath("/embed/chat")).toBe(false); | ||
| }); | ||
| }); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| /** | ||
| * When a request gets Markdown instead of HTML (see lib/markdown.ts). Kept | ||
| * free of imports: the proxy runs it on every request. | ||
| */ | ||
|
|
||
| /** Route segment, under `/sites/[teamId]`, of the Markdown handler. */ | ||
| export const MARKDOWN_ROUTE = "agent-markdown"; | ||
|
|
||
| /** True when the Accept header ranks Markdown at least as high as HTML. | ||
| * Browsers never list `text/markdown`, so they keep getting HTML. */ | ||
| export function prefersMarkdown(accept: string | null): boolean { | ||
| if (!accept) return false; | ||
| let markdown = 0; | ||
| let html = 0; | ||
| for (const part of accept.split(",")) { | ||
| const [type = "", ...params] = part.split(";").map((value) => value.trim()); | ||
| const qParam = params.find((param) => param.startsWith("q=")); | ||
| const q = qParam ? Number(qParam.slice(2)) : 1; | ||
| if (Number.isNaN(q)) continue; | ||
| const mediaType = type.toLowerCase(); | ||
| if (mediaType === "text/markdown" || mediaType === "text/x-markdown") | ||
| markdown = Math.max(markdown, q); | ||
| else if (mediaType === "text/html") html = Math.max(html, q); | ||
| } | ||
| return markdown > 0 && markdown >= html; | ||
| } | ||
|
|
||
| /** Paths whose response depends on Accept: pages, not files or private | ||
| * routes. `.md` URLs, feeds and assets have one representation. */ | ||
| export function isNegotiablePath(pathname: string): boolean { | ||
| if (/\.[a-z0-9]+$/i.test(pathname)) return false; | ||
| return !/^\/(_preview|embed|help\/tickets|opengraph-image)(\/|$)/.test(pathname); | ||
| } | ||
|
|
||
| /** `/docs/intro.md` → `/docs/intro`; `/index.md` → `/`. Null when the path | ||
| * isn't a `.md` URL. */ | ||
| export function markdownSourcePath(pathname: string): string | null { | ||
| if (!pathname.endsWith(".md")) return null; | ||
| const path = pathname.slice(0, -".md".length); | ||
| return path === "/index" || path === "" ? "/" : path; | ||
| } | ||
|
|
||
| /** The `.md` URL of a public path, for links agents follow. */ | ||
| export function markdownUrl(path: string): string { | ||
| return path === "/" ? "/index.md" : `${path.replace(/\/+$/, "")}.md`; | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When a
site.jsonredirect target contains a query string,markdownUrl(match.to)appends.mdto the whole string (/new?lang=enbecomes/new?lang=en.md). The redirected URL no longer has a.mdpathname, so the destination is served as HTML instead of Markdown. Add the suffix to the target pathname while preserving its query and fragment.Useful? React with 👍 / 👎.