chore: add missing sections (#263)

### TL;DR

Renamed "Configuring Retries" to "Retrying Steps" and added a new "Validation Steps" guide to improve documentation around error handling patterns.

### What changed?

- Renamed `/build/configuring-retries/` to `/build/retrying-steps/` for better clarity
- Added a new guide on `/build/validation-steps/` explaining how to properly handle validation errors
- Enhanced the retry documentation with sections on understanding different failure types
- Updated all internal links and navigation references to point to the renamed pages
- Refined the LLM content exclusion list in `astro.config.mjs` to be more specific about which index pages to exclude

### How to test?

1. Build the website and verify the new page structure:
   - Confirm `/build/retrying-steps/` loads correctly with the enhanced content
   - Verify the new `/build/validation-steps/` page is accessible
   - Check that all links to the old "Configuring Retries" page redirect properly

2. Review the content of both pages to ensure they provide clear guidance on:
   - Distinguishing between transient and permanent failures
   - Setting up validation steps with `maxAttempts: 1`
   - Following best practices for synchronous validation

### Why make this change?

Users were confused about when to use retries versus when to fail fast. The documentation needed clearer guidance on handling different types of errors appropriately. By separating validation patterns into their own guide and enhancing the retry documentation, we provide more specific guidance on error handling best practices, helping users build more robust flows that don't waste resources on unnecessary retries.

Author: jumski

pkgs/website/astro.config.mjs

@@ -152,15 +152,22 @@ export default defineConfig({
         starlightLlmsTxt({
           exclude: [
             'index',
-            '**/index',
+            // Navigation-only index files (pure CardGrid hubs with no unique content)
+            'build/index',
+            'concepts/index',
+            'deploy/index',
+            'reference/index',
+            'tutorials/index',
+            'comparisons/index',
+            // Tutorials (lengthy, patterns covered elsewhere)
             'tutorials/ai-web-scraper/*',
-            'concepts/naming-steps',
-            'deploy/tune-flow-config',
-            'get-started/faq',
+            // News/blog and non-technical content
             'news/**',
             'hire/**',
-            'build/configuring-retries',
-            'build/delaying-steps',
+            'author/**',
+            // Note: The following index files ARE included because they have valuable content:
+            // - build/starting-flows/index (comparison guide for starting flows)
+            // - reference/configuration/index (config philosophy and structure)
           ],
           promote: [
             'get-started/installation',
@@ -225,8 +232,12 @@ export default defineConfig({
                       link: '/build/organize-flow-code/',
                     },
                     {
-                      label: 'Configuring retries',
-                      link: '/build/configuring-retries/',
+                      label: 'Retrying steps',
+                      link: '/build/retrying-steps/',
+                    },
+                    {
+                      label: 'Validation steps',
+                      link: '/build/validation-steps/',
                     },
                     {
                       label: 'Delaying steps',

pkgs/website/src/content/docs/build/delaying-steps.mdx

@@ -10,7 +10,7 @@ import { Aside, CardGrid, LinkCard } from "@astrojs/starlight/components";
 Use [`startDelay`](/reference/configuration/step-execution/#startdelay) to schedule steps for execution after a specified time period. Delays are relative to when the step's dependencies complete.
 
 <Aside type="tip">
-For retry configuration patterns, see [Configuring Retries](/build/configuring-retries/).
+For retry configuration patterns, see [Retrying Steps](/build/retrying-steps/).
 </Aside>
 
 For detailed information about each configuration option, see the [Step Execution Options](/reference/configuration/step-execution/) reference.
@@ -82,8 +82,8 @@ new Flow({
     description="Complete reference for all configuration options and their defaults"
   />
   <LinkCard
-    title="Configuring Retries"
-    href="/build/configuring-retries/"
+    title="Retrying Steps"
+    href="/build/retrying-steps/"
     description="Configure retry policies for different reliability requirements"
   />
   <LinkCard

pkgs/website/src/content/docs/build/index.mdx

@@ -16,8 +16,8 @@ Now that you've created your first flow, learn how to structure your code, integ
     description="Structure your flows and tasks for maintainability and reusability"
   />
   <LinkCard
-    title="Configuring retries"
-    href="/build/configuring-retries/"
+    title="Retrying steps"
+    href="/build/retrying-steps/"
     description="Configure retry policies for different step reliability requirements"
   />
   <LinkCard

pkgs/website/src/content/docs/build/configuring-retries.mdx renamed to pkgs/website/src/content/docs/build/retrying-steps.mdx

@@ -1,5 +1,5 @@
 ---
-title: Configuring Retries
+title: Retrying Steps
 description: Configure retry policies for different step reliability requirements.
 sidebar:
   order: 25
@@ -15,6 +15,56 @@ For scheduling delays between steps, see [Delaying Steps](/build/delaying-steps/
 
 For detailed information about each configuration option, see the [Step Execution Options](/reference/configuration/step-execution/) reference.
 
+## Understanding Failure Types
+
+Not all failures should retry. Understanding the difference helps you configure appropriate retry policies.
+
+### Transient Failures
+
+Temporary problems that might succeed on retry:
+- Network timeouts
+- Rate limiting (429 responses)
+- Temporary service unavailability (503 responses)
+- Database connection issues
+
+Configure with retries:
+```typescript
+.step({
+  slug: 'fetchExternalData',
+  maxAttempts: 5,   // Retry transient failures
+  baseDelay: 2,
+}, async (input) => await fetchFromAPI(input.run.url))
+```
+
+### Permanent Failures
+
+Problems that will never succeed on retry:
+- Invalid input format (malformed email, negative numbers)
+- Missing required fields
+- Business rule violations
+- Schema validation errors
+
+Configure without retries:
+```typescript
+.step({
+  slug: 'validInput',
+  maxAttempts: 1,   // No retries for validation
+}, (input) => {
+  if (!input.run.email) throw new Error('Email required');
+  return input.run;
+})
+```
+
+<Aside type="note">
+`maxAttempts: 1` means "run once, do not retry". If the step fails, it fails immediately without retry attempts.
+</Aside>
+
+<Aside type="caution" title="Current Limitation">
+pgflow does not distinguish between transient and permanent failures automatically. All exceptions trigger retry logic based on `maxAttempts`. Use `maxAttempts: 1` for steps that perform validation or other operations that should fail fast.
+</Aside>
+
+For detailed guidance on validation patterns, see [Validation Steps](/build/validation-steps/).
+
 ## Guiding Principle
 
 **Set conservative flow-level defaults, override per-step as needed.**
@@ -56,6 +106,11 @@ new Flow({
 ## Learn More
 
 <CardGrid>
+  <LinkCard
+    title="Validation Steps"
+    href="/build/validation-steps/"
+    description="Use explicit validation steps to fail fast on invalid input"
+  />
   <LinkCard
     title="Step Execution Options"
     href="/reference/configuration/step-execution/"

pkgs/website/src/content/docs/build/validation-steps.mdx

@@ -0,0 +1,87 @@
+---
+title: Validation Steps
+description: Use explicit validation steps to fail fast on invalid input
+sidebar:
+  order: 30
+---
+
+import { Aside } from '@astrojs/starlight/components';
+
+Validation steps catch invalid input early, before expensive operations run. Structure them to fail fast with no retries.
+
+## Why Explicit Validation Steps?
+
+pgflow retries all exceptions based on `maxAttempts`. Without explicit validation, input errors waste retry attempts:
+
+```typescript
+// Without validation - wastes retry attempts on invalid input
+new Flow<{ email: string }>({ slug: 'sendEmail', maxAttempts: 5 })
+  .step(
+    { slug: 'send' },
+    async (input) => {
+      if (!input.run.email.includes('@')) {
+        throw new Error('Invalid email');  // Retries 5 times!
+      }
+      return await sendEmail(input.run.email);
+    }
+  )
+```
+
+With explicit validation, failures stop immediately:
+
+```typescript
+// With validation - fails immediately on invalid input
+new Flow<{ email: string }>({ slug: 'sendEmail' })
+  .step(
+    { slug: 'validInput', maxAttempts: 1 },
+    (input) => {
+      if (!input.run.email.includes('@')) {
+        throw new Error('Invalid email');
+      }
+      return input.run;
+    }
+  )
+  .step(
+    { slug: 'send', dependsOn: ['validInput'], maxAttempts: 5 },
+    async (input) => await sendEmail(input.validInput.email)
+  )
+```
+
+## Keep Validation Synchronous
+
+Validation steps should be fast, synchronous functions that check input format and structure. Avoid async operations like database queries or API calls - those belong in separate steps with appropriate retry configuration.
+
+```typescript
+// Good: Fast, synchronous validation
+.step(
+  { slug: 'validOrder', maxAttempts: 1 },
+  (input) => {
+    const { amount, items } = input.run;
+
+    if (amount <= 0) throw new Error('amount must be positive');
+    if (!items?.length) throw new Error('items cannot be empty');
+
+    return input.run;
+  }
+)
+
+// Bad: Async checks in validation
+.step(
+  { slug: 'validCustomer', maxAttempts: 1 },
+  async (input) => {
+    // Database lookups belong in separate steps with retries
+    const exists = await checkCustomerExists(input.run.customerId);
+    if (!exists) throw new Error('Customer not found');
+    return input.run;
+  }
+)
+```
+
+<Aside type="tip">
+Validation steps check **format and structure** (email format, positive numbers, array length). Existence checks and external validations belong in regular steps with retry configuration.
+</Aside>
+
+## See Also
+
+- [Retrying Steps](/build/retrying-steps/) - Understanding failure types and retry policies
+- [Context Object](/concepts/context-object/) - Accessing validated input in dependent steps

pkgs/website/src/content/docs/index.mdx

@@ -372,7 +372,7 @@ npx pgflow@latest install
   </Card>
 
   <Card title="🔄 Automatic retries">
-    Built-in retry logic with exponential backoff for flaky AI APIs. When OpenAI times out or rate-limits, only that step retries - your workflow continues. Configure max attempts and delays per step, no retry code needed. [Learn more →](/build/configuring-retries/)
+    Built-in retry logic with exponential backoff for flaky AI APIs. When OpenAI times out or rate-limits, only that step retries - your workflow continues. Configure max attempts and delays per step, no retry code needed. [Learn more →](/build/retrying-steps/)
   </Card>
 
   <Card title="⚡ Parallel array processing">

pkgs/website/src/content/docs/reference/configuration/index.mdx

@@ -44,9 +44,9 @@ new Flow({ slug: 'my_flow' })
 
 <CardGrid>
   <LinkCard
-    title="Configuring Retries"
+    title="Retrying Steps"
     description="Configure retry policies for different step reliability requirements."
-    href="/build/configuring-retries/"
+    href="/build/retrying-steps/"
   />
   <LinkCard
     title="Delaying Steps"