Skip to content

INDEXNOW_GUIDE.md

Latest commit

 

History

History
215 lines (149 loc) · 6.06 KB

INDEXNOW_GUIDE.md

File metadata and controls

215 lines (149 loc) · 6.06 KB

IndexNow Setup and Usage Guide

IndexNow is a protocol that allows websites to instantly notify search engines about content updates. This implementation supports Bing, Yandex, and all other search engines that participate in the IndexNow protocol.

What's Been Set Up

1. API Key File

  • Location: /public/51c2d562781e4992ab751d34d23f9ab7.txt
  • Purpose: Verifies your ownership of the domain to search engines
  • Key: 51c2d562781e4992ab751d34d23f9ab7

2. Utility Functions

  • Location: /lib/indexnow.ts
  • Functions:
    • submitUrlToIndexNow(url: string) - Submit a single URL
    • submitUrlsToIndexNow(urls: string[]) - Submit multiple URLs (up to 10,000 per batch)
    • submitCommonPages() - Submit frequently updated pages

3. API Route

  • Location: /app/api/indexnow/route.ts
  • Endpoints:
    • GET /api/indexnow - Verify IndexNow configuration
    • POST /api/indexnow - Submit URLs programmatically

4. Bulk Submission Script

  • Location: /scripts/submit-to-indexnow.ts
  • Purpose: Submit all sitemap URLs to search engines at once

How to Use

Option 1: Bulk Submit All Pages (Recommended for Initial Setup)

Run this command to submit all your sitemap URLs to search engines:

```bash npm run indexnow ```

or

```bash pnpm indexnow ```

This will:

  1. Fetch all URLs from your sitemap (https://www.bigjson.online/sitemap.xml)
  2. Submit them to Bing, Yandex, and the central IndexNow API
  3. Display the results for each search engine

Option 2: Submit Individual URLs Programmatically

Use the API route to submit URLs when content changes:

```typescript // Submit a single URL await fetch('/api/indexnow', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ url: 'https://www.bigjson.online/en/new-blog-post' }) });

// Submit multiple URLs await fetch('/api/indexnow', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ urls: [ 'https://www.bigjson.online/en/post-1', 'https://www.bigjson.online/en/post-2', 'https://www.bigjson.online/en/post-3' ] }) }); ```

Option 3: Use Utility Functions Directly

Import and use the utility functions in your code:

```typescript import { submitUrlToIndexNow, submitCommonPages } from '@/lib/indexnow';

// Submit when creating/updating a blog post async function publishBlogPost(slug: string, lang: string) { // ... your publishing logic ...

// Notify search engines await submitUrlToIndexNow(`https://www.bigjson.online/\${lang}/\${slug}\`); }

// Submit common pages after deployment async function afterDeploy() { await submitCommonPages(); } ```

Option 4: Verify Configuration

Check if IndexNow is properly configured:

```bash curl https://www.bigjson.online/api/indexnow ```

Or visit: https://www.bigjson.online/api/indexnow in your browser

When to Submit URLs

Recommended Triggers:

  1. After deployment - Submit all pages using npm run indexnow
  2. When publishing new content - Submit the new URL immediately
  3. When updating existing content - Submit the updated URL
  4. Weekly/Monthly maintenance - Resubmit all pages periodically

Don't Submit:

  • Same URL multiple times in a short period (wait at least 24 hours)
  • Private/draft content
  • Pages you want to keep out of search engines

Search Engine Support

This implementation submits to three endpoints:

  1. api.indexnow.org - Central endpoint that distributes to all participating search engines
  2. bing.com/indexnow - Bing and Microsoft search
  3. yandex.com/indexnow - Yandex search

Response Codes

  • 200 OK - URL successfully submitted
  • 202 Accepted - URL received (will be processed later)
  • 400 Bad Request - Invalid URL format or missing key
  • 403 Forbidden - Key validation failed
  • 422 Unprocessable Entity - URL doesn't belong to the host
  • 429 Too Many Requests - Rate limit exceeded (wait and retry)

Verification

To verify your key file is accessible:

  1. Visit: https://www.bigjson.online/51c2d562781e4992ab751d34d23f9ab7.txt
  2. You should see: 51c2d562781e4992ab751d34d23f9ab7

Best Practices

  1. Don't spam - Only submit when content actually changes
  2. Batch submissions - Group multiple URLs when possible (more efficient)
  3. Monitor results - Check the API responses for errors
  4. Keep the key file - Never delete 51c2d562781e4992ab751d34d23f9ab7.txt
  5. Use after builds - Integrate into your CI/CD pipeline

Integration with CI/CD

Add to your deployment script:

```bash

After successful deployment

npm run build npm run indexnow ```

Or use Vercel deploy hooks:

```typescript // In your build script or API route if (process.env.VERCEL_ENV === 'production') { await submitCommonPages(); } ```

Troubleshooting

Key file not found (403 error)

URLs not being indexed

  • IndexNow notifies search engines, but doesn't guarantee immediate indexing
  • Allow 1-2 weeks for search engines to crawl and index
  • Ensure your robots.txt doesn't block search engines
  • Check sitemap.xml is properly formatted

Rate limiting (429 error)

  • Wait before retrying (typically 24 hours)
  • Reduce submission frequency
  • Use batch submissions instead of individual ones

Additional Resources

Support

For issues specific to this implementation, check:

  • API logs: Check your Next.js server logs
  • Search engine status: Use Bing Webmaster Tools and Yandex Webmaster
  • Test endpoint: GET /api/indexnow