Feature/final

.eslintrc.js

@@ -0,0 +1,53 @@
+module.exports = {
+  extends: [
+    'airbnb-typescript/base', // Use the base config for non-React projects
+    'prettier'
+  ],
+  parser: '@typescript-eslint/parser',
+  parserOptions: {
+    project: './tsconfig.json',
+    ecmaVersion: 2020,
+    sourceType: 'module',
+  },
+  plugins: [
+    '@typescript-eslint',
+    'prettier',
+    'import'
+  ],
+  rules: {
+    'prettier/prettier': 'error',
+  '@typescript-eslint/ban-types': 'error',
+  // These are intentionally relaxed to match the current codebase which
+  // contains many legacy `any` usages and some unused variables. We prefer
+  // warnings for now to avoid mass refactors.
+  '@typescript-eslint/no-explicit-any': 'off',
+  '@typescript-eslint/no-unused-vars': ['warn', {"argsIgnorePattern":"^_","varsIgnorePattern":"^_"}],
+    '@typescript-eslint/naming-convention': [
+      'error',
+      {
+        selector: 'variable',
+        format: ['camelCase', 'PascalCase', 'UPPER_CASE']
+      }
+    ],
+    // Allow importing TypeScript modules without requiring a .js extension
+    // in source files (common in TS projects targeting ESM output).
+    'import/extensions': [
+      'error',
+      'ignorePackages',
+      {
+        js: 'never',
+        ts: 'never'
+      }
+    ],
+    // Allow functions and declarations to be used before they're defined in
+    // some cases to avoid mass reordering refactors across the codebase.
+    '@typescript-eslint/no-use-before-define': 'off'
+  },
+  settings: {
+    'import/resolver': {
+      node: {
+        extensions: ['.js', '.jsx', '.ts', '.tsx'],
+      }
+    }
+  }
+};

.github/copilot-instructions.md

@@ -0,0 +1,194 @@
+# Copilot Instructions for USDA Nutrition API Worker
+
+## Project Overview
+Production-grade Cloudflare Worker for USDA nutritional data with advanced NLP search capabilities. Built for high performance with multi-layer caching (Edge → KV → Hot Cache → D1 → USDA API).
+
+**Key Features:**
+- Zero-cost NLP query parsing (local fuzzy matching, no external AI by default)
+- Multi-layer caching strategy (Edge, KV L1, Hot Cache for top 100 foods, D1 L0)
+- USDA Batch API service (fetch up to 20 foods in single call)
+- Credit-based quota system with tier-based rate limiting (free/pro/premium/enterprise)
+- Scheduled jobs (hourly keep-alive, daily cache warming, branded food fetcher)
+- Background task safety patterns with `safeBackgroundTask` wrappers
+
+## Architecture & Data Flow
+
+**Multi-Layer Cache Hierarchy:**
+```
+Request → Edge Cache (12h TTL, Cloudflare Edge)
+       → KV L1 Cache (30d TTL, hot foods)
+       → D1 Hot Cache (top 100 foods, <5ms)
+       → D1 L0 Cache (1h TTL, stale-while-revalidate)
+       → USDA Batch API (deduplication + batching)
+```
+
+**Request Lifecycle:**
+1. `src/index.ts` - Entry point, applies middleware chain:
+   - `withEdgeCache` (GET requests, 12h cache)
+   - `withLogging` (adds request IDs)
+   - `sanitizeHeaders` (hides infrastructure details)
+   - `withCors` (CORS handling)
+   - `withAuth` (API key validation via D1)
+   - `withRateLimiting` (tier-based limits)
+   - `validateRequest` (Zod schema validation)
+   - `createCreditCheck` (deducts credits per operation)
+
+2. Handler orchestration:
+   - `src/handlers/naturalLanguageSearchHandler.ts` - NLP query flow
+   - `src/handlers/foodHandlers.ts` - Food details, search, analyze
+   - `src/handlers/calculateHandler.ts` - Nutrition calculations
+   - `src/handlers/parseHandler.ts` - Query parsing endpoint
+
+3. Service layer (business logic):
+   - `src/services/usdaBatch.ts` - Batch API requests (up to 20 foods)
+   - `src/services/hotCache.ts` - Top 100 foods from D1 hot_foods_cache table
+   - `src/services/kvCacheService.ts` - KV L1 cache (30d TTL)
+   - `src/services/cache.ts` - D1 L0 cache (1h TTL, stale-while-revalidate)
+   - `src/services/apiKeyService.ts` - API key CRUD, credit/quota validation
+   - `src/services/requestDeduplicator.ts` - Prevents duplicate in-flight requests
+
+**Background Jobs (Cron):**
+- `0 3 * * *` - Daily cache warming (popular foods) via `cacheWarmer.ts`
+- `0 * * * *` - Hourly keep-alive + L1 cache warming
+- `*/2 * * * *` - Branded food fetcher (queue processor)
+
+**Queue Consumer:**
+- Processes `TASK_QUEUE` messages for branded food fetching
+- Updates KV L1 and hot cache after successful USDA fetch
+
+## Critical Patterns & Conventions
+
+**1. Background Task Safety:**
+Always use `safeBackgroundTask()` instead of raw `ctx.waitUntil()`:
+```typescript
+safeBackgroundTask(ctx, async () => {
+  await kvCacheService.put(key, data, env, requestId, 3600);
+}, 'cache-write', { key });
+```
+
+**2. Cache Key Normalization:**
+- POST requests: Hash normalized query (see `createRequestCacheKey` in `src/utils/cacheKey.ts`)
+- For `/v1/calculate/natural`: Use `normalizeQuery()` before hashing to avoid brittle keys
+- GET requests: Include URL + auth headers in cache key
+
+**3. Credit System:**
+Each endpoint has a cost:
+- Regex parse: 1 credit
+- Natural language search: 1 credit per item
+- Calculate: 1 credit
+- Search: 1 credit
+Deduct via `createCreditCheck(COST)` middleware, enforce via `apiKeyService.useCredits()`
+
+**4. Error Handling:**
+All errors extend `APIError` (see `src/types.ts`):
+- `InvalidInputError` (400)
+- `NotFoundError` (404)
+- `USDAFoodNotFound` (404)
+- `UpstreamServiceError` (503)
+Use `handleAPIError()` in `src/errorHandler.ts` to format responses
+
+**5. Request Body Caching:**
+`src/index.ts` pre-reads request bodies and attaches to `(routerRequest as any).__cachedBodyText` to prevent stream exhaustion across middleware
+
+**6. Structured Logging:**
+All logs use `logger` from `src/logger.ts` with request IDs:
+```typescript
+logger.info('Message', { key: 'value', requestId });
+```
+
+**7. Zod Validation:**
+Define schemas in `src/schemas/requestSchemas.ts`, apply via `validateRequest(Schema, 'body'|'query'|'params')` middleware
+
+## Database Schema (`schema.sql`)
+- `api_keys` - API key metadata, credits, tier
+- `cache` - Generic KV cache (NLP/USDA responses, 1h TTL)
+- `hot_foods_cache` - Top 100 foods (query_count tracking)
+- `rate_limit_logs` - Rate limit history per API key
+- `dead_letter_queue` - Failed requests for replay
+- `unmatched_logs` - Search terms without matches (self-improving)
+
+**Key SQL Pattern:**
+Use `ON CONFLICT` for upsert operations (see `hotCache.ts` set method)
+
+## Developer Workflows
+
+**Setup:**
+```bash
+npm install
+wrangler d1 create usda-cache-dev
+wrangler kv:namespace create API_KEY_CACHE_KV --env development
+wrangler d1 execute usda-cache-dev --file=schema.sql --env development
+wrangler secret put USDA_API_KEY --env development
+```
+
+**Local Development:**
+```bash
+npm run dev  # Starts local worker on http://localhost:8787
+```
+
+**Testing:**
+```bash
+npm test                  # All tests (vitest)
+npm run test:unit         # Unit tests only
+npm run test:integration  # Integration tests only
+```
+Tests use `@cloudflare/vitest-pool-workers` with isolated D1/KV storage per test
+
+**Deployment:**
+```bash
+wrangler deploy --env development  # Dev environment
+wrangler deploy --env production   # Prod environment
+```
+DO NOT deploy as Cloudflare Pages - this is a Worker project!
+
+**Generate Test API Key:**
+```
+GET /_admin/generate-key?tier=free|pro
+```
+
+**Database Migrations:**
+```bash
+wrangler d1 migrations apply usda-cache-prod --env production
+```
+
+## Common Pitfalls & Solutions
+
+❌ **DON'T:** Use `ctx.waitUntil()` directly - errors will terminate worker
+✅ **DO:** Use `safeBackgroundTask(ctx, fn, 'task-name', { context })`
+
+❌ **DON'T:** Read request body multiple times (stream exhaustion)
+✅ **DO:** Use `(request as any).__cachedBodyText` from `src/index.ts` pre-read
+
+❌ **DON'T:** Forget to add new endpoints to rate limit config in `src/config.ts`
+✅ **DO:** Define endpoint-specific limits in `rateLimits.free.endpoints`
+
+❌ **DON'T:** Log sensitive data (API keys, tokens)
+✅ **DO:** Use `sanitizeHeaders()` middleware (auto-redacts Auth headers)
+
+❌ **DON'T:** Make duplicate USDA API calls for same food
+✅ **DO:** Use `requestDeduplicator.deduplicate()` or `usdaBatch.queueFoodRequest()`
+
+## Key Files Reference
+- `src/index.ts` - Worker entry point, middleware chain, cron handlers
+- `src/config.ts` - Environment variable config, rate limit definitions
+- `src/types.ts` - All TypeScript types, error classes
+- `src/schemas/requestSchemas.ts` - Zod validation schemas
+- `src/services/usdaBatch.ts` - Batch API (20 foods per call)
+- `src/services/hotCache.ts` - Top 100 foods cache
+- `src/utils/backgroundTasks.ts` - Safe wrappers for ctx.waitUntil
+- `src/utils/cacheKey.ts` - Cache key normalization
+- `wrangler.toml` - Cloudflare bindings, cron triggers, environments
+- `schema.sql` - D1 database schema
+
+## When Adding New Endpoints
+1. Define Zod schema in `src/schemas/requestSchemas.ts`
+2. Add handler in `src/handlers/`
+3. Register route in `src/index.ts` with middleware:
+   - `edgeCache(TTL)` for GET requests
+   - `withAuth` for API key check
+   - `withRateLimiting` for rate limits
+   - `validateRequest(Schema, 'body')` for input validation
+   - `createCreditCheck(COST)` for credit deduction
+4. Add endpoint-specific rate limits to `src/config.ts` rateLimits
+5. Update `openapi.json` if public-facing
+6. Add integration tests in `tests/`