[REVIEW] fetchHeaders: HEAD request systematically miss-grades sites that emit security headers only on GET

[BodenMcHale]

Summary

fetchHeaders (src/fetch.ts:2) issues an HTTP HEAD request to retrieve response headers. In real-world deployments many sites do not return the same security headers on HEAD as they do on GET — most commonly because security headers (especially Content-Security-Policy) are set by middleware that only runs against HTML responses, by content-type-conditional logic, by CDN page rules, or by edge workers that short-circuit HEAD. The result is that this analyzer reports missing for CSP/HSTS/etc. on sites that actually have them, producing false-failure grades (F/D) and — because the CLI exits non-zero on D/F (src/cli.ts:49) — breaking CI gates against genuinely well-configured sites.

Every comparable analyzer in this category (securityheaders.com, Mozilla Observatory, Hardenize) issues GET, not HEAD, for exactly this reason.

Investigation Path

1. Read every source file: src/fetch.ts, src/analyzer.ts, src/rules.ts, src/cli.ts, src/index.ts, src/types.ts.
2. Confirmed HEAD is hard-coded with no fallback path: grep -rn "HEAD" src/ returns exactly one match (src/fetch.ts:2).
3. Reviewed open issues (agent-escalate: Permissions-Policy rule should require camera/microphone/geolocation to award full score #4, [REVIEW] CSP: checkCSP awards full score without verifying base-uri restriction, enabling silent <base> injection bypass #5) and open PRs (test: add comprehensive test coverage + fetch timeout + CLI help/version #2, chore: add dependabot.yml for automated dependency updates #3, Add triage state tracking for issue automation #6) — none of them change the HTTP method. PR test: add comprehensive test coverage + fetch timeout + CLI help/version #2 adds a timeoutMs option but keeps method: 'HEAD'.
4. Cross-referenced behavior against:
   • RFC 9110 §9.3.2 (HEAD is defined to return identical headers, but the spec also notes "the server SHOULD send the same header fields … as it would have sent if the request had been a GET" — i.e. SHOULD, not MUST, and well-known exceptions exist).
   • Cloudflare's documented behavior of stripping certain response headers on HEAD for cached static assets.
   • Express helmet middleware behavior in apps that set CSP only on text/html responses (a very common pattern, since CSP is meaningless on JSON/asset responses).
5. Traced the failure path: empty CSP/HSTS in the returned record → checkCSP/checkHSTS return status: 'missing' and score: 0 → cumulative score drops two grade tiers → CLI exits 1.

Evidence

• File: src/fetch.ts line 2, commit 8d29a8c

  const res = await fetch(url, { method: 'HEAD', redirect: 'follow' });

• Current behavior: A request is issued with method: 'HEAD'. Whatever headers the server returns on HEAD are the only headers the analyzer sees. There is no GET fallback when key security headers are absent.
• Expected behavior: Use GET (and discard the body, or stream-and-abort after headers). GET is the method every browser actually uses to navigate to the page, so it is the only method whose response headers reflect what real users receive.
• Real-world reproducer: A common Express pattern that demonstrates the failure mode:

  // app.js — emits CSP only on HTML responses
  app.use((req, res, next) => {
    res.on('header', () => {
      if (String(res.getHeader('content-type') || '').startsWith('text/html')) {
        res.setHeader('Content-Security-Policy', "default-src 'self'");
      }
    });
    next();
  });

  A HEAD request has no body and may never set Content-Type: text/html, so CSP is never attached. The analyzer reports CSP missing (-30 points) and very likely flips the grade from B → D or F.
• External references:
  • RFC 9110 §9.3.2 — HEAD: headers should match GET, but spec uses SHOULD; servers in the wild diverge.
  • Mozilla HTTP Observatory source — uses GET.
  • securityheaders.com FAQ — analyzes the response to a GET.
  • Cloudflare Workers docs — HEAD handling — Workers can short-circuit HEAD and return a different header set than GET.

Impact

Every developer who runs this tool against a real production site is at risk of receiving a wrong grade. Two failure modes follow:

1. False low grade → broken CI gate. A site that genuinely has CSP, HSTS, etc. but only attaches them to HTML responses (the common pattern) gets graded D or F. The CLI exits 1 (src/cli.ts:49). Deployment pipelines that use this as a gate (the README's chore: add dependabot.yml for automated dependency updates #3 recommended use case) reject good deployments.
2. Silent miscommunication. A developer scans their own site, sees Content-Security-Policy: missing, and either (a) adds a second CSP via meta tag or duplicate middleware — creating policy-merge surprises — or (b) loses trust in the tool and stops using it.

Concretely, the README's quick-start example — npx @hailbytes/security-headers https://example.com — produces materially different output against the same site depending on whether that site's stack happens to set headers uniformly on HEAD and GET. The user has no way to know this without reading fetch.ts.

This is the primary correctness contract of an HTTP-header analyzer: "the headers we report are the headers a browser sees." HEAD breaks that contract.

Suggested Remediation

Switch the default request to GET and discard the body. Two implementation sketches:

Option A — minimal change (preferred):

// src/fetch.ts
export async function fetchHeaders(url: string): Promise<Record<string, string>> {
  const res = await fetch(url, { method: 'GET', redirect: 'follow' });
  // We only need headers; release the body so the connection can be reused.
  // (Node's undici returns a streamed body; cancelling it is safe.)
  try { await res.body?.cancel(); } catch { /* already closed */ }
  const headers: Record<string, string> = {};
  res.headers.forEach((value, key) => { headers[key.toLowerCase()] = value; });
  return headers;
}

Option B — GET with HEAD opt-in for users who explicitly want it:

Combine with the in-flight FetchOptions work in PR #2:

export interface FetchOptions {
  timeoutMs?: number;
  method?: 'GET' | 'HEAD'; // default: 'GET'
}

Default to 'GET', document 'HEAD' as a performance opt-in with a warning about under-reporting.

Additional hardening to consider in the same PR (each one-liner):

• Set a non-default User-Agent (e.g. @hailbytes/security-headers/1.0.x) — many WAFs and CDNs respond differently (or block) requests with no UA or node-fetch's default UA, which can also skew which headers come back.
• Add an Accept: text/html request header so servers that vary their response middleware on Accept return their HTML-response headers.

Acceptance Criteria

• src/fetch.ts issues a GET request by default (or makes GET the documented default of a new options object)
• Response body is explicitly cancelled/discarded so memory usage does not scale with page size
• An integration-style test (vitest with a Hono/node:http server, or a vi.spyOn(globalThis, 'fetch') mock) asserts the request method is GET
• An integration-style test asserts that a server which returns CSP/HSTS only on GET (and an empty header set on HEAD) is now graded correctly
• README's "Library — analyze a URL" section mentions that the tool issues GET (matches what a browser does)
• If a method option is exposed, the README documents the trade-off

---

Claude Code review routine · commit 8d29a8c183b9d0065dddaf03fe18ea61351750dc · 2026-05-26T00:42:00Z

[BodenMcHale]

Triage Review

Reviewed at: 2026-05-26T05:10:00Z
Commit checked: 5083c52b64190381eadbb9f0c13b42e52c358a5a (current main HEAD)
Version: 1.0.1 (per package.json)

Verification

Confirmed against the current source tree:

• src/fetch.ts:2 — const res = await fetch(url, { method: 'HEAD', redirect: 'follow' });
• grep -rn 'HEAD' src/ test/ returns exactly one match (the line above). There is no GET fallback path, no method option, and no test that exercises fetchHeaders at all (test/analyzer.test.ts only calls analyzeHeaders against in-memory header maps).
• src/cli.ts:49 — if (report.grade === 'D' || report.grade === 'F') process.exit(1); confirms the gate-failure path the issue describes.
• git log --all --oneline -- src/fetch.ts shows the file has not been touched since 6d2180b (initial 1.0.1 release), so this behavior is shipped to every npm consumer today.
• PR test: add comprehensive test coverage + fetch timeout + CLI help/version #2 (a2d360d — adds FetchOptions / timeoutMs) and PR chore: add dependabot.yml for automated dependency updates #3 (dependabot config) do not change the method — PR test: add comprehensive test coverage + fetch timeout + CLI help/version #2 still calls fetch(url, { method: 'HEAD', signal, redirect: 'follow' }). The fix has not been silently absorbed elsewhere.

Root Cause

fetchHeaders chose HEAD for bandwidth efficiency, but the correctness contract of the tool is "report the headers a browser receives." Per RFC 9110 §9.3.2, servers SHOULD return the same header set for HEAD as for GET, but several extremely common deployment patterns violate this — notably middleware that attaches Content-Security-Policy only on text/html responses (a HEAD request never produces a body and may never trip the content-type branch). The result is checkCSP (src/rules.ts:42-47) returning status: 'missing', score: 0/30, which alone is enough to drop two grade tiers and trip the CLI gate.

Comparable tools (Mozilla Observatory, securityheaders.com, Hardenize) all use GET for this reason.

Suggested Fix

The minimal change in src/fetch.ts:

export async function fetchHeaders(url: string): Promise<Record<string, string>> {
  const res = await fetch(url, {
    method: 'GET',
    redirect: 'follow',
    headers: { 'Accept': 'text/html,*/*;q=0.8', 'User-Agent': '@hailbytes/security-headers/1.0.x' },
  });
  try { await res.body?.cancel(); } catch { /* already closed */ }
  const headers: Record<string, string> = {};
  res.headers.forEach((value, key) => { headers[key.toLowerCase()] = value; });
  return headers;
}

The body.cancel() matters because undici streams the response — without it, memory usage scales with page size against arbitrary URLs.

Because PR #2 is in flight and introduces FetchOptions, the cleanest path is probably to land this into that PR (or rebase it on top): add method?: 'GET' | 'HEAD' with a 'GET' default, document 'HEAD' as a performance opt-in. That avoids merge-conflict churn between two fetch-touching PRs.

Test gap to close in the same change

There is currently zero coverage of src/fetch.ts. A vi.spyOn(globalThis, 'fetch') test that asserts (a) the request method is GET and (b) the body is cancelled would be enough to lock in the contract.

Reference

• RFC 9110 §9.3.2 — HEAD
• Mozilla HTTP Observatory — uses GET
• OWASP Secure Headers Project — analyzers should reflect what browsers see

Recommendation

This is the highest-impact correctness bug currently open in the repo: it directly invalidates the CLI-as-CI-gate use case advertised in the README. I'd prioritize it ahead of #5 (which is also valid but affects scoring, not request semantics) and well ahead of #4 (already addressed in 8d29a8c).

---

Claude Code triage routine · auto-generated

---

Generated by Claude Code