Respect RFC 6901, use ESM output, add new mergeKeys dereference option. (#398)

This changes the URL parsing behavior to actually respect RFC 6901. Previously we were URL encoding too often. This will be considered a breaking change, as it will change how previously parsed documents are bundled and dereferenced.

This also adds a new mergeKey option.

Additionally, the new output will be ESM.

Should fix #383, #356, and makes progress on #370

BREAKING CHANGE: Change URL encoding to use strict RFC 6901, add new mergeKeys dereference option

Author: jonluca

.claude/settings.local.json

@@ -0,0 +1,6 @@
+{
+  "permissions": {
+    "allow": ["Bash(npx vitest:*)", "Bash(bash:*)", "Bash(node test-debug.js:*)"]
+  },
+  "enableAllProjectMcpServers": false
+}

.yarn/releases/yarn-4.9.4.cjs renamed to .yarn/releases/yarn-4.11.0.cjs

@@ -1,3 +1,3 @@
 nodeLinker: node-modules
 
-yarnPath: .yarn/releases/yarn-4.9.4.cjs
+yarnPath: .yarn/releases/yarn-4.11.0.cjs

.yarnrc.yml

@@ -0,0 +1,152 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Development Commands
+
+### Building
+```bash
+yarn build          # Compile TypeScript to dist/
+yarn typecheck      # Run TypeScript type checking without emitting files
+```
+
+### Testing
+```bash
+yarn test           # Run all tests with coverage (Vitest)
+yarn test:node      # Run tests in Node.js environment
+yarn test:browser   # Run tests in browser environment (jsdom)
+yarn test:watch     # Run tests in watch mode
+yarn test:update    # Update test snapshots
+```
+
+### Code Quality
+```bash
+yarn lint           # Run ESLint on lib/ directory
+yarn prettier       # Format all code files
+```
+
+### Running Individual Tests
+To run a single test file:
+```bash
+npx vitest test/specs/circular/circular.spec.ts
+```
+
+To run tests matching a pattern:
+```bash
+npx vitest --grep "circular"
+```
+
+## Architecture Overview
+
+### Core Purpose
+This library parses, resolves, and dereferences JSON Schema `$ref` pointers. It handles references to:
+- External files (local filesystem)
+- HTTP/HTTPS URLs
+- Internal JSON pointers within schemas
+- Mixed JSON and YAML formats
+- Circular/recursive references
+
+### Key Architecture Components
+
+#### 1. $RefParser (lib/index.ts)
+The main entry point and orchestrator class. Provides four primary operations:
+- **parse()**: Reads a single schema file (JSON/YAML) without resolving references
+- **resolve()**: Parses and resolves all `$ref` pointers, returns a `$Refs` object mapping references to values
+- **bundle()**: Converts external `$ref` pointers to internal ones (single file output)
+- **dereference()**: Replaces all `$ref` pointers with their actual values (fully expanded schema)
+
+All methods support both callback and Promise-based APIs, with multiple overload signatures.
+
+#### 2. $Refs (lib/refs.ts)
+A map/registry of all resolved JSON references and their values. Tracks:
+- All file paths/URLs encountered
+- Circular reference detection
+- Helper methods to query references by type (file, http, etc.)
+
+#### 3. Pointer (lib/pointer.ts)
+Represents a single JSON pointer (`#/definitions/person`) and implements JSON Pointer RFC 6901 spec:
+- Parses JSON pointer syntax (`/`, `~0`, `~1` escaping)
+- Resolves pointers to actual values within objects
+- Handles edge cases (null values, missing properties)
+
+#### 4. $Ref (lib/ref.ts)
+Wraps a single reference with metadata:
+- The reference path/URL
+- The resolved value
+- Path type (file, http, etc.)
+- Error information (when continueOnError is enabled)
+
+#### 5. Plugin System
+Two types of plugins, both configurable via options:
+
+**Parsers** (lib/parsers/):
+- JSON parser (json.ts)
+- YAML parser (yaml.ts) - uses js-yaml
+- Text parser (text.ts)
+- Binary parser (binary.ts)
+- Execute in order based on `order` property and `canParse()` matching
+
+**Resolvers** (lib/resolvers/):
+- File resolver (file.ts) - reads from filesystem (Node.js only)
+- HTTP resolver (http.ts) - fetches from URLs using native fetch
+- Custom resolvers can be added via options
+- Execute in order based on `order` property and `canRead()` matching
+
+#### 6. Core Operations
+
+**lib/parse.ts**: Entry point for parsing a single schema file
+**lib/resolve-external.ts**: Crawls schema to find and resolve external `$ref` pointers
+**lib/bundle.ts**: Replaces external refs with internal refs
+**lib/dereference.ts**: Replaces all `$ref` pointers with actual values, handles circular references
+
+#### 7. Options System (lib/options.ts)
+Hierarchical configuration with defaults for:
+- Which parsers/resolvers to enable
+- Circular reference handling (boolean or "ignore")
+- External reference resolution (relative vs root)
+- Continue on error mode (collect all errors vs fail fast)
+- Bundle/dereference callbacks and matchers
+- Input mutation control (mutateInputSchema)
+
+### Key Design Patterns
+
+1. **Flexible Arguments**: normalizeArgs() (lib/normalize-args.ts) unifies various call signatures into consistent internal format
+
+2. **Path Handling**: Automatic conversion between filesystem paths and file:// URLs. Cross-platform support via util/url.ts and util/convert-path-to-posix.ts
+
+3. **Error Handling**:
+   - Fail-fast by default
+   - Optional continueOnError mode collects errors in JSONParserErrorGroup
+   - Specific error types: JSONParserError, InvalidPointerError, MissingPointerError, ResolverError, ParserError
+
+4. **Circular Reference Management**:
+   - Detected during dereference
+   - Can throw error, ignore, or handle via dereference.circular option
+   - Reference equality maintained (same `$ref` → same object instance)
+
+5. **Browser/Node Compatibility**:
+   - Uses native fetch (requires Node 18+)
+   - File resolver disabled in browser builds (package.json browser field)
+   - Tests run in both environments
+
+## Testing Strategy
+
+Tests are organized in test/specs/ by scenario:
+- Each scenario has test files (*.spec.ts) and fixture data
+- Tests validate parse, resolve, bundle, and dereference operations
+- Extensive coverage of edge cases: circular refs, deep nesting, special characters in paths
+- Browser-specific tests use test/fixtures/server.ts for HTTP mocking
+
+Test utilities:
+- test/utils/helper.js: Common test patterns
+- test/utils/path.js: Path handling for cross-platform tests
+- test/utils/serializeJson.ts: Custom snapshot serializer
+
+## Important Constraints
+
+1. **TypeScript Strict Mode**: Project uses strict TypeScript including exactOptionalPropertyTypes
+2. **JSON Schema Support**: Compatible with JSON Schema v4, v6, and v7
+3. **Minimum Node Version**: Requires Node >= 20 (for native fetch support)
+4. **Circular JSON**: Dereferenced schemas may contain circular references (not JSON.stringify safe)
+5. **Path Normalization**: Always converts filesystem paths to POSIX format internally
+6. **URL Safety**: HTTP resolver has safeUrlResolver option to block internal URLs (default: unsafe allowed)

CLAUDE.md

@@ -29,19 +29,15 @@ format. Maybe some of the files contain cross-references to each other.
 {
   "definitions": {
     "person": {
-      // references an external file
       "$ref": "schemas/people/Bruce-Wayne.json"
     },
     "place": {
-      // references a sub-schema in an external file
       "$ref": "schemas/places.yaml#/definitions/Gotham-City"
     },
     "thing": {
-      // references a URL
       "$ref": "http://wayne-enterprises.com/things/batmobile"
     },
     "color": {
-      // references a value in an external file via an internal reference
       "$ref": "#/definitions/thing/properties/colors/black-as-the-night"
     }
   }

README.md

@@ -2,10 +2,10 @@ import $Ref from "./ref.js";
 import Pointer from "./pointer.js";
 import * as url from "./util/url.js";
 import type $Refs from "./refs.js";
-import type $RefParser from "./index";
-import type { ParserOptions } from "./index";
-import type { JSONSchema } from "./index";
-import type { BundleOptions } from "./options";
+import type $RefParser from "./index.js";
+import type { ParserOptions } from "./index.js";
+import type { JSONSchema } from "./index.js";
+import type { BundleOptions } from "./options.js";
 
 export interface InventoryEntry {
   $ref: any;
@@ -40,7 +40,7 @@ function bundle<S extends object = JSONSchema, O extends ParserOptions<S> = Pars
   crawl<S, O>(parser, "schema", parser.$refs._root$Ref.path + "#", "#", 0, inventory, parser.$refs, options);
 
   // Remap all $ref pointers
-  remap(inventory);
+  remap<S, O>(inventory, options);
 }
 
 /**
@@ -203,7 +203,10 @@ function inventory$Ref<S extends object = JSONSchema, O extends ParserOptions<S>
  *
  * @param inventory
  */
-function remap(inventory: InventoryEntry[]) {
+function remap<S extends object = JSONSchema, O extends ParserOptions<S> = ParserOptions<S>>(
+  inventory: InventoryEntry[],
+  options: O,
+) {
   // Group & sort all the $ref pointers, so they're in the order that we need to dereference/remap them
   inventory.sort((a: InventoryEntry, b: InventoryEntry) => {
     if (a.file !== b.file) {
@@ -254,10 +257,10 @@ function remap(inventory: InventoryEntry[]) {
       // This $ref already resolves to the main JSON Schema file
       entry.$ref.$ref = entry.hash;
     } else if (entry.file === file && entry.hash === hash) {
-      // This $ref points to the same value as the prevous $ref, so remap it to the same path
+      // This $ref points to the same value as the previous $ref, so remap it to the same path
       entry.$ref.$ref = pathFromRoot;
     } else if (entry.file === file && entry.hash.indexOf(hash + "/") === 0) {
-      // This $ref points to a sub-value of the prevous $ref, so remap it beneath that path
+      // This $ref points to a sub-value of the previous $ref, so remap it beneath that path
       entry.$ref.$ref = Pointer.join(pathFromRoot, Pointer.parse(entry.hash.replace(hash, "#")));
     } else {
       // We've moved to a new file or new hash
@@ -267,7 +270,7 @@ function remap(inventory: InventoryEntry[]) {
 
       // This is the first $ref to point to this value, so dereference the value.
       // Any other $refs that point to the same value will point to this $ref instead
-      entry.$ref = entry.parent[entry.key] = $Ref.dereference(entry.$ref, entry.value);
+      entry.$ref = entry.parent[entry.key] = $Ref.dereference(entry.$ref, entry.value, options);
 
       if (entry.circular) {
         // This $ref points to itself

lib/bundle.ts

@@ -3,9 +3,8 @@ import Pointer from "./pointer.js";
 import * as url from "./util/url.js";
 import type $Refs from "./refs.js";
 import type { DereferenceOptions, ParserOptions } from "./options.js";
-import type { JSONSchema } from "./types";
-import type $RefParser from "./index";
-import { TimeoutError } from "./util/errors";
+import { type $RefParser, type JSONSchema } from "./index.js";
+import { TimeoutError } from "./util/errors.js";
 
 export default dereference;
 
@@ -278,7 +277,7 @@ function dereference$Ref<S extends object = JSONSchema, O extends ParserOptions<
   }
 
   // Dereference the JSON reference
-  let dereferencedValue = $Ref.dereference($ref, pointer.value);
+  let dereferencedValue = $Ref.dereference($ref, pointer.value, options);
 
   // Crawl the dereferenced value (unless it's circular)
   if (!circular) {

lib/dereference.ts

@@ -1,6 +1,6 @@
 import type { Options, ParserOptions } from "./options.js";
 import { getNewOptions } from "./options.js";
-import type { JSONSchema, SchemaCallback } from "./types";
+import type { JSONSchema, SchemaCallback } from "./index.js";
 
 // I really dislike this function and the way it's written. It's not clear what it's doing, and it's way too flexible
 // In the future, I'd like to deprecate the api and accept only named parameters in index.ts

lib/normalize-args.ts

@@ -82,6 +82,13 @@ export interface DereferenceOptions {
    * Default: `relative`
    */
   externalReferenceResolution?: "relative" | "root";
+
+  /**
+   * Whether duplicate keys should be merged when dereferencing objects.
+   *
+   * Default: `true`
+   */
+  mergeKeys?: boolean;
 }
 
 /**
@@ -229,6 +236,7 @@ export const getJsonSchemaRefParserDefaultOptions = () => {
        */
       excludedPathMatcher: () => false,
       referenceResolution: "relative",
+      mergeKeys: true,
     },
 
     mutateInputSchema: true,