openapi3: return a typed error for a disallowed external $ref

.github/docs/openapi3.txt

@@ -739,6 +739,19 @@ func (e *DependentSchemasFieldFor31Plus) As(target any) bool
 
 func (e *DependentSchemasFieldFor31Plus) Code() string
 
+type DisallowedExternalRefError struct {
+	// Ref is the $ref value as it appears in the document, e.g.
+	// "other.yaml#/components/schemas/Thing".
+	Ref string
+}
+    DisallowedExternalRefError is returned for a $ref to another document
+    encountered while Loader.IsExternalRefsAllowed is false. A caller that tells
+    a refused $ref apart from a document that failed to load for another reason,
+    to report the setting as the remedy or to map it to its own error, matches
+    this type with errors.As rather than the message text.
+
+func (e *DisallowedExternalRefError) Error() string
+
 type Discriminator struct {
 	Extensions map[string]any `json:"-" yaml:"-"`
 	Origin     *Origin        `json:"-" yaml:"-"`
@@ -2282,6 +2295,8 @@ type PathItem struct {
 	// Keys are case-sensitive HTTP method tokens and must not duplicate
 	// the methods covered by the fixed fields above.
 	AdditionalOperations map[string]*Operation `json:"additionalOperations,omitempty,case:strict" yaml:"additionalOperations,omitempty"` // OpenAPI >=3.2
+
+	// Has unexported fields.
 }
     PathItem is specified by OpenAPI/Swagger standard version 3. See
     https://github.com/OAI/OpenAPI-Specification/blob/main/versions/3.0.3.md#path-item-object
@@ -4207,7 +4222,10 @@ func AllowExtensionsWithRef() ValidationOption
 
 func AllowExtraSiblingFields(fields ...string) ValidationOption
     AllowExtraSiblingFields called as AllowExtraSiblingFields("description")
-    makes Validate not return an error when said field appears next to a $ref.
+    makes Validate not return an error when said field appears where it is
+    not expected: as an unknown field of an object, as an OpenAPI 3.1 schema
+    keyword in an OpenAPI 3.0 document, or as an extension next to a $ref under
+    ProhibitExtensionsWithRef. Other fields next to a $ref are never an error.
 
 func DisableExamplesValidation() ValidationOption
     DisableExamplesValidation disables all example schema validation.
@@ -4269,8 +4287,8 @@ func IsOpenAPI32OrLater() ValidationOption
 func ProhibitExtensionsWithRef() ValidationOption
     ProhibitExtensionsWithRef causes the validation to return an error if
     extensions (fields starting with 'x-') are found as siblings for $ref
-    fields, even in OpenAPI 3.0. Use AllowExtraSiblingFields to allow specific
-    extensions.
+    fields, in every OpenAPI version. Use AllowExtraSiblingFields to allow
+    specific extensions.
 
 func SetRegexCompiler(c RegexCompilerFunc) ValidationOption
     SetRegexCompiler allows to override the regex implementation used to

.github/workflows/fuzz.yml

@@ -0,0 +1,119 @@
+name: fuzz
+on:
+  pull_request:
+  push:
+    branches:
+    - master
+  schedule:
+  # Nightly, so a long run has a chance to reach code a 90s PR run never does.
+  - cron: '17 3 * * *'
+  workflow_dispatch:
+    inputs:
+      fuzztime:
+        description: 'How long to fuzz each target (go test -fuzztime syntax)'
+        required: false
+        default: 30m
+
+permissions:
+  contents: read
+
+jobs:
+  fuzz:
+    name: ${{ matrix.target }}
+    runs-on: ubuntu-latest
+    # A budget of 15m plus corpus minimization has to fit, and a hung target
+    # must not hold a runner forever.
+    timeout-minutes: 45
+    env:
+      # Keep in sync with the go workflow, whose cache this restores.
+      APISGURU_COMMIT: f7207cf0a5c56081d275ebae4cf615249323385d
+      GO111MODULE: 'on'
+      CGO_ENABLED: '0'
+    strategy:
+      # Every target reports independently: one crasher must not hide the rest.
+      fail-fast: false
+      matrix:
+        include:
+        - {package: ./openapi3, target: FuzzLoadFromData}
+        - {package: ./openapi3, target: FuzzLoadWithOrigins}
+        - {package: ./openapi3, target: FuzzYAMLMatchesJSON}
+        - {package: ./openapi3, target: FuzzSchemaVisitJSON}
+        - {package: ./openapi2, target: FuzzUnmarshalFromData}
+        - {package: ./openapi2conv, target: FuzzToV3}
+        - {package: ./openapi3filter, target: FuzzValidateRequest}
+    steps:
+
+    - uses: actions/setup-go@v6
+      with:
+        go-version: stable
+
+    - uses: actions/checkout@v6
+
+    # The APIs-guru corpus seeds the fuzzers alongside testdata. The go
+    # workflow caches it, so fetch it only when that cache is missing.
+    - name: APIs-guru corpus cache
+      id: apisguru
+      uses: actions/cache/restore@v4
+      with:
+        path: |
+          openapi2/testdata/APIs-guru-openapi-directory-*
+          openapi3/testdata/APIs-guru-openapi-directory-*
+        key: suite-${{ env.APISGURU_COMMIT }}
+
+    - name: Fetch APIs-guru corpus
+      if: steps.apisguru.outputs.cache-hit != 'true'
+      run: |
+        dir=openapi3/testdata/APIs-guru-openapi-directory-${APISGURU_COMMIT:0:7}
+        mkdir -p "$dir"
+        curl -fsSL "https://github.com/APIs-guru/openapi-directory/tarball/$APISGURU_COMMIT" | tar -xz -C "$dir"
+
+    # The generated corpus lives under $GOCACHE/fuzz and is what makes fuzzing
+    # cumulative: without it every run restarts from the seed corpus. The key
+    # carries the run id so each run writes a fresh entry, and restore-keys
+    # falls back to the most recent one for this target.
+    - name: Fuzzing corpus cache
+      uses: actions/cache@v3
+      with:
+        path: ~/.cache/go-build/fuzz
+        key: fuzz-${{ matrix.target }}-${{ github.run_id }}
+        restore-keys: fuzz-${{ matrix.target }}-
+
+    - name: Pick a budget
+      id: budget
+      run: |
+        if [[ -n '${{ inputs.fuzztime }}' ]]; then
+          fuzztime='${{ inputs.fuzztime }}'
+        elif [[ '${{ github.event_name }}' = schedule ]]; then
+          fuzztime=15m
+        else
+          fuzztime=90s
+        fi
+        echo "fuzztime=$fuzztime" >>"$GITHUB_OUTPUT"
+
+    # -run='^$' skips the ordinary tests: the seed corpus is already replayed by
+    # `go test ./...` in the go workflow, so this job only spends time fuzzing.
+    #
+    # The minimize budget is small on purpose. A minimize that cannot finish
+    # inside it writes the candidate it had and the job fails with "fuzzing
+    # process hung or terminated unexpectedly while minimizing", whose
+    # candidate then passes on replay: a red build and nothing to act on. The
+    # budget only decides how small the saved input gets, so spending less of
+    # it costs a larger seed and buys a signal that means what it says.
+    - name: Fuzz ${{ matrix.target }}
+      run: |
+        go test \
+          -run='^$' \
+          -fuzz='^${{ matrix.target }}$' \
+          -fuzztime='${{ steps.budget.outputs.fuzztime }}' \
+          -fuzzminimizetime=10s \
+          ${{ matrix.package }}
+
+    # A failing input is written next to the package it broke. Upload it so the
+    # crasher can be committed to the seed corpus, where it becomes a permanent
+    # regression test replayed by `go test ./...`.
+    - if: failure()
+      uses: actions/upload-artifact@v4
+      with:
+        name: crashers-${{ matrix.target }}
+        path: '**/testdata/fuzz/**'
+        if-no-files-found: ignore

Makefile

@@ -1,28 +1,40 @@
-.DEFAULT_GOAL := prepare
+.DEFAULT_GOAL := all
+.PHONY: help generate format test test-fast fuzz all
 
-.PHONY: help generate format test prepare
+# How long to spend on each fuzz target. Override: make fuzz FUZZTIME=10m
+FUZZTIME ?= 60s
 
-help:
-	@printf '%s\n' \
-		'Usage: make <target>' \
-		'' \
-		'  generate  Regenerate Go code, map helpers, and API documentation.' \
-		'  format    Apply the repository formatters and import cleanup.' \
-		'  test      Run the full Go package test suite.' \
-		'  prepare   Run generation, formatting, and tests in sequence.'
+## all: Run generation, formatting, and tests in sequence
+all: generate format test
 
+## generate: Regenerate Go code, map helpers, and API documentation
 generate:
 	go generate ./...
 	./maps.sh
 	./docs.sh
 
+## format: Apply the repository formatters and import cleanup
 format:
 	go fmt ./...
 
+## test: Run the full Go package test suite
 test:
 	go test ./...
 
-prepare:
-	$(MAKE) generate
-	$(MAKE) format
-	$(MAKE) test
+## test-fast: Run tests skipping the slow ones
+test-fast:
+	go test -short ./...
+
+## fuzz: Fuzz each target for FUZZTIME (default 60s)
+fuzz:
+	go test -run='^$$' -fuzz='^FuzzLoadFromData$$' -fuzztime=$(FUZZTIME) ./openapi3
+	go test -run='^$$' -fuzz='^FuzzLoadWithOrigins$$' -fuzztime=$(FUZZTIME) ./openapi3
+	go test -run='^$$' -fuzz='^FuzzYAMLMatchesJSON$$' -fuzztime=$(FUZZTIME) ./openapi3
+	go test -run='^$$' -fuzz='^FuzzSchemaVisitJSON$$' -fuzztime=$(FUZZTIME) ./openapi3
+	go test -run='^$$' -fuzz='^FuzzUnmarshalFromData$$' -fuzztime=$(FUZZTIME) ./openapi2
+	go test -run='^$$' -fuzz='^FuzzToV3$$' -fuzztime=$(FUZZTIME) ./openapi2conv
+	go test -run='^$$' -fuzz='^FuzzValidateRequest$$' -fuzztime=$(FUZZTIME) ./openapi3filter
+
+## help: list targets
+help:
+	@grep -E '^## ' $(MAKEFILE_LIST) | sed 's/## /  make /'

README.md

@@ -329,14 +329,30 @@ for _, path := range doc.Paths.InMatchingOrder() {
 
 ## CHANGELOG: Sub-v1 breaking API changes
 
-### v0.151.0
-* Removed the deprecated package-global `openapi3.IncludeOrigin` variable. Set the `IncludeOrigin` field of `openapi3.Loader` instead.
-* `openapi3filter.UrlencodedBodyDecoder` (hence `application/x-www-form-urlencoded` request body validation) now returns a `*openapi3filter.ParseError` for a value that cannot be parsed into its property's schema type, such as `n=abc` for an `integer` property, instead of silently dropping it from the decoded body.
-* `openapi3filter` now writes a query parameter's array `default` into the request URL following the parameter's `style` and `explode` (including their defaults), so an array without `explode` set is written as `?ids=1&ids=2` rather than `?ids=1,2`, and `pipeDelimited`/`spaceDelimited` arrays use their own delimiter.
+### v0.153.0
+* `openapi3filter` now parses `number` values of parameters and `application/x-www-form-urlencoded`/`multipart/form-data` bodies as decimal numbers only, so Go float literals such as `0x1p4`, `1_000`, `Inf`, `-infinity` or `NaN` are now rejected as invalid numbers.
+
+### v0.152.0
+* JSON keys of OpenAPI documents only fill the field of the same case (`json:",case:strict"`) when `encoding/json` is implemented with `encoding/json/v2` (by default from Go 1.27, and on Go 1.25 and 1.26 with `GOEXPERIMENT=jsonv2`): a key such as `"Components"` (not `"components"`) is kept as an extension instead of filling `openapi3.T.Components`. Other builds still match keys case-insensitively, as `encoding/json` v1 does.
+* On Go 1.27+, unless built with `GOEXPERIMENT=nojsonv2`, `openapi2` and `openapi3` decode documents with `encoding/json/v2`: objects with duplicate member names and strings with invalid UTF-8 are rejected, and decoding errors are worded differently and locate the faulty value from the root of the document, keys of paths, responses and callbacks included (e.g. `.paths.~1pets.get.responses.200.description`). On these builds `openapi2.Responses`, `openapi3.Callback`, `openapi3.Paths` and `openapi3.Responses` also implement `encoding/json/v2.UnmarshalerFrom`.
 * A schema with an empty `type: []` no longer marshals as `"type": null` which did not survive being read back.
 * The error for a header with neither or both of `content` and `schema` reads `header must contain exactly one of content and schema`. It used to say `parameter ...` followed by a `%v` dump of the `Header` struct, which printed heap addresses: the message differed between two runs over the same document, and leaked addresses into anything that logged it.
 * `openapi3filter` now parses `integer` values of parameters and `application/x-www-form-urlencoded`/`multipart/form-data` bodies as base 10 only, so `010` decodes to `10` rather than `8`, and Go integer literals such as `0x10`, `0b10`, `0o10` or `1_000` are now rejected as invalid integers.
 * `openapi3filter.ValidateResponse` now decodes and validates response headers described with `content` rather than `schema`, with the body decoder registered for the header's media type (see `RegisterBodyDecoder`). Such a header is no longer reported missing when it is required and present, and a value that does not match its schema is now rejected. As with a response body, a media type with no registered decoder fails with `unsupported content type`.
+* `Validate` no longer reports `extra sibling fields` for fields written next to a `$ref`: every version's Reference Object says they "SHALL be ignored". From OpenAPI 3.1 on, `summary` and `description` still override the referenced component's. `ExtraSiblingFieldsError` is still returned for unknown fields of inline objects, and for `x-` extensions next to a `$ref` under `ProhibitExtensionsWithRef`, which now applies to OpenAPI 3.1+ schemas too. It is no longer returned after `InternalizeRefs` for the fields that were next to a component's `$ref` it replaced, and which it leaves out.
+* In OpenAPI 3.1+ documents, the keywords next to a schema's `$ref` now apply alongside the referenced schema, as JSON Schema 2020-12 has it, instead of replacing its keywords of the same name: `{"$ref": ..., "maxLength": 10}` next to a referenced `maxLength: 3` still rejects `"abcd"`, and sibling `required` or `properties` add to the referenced ones. The resolved `SchemaRef.Value` keeps the referenced schema's shape: annotations such as `description`, `readOnly` or `default` replace the referenced ones there, and the other sibling keywords are appended to its `AllOf`.
+* Marshaling an `openapi3.*Ref` with a `$ref` now writes back the fields that were written next to it, instead of emitting a lone `{"$ref": ...}`. An OpenAPI 3.1 schema's sibling keywords, which constrain the referenced schema, were lost that way, and so was every other field next to a `$ref`.
+* The loader now resolves a Path Item Object's `$ref` even when fields are written next to it, and merges the two: it used to skip the `$ref` then, silently dropping every operation of the referenced path item and never noticing a `$ref` to nothing. The specification leaves the result undefined only for a field set on both sides, which is then taken from next to the `$ref`. Marshaling writes back the `$ref` with the fields that were next to it, which it used to drop.
+* The loader no longer panics with `interface conversion` when the same `$ref` is written where two kinds of objects are expected, one nested in the other, such as a path item and one of its responses.
+* `InternalizeRefs` now makes a component of the schema an external `$ref` points to, rather than of that schema with the OpenAPI 3.1 keywords next to the `$ref` applied to it, which leaked them into every other use of the component. The external `$ref`s within those keywords are now rewritten in the marshaled document too.
+* The keywords next to a schema's `$ref` in a file of bare schemas, which has no `openapi` field, now follow the version of the document that refers to the file: they were ignored, as in OpenAPI 3.0, even from an OpenAPI 3.1 document.
+* In OpenAPI 3.1+ documents, loading now fails when the keywords next to a schema's `$ref` do not form a schema, such as `allOf: {type: integer}`, instead of silently ignoring them all.
+* The errors raised by the keywords next to an OpenAPI 3.1 schema's `$ref` now carry their `Origin`.
+
+### v0.151.0
+* Removed the deprecated package-global `openapi3.IncludeOrigin` variable. Set the `IncludeOrigin` field of `openapi3.Loader` instead.
+* `openapi3filter.UrlencodedBodyDecoder` (hence `application/x-www-form-urlencoded` request body validation) now returns a `*openapi3filter.ParseError` for a value that cannot be parsed into its property's schema type, such as `n=abc` for an `integer` property, instead of silently dropping it from the decoded body.
+* `openapi3filter` now writes a query parameter's array `default` into the request URL following the parameter's `style` and `explode` (including their defaults), so an array without `explode` set is written as `?ids=1&ids=2` rather than `?ids=1,2`, and `pipeDelimited`/`spaceDelimited` arrays use their own delimiter.
 
 ### v0.150.0
 * `openapi2.Operation.Responses` field type changed from `map[string]*Response` to `*Responses`, which keeps the Responses Object's `x-` extensions in `Responses.Extensions` instead of failing to decode them as responses.
@@ -353,8 +369,6 @@ for _, path := range doc.Paths.InMatchingOrder() {
 * `(*openapi3.Parameter).SerializationMethod()` (for query parameters) and `(*openapi3.Encoding).SerializationMethod()` now default `explode` to `false` for `style: spaceDelimited` and `style: pipeDelimited`, as the spec says, so `openapi3filter` splits `?ids=1|2|3` and `application/x-www-form-urlencoded` body properties such as `ids=1|2|3` instead of expecting `ids=1&ids=2&ids=3`. Set `explode: true` to keep the previous behavior.
 * Removed the package-global `openapi3.SchemaErrorDetailsDisabled` variable. Pass the new `openapi3.DisableSchemaErrorDetails()` option to `(*openapi3.Schema).VisitJSON(..)` instead, or add it to `openapi3filter.Options.SchemaValidationOptions`, which `openapi3filter` now also applies when validating parameters. Combined with `openapi3.EnableJSONSchema2020()` (hence with `openapi3filter` on OpenAPI 3.1+ documents), validation fails with `openapi3.ErrSchemaErrorDetailsWithJSONSchema2020`, as that validator's error messages include values.
 * The message customizer set with `openapi3.SetSchemaErrorMessageCustomizer(..)` (hence `(*openapi3filter.Options).WithCustomSchemaErrorFunc(..)`) now also applies to discriminator errors, to errors of patterns that fail to compile and to errors of the JSON Schema 2020-12 validator (`openapi3.EnableJSONSchema2020()`).
-* JSON keys of OpenAPI documents only fill the field of the same case (`json:",case:strict"`) when `encoding/json` is implemented with `encoding/json/v2` (by default from Go 1.27, and on Go 1.25 and 1.26 with `GOEXPERIMENT=jsonv2`): a key such as `"Components"` (not `"components"`) is kept as an extension instead of filling `openapi3.T.Components`. Other builds still match keys case-insensitively, as `encoding/json` v1 does.
-* On Go 1.27+, unless built with `GOEXPERIMENT=nojsonv2`, `openapi2` and `openapi3` decode documents with `encoding/json/v2`: objects with duplicate member names and strings with invalid UTF-8 are rejected, and decoding errors are worded differently and locate the faulty value from the root of the document, keys of paths, responses and callbacks included (e.g. `.paths.~1pets.get.responses.200.description`). On these builds `openapi2.Responses`, `openapi3.Callback`, `openapi3.Paths` and `openapi3.Responses` also implement `encoding/json/v2.UnmarshalerFrom`.
 
 ### v0.147.0
 * `(*openapi3.PathItem).SetOperation(string, *Operation)` no longer panics on unhandled HTTP methods: these are now stored in the new `openapi3.PathItem.AdditionalOperations` field (passing a nil operation deletes the entry).