Add schema definition extension plumbing by jwils · Pull Request #1200 · block/elasticgraph

jwils · 2026-05-20T18:19:06Z

Why

Prepare schema definition for ingestion serializer extensions before moving JSON Schema code into its own gem.

What

Add schema definition extension_modules plumbing for APIs, factories, and rake tasks
Apply duplicate extension modules only once
Replace the rake-task sidecar enforcement option with schema-level schema.enforce_json_schema_version false
Update project templates, tests, and docs to configure JSON schema enforcement in schema definitions

Risk Assessment

Medium — this touches schema definition setup, but JSON Schema behavior remains in elasticgraph-schema_definition in this PR.

References

Stacked on Add JSON ingestion gem shell #1199.
Local spellcheck, lint, type-check, targeted specs, and schema artifact checks passed.

Stack

Current PR is marked with ->.

## Why Split the JSON ingestion extraction into reviewable steps so the gem packaging can be checked independently before code moves. ## What - Add the `elasticgraph-json_ingestion` gem shell - Wire the new path gem into the bundle without changing runtime behavior - Update generated dependency diagrams and gem READMEs for the new empty gem ## Risk Assessment Low — this only adds an empty extension gem package, lockfile wiring, and generated dependency docs. ## References - First PR in the JSON ingestion extraction stack. - Local spellcheck, lint, type-check, schema artifact, and dependency diagram checks passed. ## Stack Current PR is marked with `->`. - -> [#1199 Add JSON ingestion gem shell](#1199) - [#1200 Add schema definition extension plumbing](#1200) - [#1201 Move JSON schema helpers to json ingestion gem](#1201) - [#1203 Namespace JSON schema helpers under JSON ingestion](#1203) - [#1204 Add JSON ingestion indexing extensions](#1204) - [#1215 Wire JSON ingestion schema extension modules](#1215) - [#1205 Use JSON ingestion as schema definition extension](#1205) - [#1202 Document JSON ingestion schema APIs](#1202) - [#1224 Migrate JSON ingestion specs into gem](#1224) - [#1220 Add configurable indexing event decoder](#1220)

## Why Keep the first extraction layer reviewable by moving the JSON Schema helper files without changing their contents, so GitHub can render them as renames instead of delete/add rewrites. ## What - Move JSON Schema helper Ruby and RBS files into `elasticgraph-json_ingestion` - Update core requires/type references to load the helpers from the new gem - Keep `elasticgraph-json_ingestion` optional at runtime by using a schema_definition development dependency instead of a runtime dependency - Add the Docker/Apollo build-context wiring needed for bundles that include the new path gem explicitly ## Risk Assessment Low to medium - mostly file moves plus require/dependency wiring. ## References - Stacked on #1200. - `git diff -M --summary` reports the moved helper files as 100% renames, including `json_schema_with_metadata.rb`. - Local bundle install, lint, type check, spellcheck, dependency diagram verification, schema artifact check, and targeted unit specs passed on the restacked branch. ## Stack Current PR is marked with `->`. - [#1199 Add JSON ingestion gem shell](#1199) - [#1200 Add schema definition extension plumbing](#1200) - -> [#1201 Move JSON schema helpers to json ingestion gem](#1201) - [#1203 Namespace JSON schema helpers under JSON ingestion](#1203) - [#1204 Add JSON ingestion indexing extensions](#1204) - [#1215 Wire JSON ingestion schema extension modules](#1215) - [#1205 Use JSON ingestion as schema definition extension](#1205) - [#1202 Document JSON ingestion schema APIs](#1202) - [#1224 Migrate JSON ingestion specs into gem](#1224) - [#1220 Add configurable indexing event decoder](#1220)

## Why After the move-only PR preserves rename detection, put the moved helper constants under the `ElasticGraph::JSONIngestion` namespace. ## What - Namespace the moved JSON Schema helper classes and RBS signatures - Update schema_definition references and tests to use the new qualified constants ## Risk Assessment Medium - namespace-only behavior should be unchanged, but it updates call sites across schema_definition. ## References - Stacked on #1201. - Local lint, type check, schema artifact checks, and documentation coverage passed while building this split. ## Stack Current PR is marked with `->`. - [#1199 Add JSON ingestion gem shell](#1199) - [#1200 Add schema definition extension plumbing](#1200) - [#1201 Move JSON schema helpers to json ingestion gem](#1201) - -> [#1203 Namespace JSON schema helpers under JSON ingestion](#1203) - [#1204 Add JSON ingestion indexing extensions](#1204) - [#1215 Wire JSON ingestion schema extension modules](#1215) - [#1205 Use JSON ingestion as schema definition extension](#1205) - [#1202 Document JSON ingestion schema APIs](#1202) - [#1224 Migrate JSON ingestion specs into gem](#1224) - [#1220 Add configurable indexing event decoder](#1220)

## Why Keep the JSON ingestion indexing extensions separate from the schema-extension wiring so the indexing behavior is easier to review. ## What - Add JSON ingestion extension modules for indexing field references and field types - Make core indexing field references and enum/scalar/union field types extendable via the existing MemoizableData helper, matching the object field type - Keep JSON ingestion behavior in extension modules; no wrapper classes ## Risk Assessment Low to medium - this changes internal indexing value objects to use the existing MemoizableData wrapper so JSON ingestion can attach extension modules, without changing core mapping behavior. ## References - Stacked on #1203. - `bundle exec rspec elasticgraph-schema_definition/spec/unit/elastic_graph/schema_definition/json_schema_spec.rb elasticgraph-schema_definition/spec/unit/elastic_graph/schema_definition/json_schema_field_metadata_spec.rb elasticgraph-schema_definition/spec/unit/elastic_graph/schema_definition/indexing/json_schema_with_metadata_spec.rb elasticgraph-schema_definition/spec/unit/elastic_graph/schema_definition/factory_spec.rb` passed. - `script/type_check` passed. - `script/lint` passed. ## Stack Current PR is marked with `->`. - [#1199 Add JSON ingestion gem shell](#1199) - [#1200 Add schema definition extension plumbing](#1200) - [#1201 Move JSON schema helpers to json ingestion gem](#1201) - [#1203 Namespace JSON schema helpers under JSON ingestion](#1203) - -> [#1204 Add JSON ingestion indexing extensions](#1204) - [#1215 Wire JSON ingestion schema extension modules](#1215) - [#1205 Use JSON ingestion as schema definition extension](#1205) - [#1202 Document JSON ingestion schema APIs](#1202) - [#1224 Migrate JSON ingestion specs into gem](#1224) - [#1220 Add configurable indexing event decoder](#1220)

## Why Introduce the JSON ingestion schema-definition extension modules after the indexing extension points exist, but before making JSON ingestion the default implementation. ## What - Add `ElasticGraph::JSONIngestion::SchemaDefinition::APIExtension` and supporting factory/results/artifact/state/schema-element extension modules - Wire JSON ingestion through the factory so core indexing field references and field types are extended in place - Keep the existing core JSON Schema behavior active in this intermediate layer - Add doctest support for JSON ingestion schema-definition examples without making the extension the default yet ## Risk Assessment Medium - this adds new extension code, but the default behavior remains the existing core JSON Schema implementation in this PR. ## References - Stacked on #1204. - `bundle exec rspec elasticgraph-schema_definition/spec/unit/elastic_graph/schema_definition/json_schema_spec.rb elasticgraph-schema_definition/spec/unit/elastic_graph/schema_definition/json_schema_field_metadata_spec.rb elasticgraph-schema_definition/spec/unit/elastic_graph/schema_definition/indexing/json_schema_with_metadata_spec.rb elasticgraph-schema_definition/spec/unit/elastic_graph/schema_definition/factory_spec.rb` passed. - `script/type_check` passed. - `script/lint` passed. ## Stack Current PR is marked with `->`. - [#1199 Add JSON ingestion gem shell](#1199) - [#1200 Add schema definition extension plumbing](#1200) - [#1201 Move JSON schema helpers to json ingestion gem](#1201) - [#1203 Namespace JSON schema helpers under JSON ingestion](#1203) - [#1204 Add JSON ingestion indexing extensions](#1204) - -> [#1215 Wire JSON ingestion schema extension modules](#1215) - [#1205 Use JSON ingestion as schema definition extension](#1205) - [#1202 Document JSON ingestion schema APIs](#1202) - [#1224 Migrate JSON ingestion specs into gem](#1224) - [#1220 Add configurable indexing event decoder](#1220)

## Why Complete the extraction by loading JSON ingestion through the schema-definition extension API and removing the duplicated core JSON Schema implementation. ## What - Wire `ElasticGraph::JSONIngestion::SchemaDefinition::APIExtension` into app, template, doctest, and spec-helper schema definition setup - Move JSON Schema artifact/results behavior out of core schema_definition - Remove JSON Schema field/type/index logic from core classes now provided by the extension - Keep JSON schema version enforcement configurable through the JSON ingestion extension - Update the generated JSON ingestion dependency diagram ## Risk Assessment Medium to high - this is the behavior-transfer layer, but it sits after the move/namespace/wrapper/wiring PRs so the remaining diff is mostly deletion and wiring. ## References - Stacked on #1215. - Local `script/quick_build` passed before the generated diagram update; `script/update_dependency_diagrams --verify` and README spellcheck passed after it. ## Stack Current PR is marked with `->`. - [#1199 Add JSON ingestion gem shell](#1199) - [#1200 Add schema definition extension plumbing](#1200) - [#1201 Move JSON schema helpers to json ingestion gem](#1201) - [#1203 Namespace JSON schema helpers under JSON ingestion](#1203) - [#1204 Add JSON ingestion indexing extensions](#1204) - [#1215 Wire JSON ingestion schema extension modules](#1215) - -> [#1205 Use JSON ingestion as schema definition extension](#1205) - [#1202 Document JSON ingestion schema APIs](#1202) - [#1224 Migrate JSON ingestion specs into gem](#1224) - [#1220 Add configurable indexing event decoder](#1220)

## Why Document the JSON ingestion schema-definition APIs after extracting JSON ingestion into its own extension and making it part of generated projects by default. ## What - Document the default JSON ingestion setup and explicit extension-module wiring - Document `json_schema_version`, `json_schema_strictness`, scalar JSON Schema metadata, field/type validations, and field-level `nullable: false` ## Risk Assessment Low - README-only documentation change. Snippets are validated. ## References - Stacked on #1205. - Local `script/quick_build` passed before the generated diagram restack; `script/update_dependency_diagrams --verify` and README spellcheck passed after it. ## Stack Current PR is marked with `->`. - [#1199 Add JSON ingestion gem shell](#1199) - [#1200 Add schema definition extension plumbing](#1200) - [#1201 Move JSON schema helpers to json ingestion gem](#1201) - [#1203 Namespace JSON schema helpers under JSON ingestion](#1203) - [#1204 Add JSON ingestion indexing extensions](#1204) - [#1215 Wire JSON ingestion schema extension modules](#1215) - [#1205 Use JSON ingestion as schema definition extension](#1205) - -> [#1202 Document JSON ingestion schema APIs](#1202) - [#1224 Migrate JSON ingestion specs into gem](#1224) - [#1220 Add configurable indexing event decoder](#1220)

## Why The JSON ingestion gem should own the JSON-schema tests that cover its extracted implementation so the gem can enforce full coverage without a SimpleCov skip. ## What - Move JSON schema generation, metadata merge, pruner, and matcher specs into `elasticgraph-json_ingestion`. - Add JSON-ingestion artifact-manager integration coverage for versioned schema artifacts, version-bump enforcement, metadata refresh, and schema evolution diagnostics. - Add focused wrapper/scalar specs for the remaining extracted JSON-ingestion paths. - Remove the temporary SimpleCov filter for `elasticgraph-json_ingestion`. ## Verification - `script/run_gem_specs elasticgraph-json_ingestion` - `script/type_check elasticgraph-json_ingestion` - `BUNDLE_GEMFILE=Gemfile bundle exec rspec --format progress spec/integration/elastic_graph/schema_definition/rake_tasks_spec.rb spec/unit/elastic_graph/schema_definition/runtime_metadata/scalar_types_by_name_spec.rb` from `elasticgraph-schema_definition/` - `bundle exec standardrb ...` on touched specs/support `script/run_gem_specs elasticgraph-schema_definition` and `script/run_gem_specs elasticgraph-json_ingestion` both pass with 100% coverage (with the test datastore booted). ## Follow-ups - The migrated spec files keep the compact `module JSONIngestion::SchemaDefinition` form so that git/GitHub detects them as file moves. A follow-up PR (after this stack merges) will convert them to the standard nested module style and re-indent. - The JSON-schema-versioning scenarios that the new `schema_artifact_manager_extension_spec.rb` covers have been removed from the core `rake_tasks_spec.rb`; the core lines they exercised are now covered by focused unit specs instead. ## Stack Current PR is marked with `->`. - [#1199 Add JSON ingestion gem shell](#1199) - [#1200 Add schema definition extension plumbing](#1200) - [#1201 Move JSON schema helpers to json ingestion gem](#1201) - [#1203 Namespace JSON schema helpers under JSON ingestion](#1203) - [#1204 Add JSON ingestion indexing extensions](#1204) - [#1215 Wire JSON ingestion schema extension modules](#1215) - [#1205 Use JSON ingestion as schema definition extension](#1205) - [#1202 Document JSON ingestion schema APIs](#1202) - -> [#1224 Migrate JSON ingestion specs into gem](#1224) - [#1220 Add configurable indexing event decoder](#1220)

jwils mentioned this pull request May 20, 2026

Move JSON schema helpers to json ingestion gem #1201

Merged

jwils force-pushed the joshuaw/schema-extension-plumbing branch from edb2d90 to ef7b7e5 Compare May 20, 2026 18:29

jwils force-pushed the joshuaw/json-ingestion-gem-shell branch from 3643e2f to 63b3c9f Compare May 20, 2026 18:29

jwils marked this pull request as ready for review May 22, 2026 18:22

jwils requested review from BrianSigafoos-SQ, ayousufi, bsorbo, jwondrusch and myronmarston as code owners May 22, 2026 18:22

myronmarston reviewed May 22, 2026

View reviewed changes

Comment thread elasticgraph-apollo/apollo_tests_implementation/Rakefile Outdated

Comment thread elasticgraph-apollo/apollo_tests_implementation/Rakefile Outdated

Comment thread elasticgraph-schema_definition/lib/elastic_graph/schema_definition/api.rb

jwils force-pushed the joshuaw/schema-extension-plumbing branch from ef7b7e5 to d841965 Compare May 22, 2026 22:41

myronmarston reviewed May 22, 2026

View reviewed changes

Comment thread config/site/Rakefile Outdated

Comment thread elasticgraph-local/lib/elastic_graph/local/rake_tasks.rb

Comment thread elasticgraph-schema_definition/sig/elastic_graph/schema_definition/test_support.rbs

jwils force-pushed the joshuaw/schema-extension-plumbing branch from d841965 to ccd29a4 Compare May 24, 2026 13:33

jwils force-pushed the joshuaw/json-ingestion-gem-shell branch from 63b3c9f to a099965 Compare May 24, 2026 13:33

jwils force-pushed the joshuaw/schema-extension-plumbing branch 2 times, most recently from 6a5b3a8 to 052d30a Compare May 24, 2026 14:24

myronmarston reviewed May 24, 2026

View reviewed changes

Comment thread elasticgraph-schema_definition/lib/elastic_graph/schema_definition/state.rb

Comment thread elasticgraph-schema_definition/lib/elastic_graph/schema_definition/state.rb Outdated

jwils force-pushed the joshuaw/schema-extension-plumbing branch 2 times, most recently from 0431d7d to 1a605d6 Compare May 24, 2026 23:50

jwils force-pushed the joshuaw/json-ingestion-gem-shell branch from a099965 to b8671d4 Compare May 25, 2026 01:10

jwils force-pushed the joshuaw/schema-extension-plumbing branch 2 times, most recently from c06e080 to e69ae75 Compare May 25, 2026 01:17

myronmarston approved these changes May 26, 2026

View reviewed changes

jwils force-pushed the joshuaw/schema-extension-plumbing branch from e69ae75 to ac34ddb Compare May 27, 2026 16:22

jwils force-pushed the joshuaw/json-ingestion-gem-shell branch from b8671d4 to 3daa25c Compare May 27, 2026 16:22

myronmarston approved these changes May 27, 2026

View reviewed changes

jwils force-pushed the joshuaw/json-ingestion-gem-shell branch from 3daa25c to 86c274f Compare May 28, 2026 18:35

jwils force-pushed the joshuaw/schema-extension-plumbing branch from bb5ee96 to 4336719 Compare May 28, 2026 18:35

myronmarston approved these changes May 29, 2026

View reviewed changes

jwils mentioned this pull request Jun 1, 2026

Add JSON ingestion gem shell #1199

Merged

This was referenced Jun 11, 2026

Make the JSON ingestion spec extension fully opt-in #1249

Open

Remove remaining JSON schema usage from schema definition specs #1250

Closed

jwils force-pushed the joshuaw/schema-extension-plumbing branch from 4336719 to ef183ea Compare June 15, 2026 20:58

jwils requested review from ellisandrews-toast, marcdaniels-toast and rossroberts-toast as code owners June 15, 2026 20:58

jwils force-pushed the joshuaw/json-ingestion-gem-shell branch from 86c274f to dff43f8 Compare June 15, 2026 20:58

myronmarston approved these changes Jun 16, 2026

View reviewed changes

Base automatically changed from joshuaw/json-ingestion-gem-shell to main June 16, 2026 17:39

jwils added 3 commits June 16, 2026 10:39

Add schema definition extension plumbing

93aa587

Move JSON schema version enforcement into schemas

c56b643

Disable schema version enforcement in site examples

abe4889

jwils force-pushed the joshuaw/schema-extension-plumbing branch from ef183ea to abe4889 Compare June 16, 2026 17:39

jwils enabled auto-merge (squash) June 16, 2026 17:40

jwils merged commit ba96d2b into main Jun 16, 2026
25 checks passed

jwils deleted the joshuaw/schema-extension-plumbing branch June 16, 2026 18:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add schema definition extension plumbing#1200

Add schema definition extension plumbing#1200
jwils merged 3 commits into
mainfrom
joshuaw/schema-extension-plumbing

jwils commented May 20, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jwils commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

Risk Assessment

References

Stack

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jwils commented May 20, 2026 •

edited

Loading