Wire JSON ingestion schema extension modules by jwils · Pull Request #1215 · block/elasticgraph

jwils · 2026-05-27T15:09:34Z

Why

Introduce the JSON ingestion schema-definition extension modules after the indexing extension points exist, but before making JSON ingestion the default implementation.

What

Add ElasticGraph::JSONIngestion::SchemaDefinition::APIExtension and supporting factory/results/artifact/state/schema-element extension modules
Wire JSON ingestion through the factory so core indexing field references and field types are extended in place
Keep the existing core JSON Schema behavior active in this intermediate layer
Add doctest support for JSON ingestion schema-definition examples without making the extension the default yet

Risk Assessment

Medium - this adds new extension code, but the default behavior remains the existing core JSON Schema implementation in this PR.

References

Stacked on Add JSON ingestion indexing extensions #1204.
bundle exec rspec elasticgraph-schema_definition/spec/unit/elastic_graph/schema_definition/json_schema_spec.rb elasticgraph-schema_definition/spec/unit/elastic_graph/schema_definition/json_schema_field_metadata_spec.rb elasticgraph-schema_definition/spec/unit/elastic_graph/schema_definition/indexing/json_schema_with_metadata_spec.rb elasticgraph-schema_definition/spec/unit/elastic_graph/schema_definition/factory_spec.rb passed.
script/type_check passed.
script/lint passed.

Stack

Current PR is marked with ->.

myronmarston

Made it through the end of my review this time! Almost ready.

## Why Split the JSON ingestion extraction into reviewable steps so the gem packaging can be checked independently before code moves. ## What - Add the `elasticgraph-json_ingestion` gem shell - Wire the new path gem into the bundle without changing runtime behavior - Update generated dependency diagrams and gem READMEs for the new empty gem ## Risk Assessment Low — this only adds an empty extension gem package, lockfile wiring, and generated dependency docs. ## References - First PR in the JSON ingestion extraction stack. - Local spellcheck, lint, type-check, schema artifact, and dependency diagram checks passed. ## Stack Current PR is marked with `->`. - -> [#1199 Add JSON ingestion gem shell](#1199) - [#1200 Add schema definition extension plumbing](#1200) - [#1201 Move JSON schema helpers to json ingestion gem](#1201) - [#1203 Namespace JSON schema helpers under JSON ingestion](#1203) - [#1204 Add JSON ingestion indexing extensions](#1204) - [#1215 Wire JSON ingestion schema extension modules](#1215) - [#1205 Use JSON ingestion as schema definition extension](#1205) - [#1202 Document JSON ingestion schema APIs](#1202) - [#1224 Migrate JSON ingestion specs into gem](#1224) - [#1220 Add configurable indexing event decoder](#1220)

## Why Prepare schema definition for ingestion serializer extensions before moving JSON Schema code into its own gem. ## What - Add schema definition `extension_modules` plumbing for APIs, factories, and rake tasks - Apply duplicate extension modules only once - Replace the rake-task sidecar enforcement option with schema-level `schema.enforce_json_schema_version false` - Update project templates, tests, and docs to configure JSON schema enforcement in schema definitions ## Risk Assessment Medium — this touches schema definition setup, but JSON Schema behavior remains in `elasticgraph-schema_definition` in this PR. ## References - Stacked on #1199. - Local spellcheck, lint, type-check, targeted specs, and schema artifact checks passed. ## Stack Current PR is marked with `->`. - [#1199 Add JSON ingestion gem shell](#1199) - -> [#1200 Add schema definition extension plumbing](#1200) - [#1201 Move JSON schema helpers to json ingestion gem](#1201) - [#1203 Namespace JSON schema helpers under JSON ingestion](#1203) - [#1204 Add JSON ingestion indexing extensions](#1204) - [#1215 Wire JSON ingestion schema extension modules](#1215) - [#1205 Use JSON ingestion as schema definition extension](#1205) - [#1202 Document JSON ingestion schema APIs](#1202) - [#1224 Migrate JSON ingestion specs into gem](#1224) - [#1220 Add configurable indexing event decoder](#1220)

## Why Keep the first extraction layer reviewable by moving the JSON Schema helper files without changing their contents, so GitHub can render them as renames instead of delete/add rewrites. ## What - Move JSON Schema helper Ruby and RBS files into `elasticgraph-json_ingestion` - Update core requires/type references to load the helpers from the new gem - Keep `elasticgraph-json_ingestion` optional at runtime by using a schema_definition development dependency instead of a runtime dependency - Add the Docker/Apollo build-context wiring needed for bundles that include the new path gem explicitly ## Risk Assessment Low to medium - mostly file moves plus require/dependency wiring. ## References - Stacked on #1200. - `git diff -M --summary` reports the moved helper files as 100% renames, including `json_schema_with_metadata.rb`. - Local bundle install, lint, type check, spellcheck, dependency diagram verification, schema artifact check, and targeted unit specs passed on the restacked branch. ## Stack Current PR is marked with `->`. - [#1199 Add JSON ingestion gem shell](#1199) - [#1200 Add schema definition extension plumbing](#1200) - -> [#1201 Move JSON schema helpers to json ingestion gem](#1201) - [#1203 Namespace JSON schema helpers under JSON ingestion](#1203) - [#1204 Add JSON ingestion indexing extensions](#1204) - [#1215 Wire JSON ingestion schema extension modules](#1215) - [#1205 Use JSON ingestion as schema definition extension](#1205) - [#1202 Document JSON ingestion schema APIs](#1202) - [#1224 Migrate JSON ingestion specs into gem](#1224) - [#1220 Add configurable indexing event decoder](#1220)

## Why After the move-only PR preserves rename detection, put the moved helper constants under the `ElasticGraph::JSONIngestion` namespace. ## What - Namespace the moved JSON Schema helper classes and RBS signatures - Update schema_definition references and tests to use the new qualified constants ## Risk Assessment Medium - namespace-only behavior should be unchanged, but it updates call sites across schema_definition. ## References - Stacked on #1201. - Local lint, type check, schema artifact checks, and documentation coverage passed while building this split. ## Stack Current PR is marked with `->`. - [#1199 Add JSON ingestion gem shell](#1199) - [#1200 Add schema definition extension plumbing](#1200) - [#1201 Move JSON schema helpers to json ingestion gem](#1201) - -> [#1203 Namespace JSON schema helpers under JSON ingestion](#1203) - [#1204 Add JSON ingestion indexing extensions](#1204) - [#1215 Wire JSON ingestion schema extension modules](#1215) - [#1205 Use JSON ingestion as schema definition extension](#1205) - [#1202 Document JSON ingestion schema APIs](#1202) - [#1224 Migrate JSON ingestion specs into gem](#1224) - [#1220 Add configurable indexing event decoder](#1220)

## Why Keep the JSON ingestion indexing extensions separate from the schema-extension wiring so the indexing behavior is easier to review. ## What - Add JSON ingestion extension modules for indexing field references and field types - Make core indexing field references and enum/scalar/union field types extendable via the existing MemoizableData helper, matching the object field type - Keep JSON ingestion behavior in extension modules; no wrapper classes ## Risk Assessment Low to medium - this changes internal indexing value objects to use the existing MemoizableData wrapper so JSON ingestion can attach extension modules, without changing core mapping behavior. ## References - Stacked on #1203. - `bundle exec rspec elasticgraph-schema_definition/spec/unit/elastic_graph/schema_definition/json_schema_spec.rb elasticgraph-schema_definition/spec/unit/elastic_graph/schema_definition/json_schema_field_metadata_spec.rb elasticgraph-schema_definition/spec/unit/elastic_graph/schema_definition/indexing/json_schema_with_metadata_spec.rb elasticgraph-schema_definition/spec/unit/elastic_graph/schema_definition/factory_spec.rb` passed. - `script/type_check` passed. - `script/lint` passed. ## Stack Current PR is marked with `->`. - [#1199 Add JSON ingestion gem shell](#1199) - [#1200 Add schema definition extension plumbing](#1200) - [#1201 Move JSON schema helpers to json ingestion gem](#1201) - [#1203 Namespace JSON schema helpers under JSON ingestion](#1203) - -> [#1204 Add JSON ingestion indexing extensions](#1204) - [#1215 Wire JSON ingestion schema extension modules](#1215) - [#1205 Use JSON ingestion as schema definition extension](#1205) - [#1202 Document JSON ingestion schema APIs](#1202) - [#1224 Migrate JSON ingestion specs into gem](#1224) - [#1220 Add configurable indexing event decoder](#1220)

## Why Complete the extraction by loading JSON ingestion through the schema-definition extension API and removing the duplicated core JSON Schema implementation. ## What - Wire `ElasticGraph::JSONIngestion::SchemaDefinition::APIExtension` into app, template, doctest, and spec-helper schema definition setup - Move JSON Schema artifact/results behavior out of core schema_definition - Remove JSON Schema field/type/index logic from core classes now provided by the extension - Keep JSON schema version enforcement configurable through the JSON ingestion extension - Update the generated JSON ingestion dependency diagram ## Risk Assessment Medium to high - this is the behavior-transfer layer, but it sits after the move/namespace/wrapper/wiring PRs so the remaining diff is mostly deletion and wiring. ## References - Stacked on #1215. - Local `script/quick_build` passed before the generated diagram update; `script/update_dependency_diagrams --verify` and README spellcheck passed after it. ## Stack Current PR is marked with `->`. - [#1199 Add JSON ingestion gem shell](#1199) - [#1200 Add schema definition extension plumbing](#1200) - [#1201 Move JSON schema helpers to json ingestion gem](#1201) - [#1203 Namespace JSON schema helpers under JSON ingestion](#1203) - [#1204 Add JSON ingestion indexing extensions](#1204) - [#1215 Wire JSON ingestion schema extension modules](#1215) - -> [#1205 Use JSON ingestion as schema definition extension](#1205) - [#1202 Document JSON ingestion schema APIs](#1202) - [#1224 Migrate JSON ingestion specs into gem](#1224) - [#1220 Add configurable indexing event decoder](#1220)

## Why Document the JSON ingestion schema-definition APIs after extracting JSON ingestion into its own extension and making it part of generated projects by default. ## What - Document the default JSON ingestion setup and explicit extension-module wiring - Document `json_schema_version`, `json_schema_strictness`, scalar JSON Schema metadata, field/type validations, and field-level `nullable: false` ## Risk Assessment Low - README-only documentation change. Snippets are validated. ## References - Stacked on #1205. - Local `script/quick_build` passed before the generated diagram restack; `script/update_dependency_diagrams --verify` and README spellcheck passed after it. ## Stack Current PR is marked with `->`. - [#1199 Add JSON ingestion gem shell](#1199) - [#1200 Add schema definition extension plumbing](#1200) - [#1201 Move JSON schema helpers to json ingestion gem](#1201) - [#1203 Namespace JSON schema helpers under JSON ingestion](#1203) - [#1204 Add JSON ingestion indexing extensions](#1204) - [#1215 Wire JSON ingestion schema extension modules](#1215) - [#1205 Use JSON ingestion as schema definition extension](#1205) - -> [#1202 Document JSON ingestion schema APIs](#1202) - [#1224 Migrate JSON ingestion specs into gem](#1224) - [#1220 Add configurable indexing event decoder](#1220)

## Why The JSON ingestion gem should own the JSON-schema tests that cover its extracted implementation so the gem can enforce full coverage without a SimpleCov skip. ## What - Move JSON schema generation, metadata merge, pruner, and matcher specs into `elasticgraph-json_ingestion`. - Add JSON-ingestion artifact-manager integration coverage for versioned schema artifacts, version-bump enforcement, metadata refresh, and schema evolution diagnostics. - Add focused wrapper/scalar specs for the remaining extracted JSON-ingestion paths. - Remove the temporary SimpleCov filter for `elasticgraph-json_ingestion`. ## Verification - `script/run_gem_specs elasticgraph-json_ingestion` - `script/type_check elasticgraph-json_ingestion` - `BUNDLE_GEMFILE=Gemfile bundle exec rspec --format progress spec/integration/elastic_graph/schema_definition/rake_tasks_spec.rb spec/unit/elastic_graph/schema_definition/runtime_metadata/scalar_types_by_name_spec.rb` from `elasticgraph-schema_definition/` - `bundle exec standardrb ...` on touched specs/support `script/run_gem_specs elasticgraph-schema_definition` and `script/run_gem_specs elasticgraph-json_ingestion` both pass with 100% coverage (with the test datastore booted). ## Follow-ups - The migrated spec files keep the compact `module JSONIngestion::SchemaDefinition` form so that git/GitHub detects them as file moves. A follow-up PR (after this stack merges) will convert them to the standard nested module style and re-indent. - The JSON-schema-versioning scenarios that the new `schema_artifact_manager_extension_spec.rb` covers have been removed from the core `rake_tasks_spec.rb`; the core lines they exercised are now covered by focused unit specs instead. ## Stack Current PR is marked with `->`. - [#1199 Add JSON ingestion gem shell](#1199) - [#1200 Add schema definition extension plumbing](#1200) - [#1201 Move JSON schema helpers to json ingestion gem](#1201) - [#1203 Namespace JSON schema helpers under JSON ingestion](#1203) - [#1204 Add JSON ingestion indexing extensions](#1204) - [#1215 Wire JSON ingestion schema extension modules](#1215) - [#1205 Use JSON ingestion as schema definition extension](#1205) - [#1202 Document JSON ingestion schema APIs](#1202) - -> [#1224 Migrate JSON ingestion specs into gem](#1224) - [#1220 Add configurable indexing event decoder](#1220)

jwils mentioned this pull request May 27, 2026

Use JSON ingestion as schema definition extension #1205

Merged

jwils force-pushed the joshuaw/json-ingestion-api-polish branch from 3436470 to ea857eb Compare May 27, 2026 16:22

jwils force-pushed the joshuaw/json-ingestion-extension-modules branch from b80bef4 to bce83e3 Compare May 27, 2026 16:22

jwils force-pushed the joshuaw/json-ingestion-api-polish branch from ea857eb to 7af1788 Compare May 27, 2026 18:43

jwils force-pushed the joshuaw/json-ingestion-extension-modules branch from bce83e3 to 38d7488 Compare May 27, 2026 18:43

myronmarston mentioned this pull request May 28, 2026

Add JSON ingestion indexing extensions #1204

Merged

jwils force-pushed the joshuaw/json-ingestion-extension-modules branch from 38d7488 to 65c8468 Compare May 28, 2026 18:35

jwils force-pushed the joshuaw/json-ingestion-api-polish branch from 7af1788 to 20ee5dd Compare May 28, 2026 18:35

jwils force-pushed the joshuaw/json-ingestion-extension-modules branch from 65c8468 to d780944 Compare May 28, 2026 18:44

jwils force-pushed the joshuaw/json-ingestion-api-polish branch 2 times, most recently from 5b67eda to 28d3b58 Compare May 28, 2026 19:01

jwils force-pushed the joshuaw/json-ingestion-extension-modules branch 2 times, most recently from 08fa741 to 66a50ff Compare May 28, 2026 19:13

jwils force-pushed the joshuaw/json-ingestion-api-polish branch 2 times, most recently from 4f85649 to fd4651f Compare May 30, 2026 14:07

jwils force-pushed the joshuaw/json-ingestion-extension-modules branch 2 times, most recently from e904b45 to 7cd0f7d Compare May 30, 2026 14:26

jwils force-pushed the joshuaw/json-ingestion-api-polish branch 3 times, most recently from 2e3770f to 9ed0d47 Compare May 30, 2026 14:38

jwils force-pushed the joshuaw/json-ingestion-extension-modules branch from 3d879aa to 01d66b2 Compare May 30, 2026 20:03

jwils force-pushed the joshuaw/json-ingestion-api-polish branch from 9ed0d47 to a02256c Compare May 30, 2026 20:03

jwils force-pushed the joshuaw/json-ingestion-extension-modules branch from 01d66b2 to 4d549ec Compare May 30, 2026 20:17

jwils force-pushed the joshuaw/json-ingestion-api-polish branch from a02256c to 180c7d3 Compare May 30, 2026 20:17

jwils marked this pull request as ready for review May 31, 2026 03:58

jwils requested review from BrianSigafoos-SQ, ayousufi, bsorbo, jwondrusch and myronmarston as code owners May 31, 2026 03:58

jwils force-pushed the joshuaw/json-ingestion-api-polish branch 3 times, most recently from e28ff9c to 83f6c2a Compare June 1, 2026 18:57

myronmarston requested changes Jun 2, 2026

View reviewed changes

jwils force-pushed the joshuaw/json-ingestion-api-polish branch 2 times, most recently from 700dc21 to 8f06402 Compare June 4, 2026 14:15

myronmarston approved these changes Jun 5, 2026

View reviewed changes

This was referenced Jun 11, 2026

Make the JSON ingestion spec extension fully opt-in #1249

Open

Remove remaining JSON schema usage from schema definition specs #1250

Closed

jwils force-pushed the joshuaw/json-ingestion-extension-modules branch from 4d549ec to d097cc0 Compare June 15, 2026 20:58

jwils requested review from ellisandrews-toast, marcdaniels-toast and rossroberts-toast as code owners June 15, 2026 20:58

jwils force-pushed the joshuaw/json-ingestion-api-polish branch from 8f06402 to 471aa04 Compare June 15, 2026 20:58

myronmarston approved these changes Jun 16, 2026

View reviewed changes

jwils force-pushed the joshuaw/json-ingestion-extension-modules branch from d097cc0 to a906b8f Compare June 16, 2026 19:05

Base automatically changed from joshuaw/json-ingestion-extension-modules to main June 16, 2026 19:47

Wire JSON ingestion schema extension modules

6220a62

jwils force-pushed the joshuaw/json-ingestion-api-polish branch from 471aa04 to 6220a62 Compare June 16, 2026 19:48

jwils enabled auto-merge (squash) June 16, 2026 19:48

jwils merged commit 5c0e6e1 into main Jun 16, 2026
25 checks passed

jwils deleted the joshuaw/json-ingestion-api-polish branch June 16, 2026 20:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wire JSON ingestion schema extension modules#1215

Wire JSON ingestion schema extension modules#1215
jwils merged 1 commit into
mainfrom
joshuaw/json-ingestion-api-polish

jwils commented May 27, 2026 •

edited

Loading

Uh oh!

myronmarston left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jwils commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

Risk Assessment

References

Stack

Uh oh!

myronmarston left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jwils commented May 27, 2026 •

edited

Loading