Content addressable gems support by jenshenny · Pull Request #9674 · ruby/rubygems

jenshenny · 2026-07-03T23:33:40Z

What was the end-user or developer problem that led to this PR?

What is your fix for the problem, implemented in this PR?

Content-addressable ("skinny") gems put a content address (a SHA-256 prefix) where the platform normally goes in a gem's filename, so multiple per-Ruby-ABI builds can coexist for one version (darwin-demo-1.0.0-bd0ec167.gem). This adds the consumer side to RubyGems — resolve, install, activate, fetch. Building skinny gems is out of scope.

Notable design decisions

Content address is a separate field, not overloaded onto Gem::Platform. @platform stays the real platform; the content address is its own attribute and full_name becomes name-version-<address>. Putting the SHA inside the platform (the earlier version_suffix approach) feeds a non-platform value into match_gem?/specificity/sorting, where it parses to a bogus os="unknown" — so we keep those paths clean and thread one new field instead.
Real platform lives on a # stub-target: line, not a 5th # stub: field. The platform slot on # stub: is already the content address, so the real platform has to go elsewhere. A separate key=value comment line is (1) ignored by older parsers instead of breaking the positional format, and (2) extensible — a future ruby=~> 3.4.0 key could make activation ABI-aware with no full-spec load.
content_address is :nodoc: and not a gemspec attribute. It's the SHA of the built .gem, so it can't be authored — excluded from @@attributes, and gem build warns if set (like autorequire). But to_ruby writes it into the installed gemspec body (like installed_by_version) so a loaded spec still carries it.
Install verifies and fails loud. The filename token is re-hashed against the file; a mismatch raises Gem::InstallError rather than silently installing under the fat name. (Self-consistency check; the trust boundary is still the registry checksum.)
Specificity first, content-addressed as a tiebreak — not "skinny always wins." build_spec_for_cache sorts by [specificity, content_addressable?, source], so a more-specific fat still beats a less-specific skinny.
Activation is ABI-aware only when it must be. When skinny variants of the same name-version-platform coexist in one GEM_HOME, _resort! prefers the running-Ruby build. It's reached only when name+version+platform all tie (only CA variants can) and loads a spec only when a CA variant is involved, so ordinary default+installed duplicates stay lazy — no hot-path regression.
Gem::NameTuple gains content_address (in ==/hash) so variants stay distinct and display correctly. to_a stays the 3-field wire tuple; the SafeMarshal allowlist gains the ivar — additive and backward-compatible.

Make sure the following tasks are checked

Describe the problem / feature
Write tests for features and bug fixes
Write code to solve the problem
Make sure you follow the current code style and write meaningful commit messages without tags

Content-addressable ("skinny") gems replace the platform slot in a gem's filename with a content address (a hex prefix of the gem's SHA-256), letting multiple per-Ruby-ABI precompiled binaries coexist for one version. - Gem::BasicSpecification: content_address primitive, derived content_addressable?, content-addressed full_name, and the CONTENT_ADDRESS format predicate. - Gem::Specification: settable content_address; to_ruby writes the SHA in the stub slot, a '# stub-target: platform=' line, and s.content_address in the body. - Gem::StubSpecification: parse the stub slot + stub-target; content-addressed full_name on the fast path. - Gem::Installer: derive the content address from the .gem filename, verify it against the file's SHA-256, and raise Gem::InstallError on mismatch. - Resolver (APISet/APISpecification/Resolver): parse the compact-index platform:= requirement into a content address + real platform, and prefer a content-addressable variant among equally-specific candidates. - SpecificationPolicy: warn if a gemspec sets content_address (assigned at install). Assisted-By: devx/4ddf66ed-6c08-4b3b-b805-28b76b59ebd4

gem install --explain (and the resolver's handling of an already-installed skinny) printed the fat name-version-platform instead of the content-addressed name-version-<sha>, because Gem::NameTuple -- which ActivationRequest#full_name builds from -- had no field for the content address, and SpecSpecification never forwarded it. Give Gem::NameTuple a content_address field (with matching ==/eql?/hash so two skinny variants of the same name-version-platform don't collapse under uniq) and have Gem::Resolver::SpecSpecification delegate content_address to its backing spec. ActivationRequest#full_name now reports the content-addressed name for both freshly-resolved and already-installed skinnies. The SafeMarshal allowlist permits the new marshaled ivar; older cached tuples simply lack it. Assisted-By: devx/4ddf66ed-6c08-4b3b-b805-28b76b59ebd4

When several content-addressable variants of the same name-version-platform are installed in one GEM_HOME (e.g. a GEM_HOME shared across Ruby versions), activation/require chose one arbitrarily and could load a native extension built for a different Ruby ABI -- or a fat gem over a matching skinny. _resort! now breaks that tie in favour of the build whose required_ruby_version matches the running Ruby, then the content-addressed one. The tiebreak is only reached when name, version, and platform all tie, and only loads a full spec when a content-addressable variant is actually involved, so ordinary duplicates (a default gem plus an installed copy) stay lazy. Assisted-By: devx/4ddf66ed-6c08-4b3b-b805-28b76b59ebd4

gem fetch downloaded the fat gem and gem list --remote showed the SHA as a platform, because the SpecFetcher/NameTuple index path treated the content address (carried in the /versions platform slot) as a bogus platform and rejected the skinny as a platform mismatch. Teach that path about content addresses so gem fetch mirrors gem install: load_compact_index_specs records a content-address token as the tuple's content_address; search_for_dependency skips the platform-mismatch rejection for such tuples (their slot holds a SHA, not a platform -- the fetched spec supplies the real platform); fetch_spec fetches the content-addressed gemspec and carries the address onto it; and FetchCommand prefers the running-Ruby content-addressed build. Assisted-By: devx/4ddf66ed-6c08-4b3b-b805-28b76b59ebd4

OughtPuts

Nice job - sorry it's taken me a while to digest. Some initial thoughts but always happy to take a second / repeated look!

OughtPuts · 2026-07-10T08:27:21Z

-    case @platform
-    when nil, "", Gem::Platform::RUBY
-      "#{@name}-#{@version}"
+    if @content_address


Nit: just for readability we could do if content_addressable? here, since we have the method anyway.

OughtPuts · 2026-07-10T08:46:44Z

    tuples = []

    versions.each_value do |rows|
      gem_tuples = rows.filter_map do |name, version_string, platform|


Appreciate the comment on line 267 - it definitely helps a lot - but I can't get past how brain twisty Gem::NameTuple.new(name, version, platform, platform) is.

I think on balance it's better to change this to:

Suggested change

gem_tuples = rows.filter_map do |name, version_string, suffix|

And then add the comment above line 272 that suffix is a platform in this case, to make that a bit more explicit.

OughtPuts · 2026-07-10T10:52:39Z

+    end
+    specs_and_sources = compatible unless compatible.empty?
+
+    skinny = specs_and_sources.select {|s,| s.content_addressable? }


As per the server-side work are we trying to avoid the term 'skinny' as a rule of thumb as it could be a bit obscure for maintainers?

For the sake of cohesion, what about:

Suggested change

skinny = specs_and_sources.select {|s,| s.content_addressable? }

content_addressed = specs_and_sources.select {|s,| s.content_addressable? }

OughtPuts · 2026-07-10T10:53:03Z

+    specs_and_sources = compatible unless compatible.empty?
+
+    skinny = specs_and_sources.select {|s,| s.content_addressable? }
+    skinny.empty? ? specs_and_sources : skinny


+1 for 'skinny' here!

OughtPuts · 2026-07-10T17:55:36Z

+    return specs_and_sources unless specs_and_sources.any? {|s,| s.content_addressable? }
+
+    compatible = specs_and_sources.select do |spec,|
+      spec.required_ruby_version.satisfied_by?(Gem.ruby_version)


Question - are the spec objects NameTuples here or Specification? I expected that they would be NameTuple objects which would mean they wouldn't have access to required_ruby_version but I may have the mental model wrong.

OughtPuts · 2026-07-10T17:58:41Z

+    suffix = api_data[:suffix].freeze
+    required_platform = required_platform_from(api_data.dig(:requirements, :platform))
+
+    if required_platform


We could choose to add a check in here that the suffix is definitely a content_address:

Suggested change

if required_platform

if required_platform &&

Gem::BasicSpecification.content_address?(suffix)

jenshenny added 4 commits July 3, 2026 19:36

jenshenny force-pushed the rubygems-content-addressable branch from 2de9f81 to 8574341 Compare July 3, 2026 23:37

OughtPuts reviewed Jul 10, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Content addressable gems support#9674

Content addressable gems support#9674
jenshenny wants to merge 4 commits into
ruby:masterfrom
Shopify:rubygems-content-addressable

jenshenny commented Jul 3, 2026 •

edited

Loading

Uh oh!

OughtPuts left a comment

Uh oh!

OughtPuts Jul 10, 2026

Uh oh!

OughtPuts Jul 10, 2026

Uh oh!

OughtPuts Jul 10, 2026

Uh oh!

OughtPuts Jul 10, 2026

Uh oh!

OughtPuts Jul 10, 2026

Uh oh!

OughtPuts Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


	gem_tuples = rows.filter_map do \|name, version_string, suffix\|

	skinny = specs_and_sources.select {\|s,\| s.content_addressable? }
	content_addressed = specs_and_sources.select {\|s,\| s.content_addressable? }

	if required_platform
	if required_platform &&
	Gem::BasicSpecification.content_address?(suffix)

Uh oh!

Uh oh!

Conversation

jenshenny commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was the end-user or developer problem that led to this PR?

What is your fix for the problem, implemented in this PR?

Notable design decisions

Make sure the following tasks are checked

Uh oh!

OughtPuts left a comment

Choose a reason for hiding this comment

Uh oh!

OughtPuts Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

OughtPuts Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

OughtPuts Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

OughtPuts Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

OughtPuts Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

OughtPuts Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jenshenny commented Jul 3, 2026 •

edited

Loading