[larrymou9]

Overview

This PR adds an index on ContentArtifact.relative_path and changes the content handler's lookups to use it (PULP-2346).

What changes

• Index: A new migration adds core_ca_relpath_pattern_idx, a text_pattern_ops btree on relative_path. Our database uses en_US.UTF-8, where a default index can't serve LIKE 'prefix%'. Production already has this index from a downstream migration, so the migration reuses it when it is valid and has the expected definition.
• Lookups: The handler's directory listing, index.html probe and file lookups now check membership with a correlated EXISTS over RepositoryContent. They no longer use content__in=repo_version.content, which unnests the whole content_ids array.

Why: We ran this as an A/B experiment in production. B averaged 4.44 against 12.46 for A, and p99 was 8.96 against 260.9 (kdelee's Nov 4 numbers below).

Where I'd like review: The migration's handling of an already-existing index, and the test coverage. I used database-backed unit tests, because pulp_file distributions serve from publications and never reach these handler paths.

Summary

Makes the validated PULP-2346 Variant B behavior permanent and adds the index it relies on.

• The content handler's directory listing, index.html probe, exact file lookup and pass-through file lookup check membership per matching ContentArtifact row with a correlated EXISTS over the RepositoryContent interval, rather than content__in=repo_version.content.
• The index.html probe is existence-only (values_list("pk", flat=True)[:1]) and always runs, including for dotted final path segments such as 12.1.12, which can be directories.
• New migration 0162 adds core_ca_relpath_pattern_idx (text_pattern_ops on core_contentartifact.relative_path) with CREATE INDEX CONCURRENTLY IF NOT EXISTS. It reuses an existing index only if it is valid and has exactly the expected definition (btree on relative_path with text_pattern_ops); an invalid or differently defined index of the same name makes the migration fail instead of being reused or dropped, and the reverse migration only drops an index it created (tracked via an index comment). Model state declares the same index, so makemigrations --check stays clean.

Why text_pattern_ops

• The handler's directory lookups filter with relative_path LIKE 'prefix%' (Django startswith), and exact lookups use =.
• A default btree orders text by the database collation. Production uses en_US.UTF-8, not the C locale, so Postgres can't turn LIKE 'prefix%' into a range scan on a default index.
• text_pattern_ops compares strictly character by character, so a prefix becomes one contiguous range and LIKE 'prefix%' becomes an index range scan. Equality can use it too. <, <=, > and >= cannot, but the existing default index still covers those. See the PostgreSQL docs.
• It does not help ILIKE, a leading wildcard or unanchored patterns. Our lookups are case-sensitive and anchored, so it fits.

Checked on the production read replica via gabi: datcollate = en_US.UTF-8, datctype = en_US.UTF-8, encoding UTF8. The existing core_ca_relpath_pattern_idx is valid and is 2032 MB (2,130,673,664 bytes).

Indexes on core_contentartifact on the production read replica, largest first (partial list, as returned by gabi):

Index	Valid	Size	Bytes
core_contentartifact_artifact_id_f5d9a66b	yes	8378 MB	8,784,871,424
core_contentartifact_content_id_04d8a3d5	yes	6824 MB	7,155,376,128
core_contentartifact_pkey	yes	5268 MB	5,523,603,456
core_conten_relativ_ca1ae5_idx	yes	4764 MB	4,995,031,040
core_ca_relpath_pattern_idx (this PR)	yes	2032 MB	2,130,673,664

Production A/B results

Variant A is the previous code, variant B is this change, dispatched 50/50 per call in production. Window: 2026-11-01 20:29:38 to 2026-11-02 20:29:38 UTC (24 hours). Values are exactly as emitted by our CloudWatch collector (the output does not state a unit); only the columns and the repository ordering are unchanged.

Overall

Experiment	A n	B n	A avg	B avg	Change	A p50	B p50	A p95	B p95	A p99	B p99
PULP-2346	973	997	19.5208	6.0312	-69.1%	5.9128	3.7748	13.6221	7.5609	600.5234	10.3795

B is better at p50, p95 and p99 overall.

PULP-2346, repository by repository

Repository	A n	B n	A avg	B avg	Change	A p50	B p50	A p95	B p95	A p99	B p99
0198ece9-d134-77f2-9338-0152fd86704e	31	43	11.915	5.101	-57.2%	5.012	4.651	51.324	8.528	51.767	9.98
019c2424-06b1-7ed7-8e55-b955a14f99dd	695	712	16.3985	4.1769	-74.5%	6.0503	3.5872	9.8834	6.7197	838.5169	9.0875
019f230c-32a3-7796-b895-8b536c1460d3	69	63	60.5277	5.074	-91.6%	11.776	4.347	590.777	8.284	605.955	10.383
019f3818-8450-710a-bbf9-0d3ccbb483f6	13	10	74.3241	165.0992	+122.1%	4.44	4.355	308.098	1596.034	308.098	1596.034
019f3d82-4e2c-7bb2-bf5a-e9b2f1475ef0	1	1	2.75	6.109	+122.1%	2.75	6.109	2.75	6.109	2.75	6.109
019f42bb-1530-7407-92bc-104bbaa8a9ce	3	1	7.73	8.109	+4.9%	4.808	8.109	14.115	8.109	14.115	8.109
019fd363-bd99-7e7c-8bc1-205ec94a0836	161	167	12.7876	4.9996	-60.9%	3.7223	4.7078	81.5976	7.9008	127.3039	15.2814

Four repositories with meaningful sample sizes improved on average by 57-92%. Three show B slower on average (+122%, +122%, +4.9%); they are discussed below.

Update from the 2026-11-04 results (kdelee's PR comment)

Later CloudWatch results from the same experiment, with many more samples. The comment does not state the time window. The Nov 2 tables above are kept unchanged. The PULP-2505 row in that comment is a separate experiment and is omitted here.

Overall:

Experiment	A n	B n	A avg	B avg	Change	A p50	B p50	A p95	B p95	A p99	B p99
PULP-2346	2824	2925	12.4624	4.4425	-64.4%	5.6697	3.9366	12.9322	7.2428	260.9212	8.9612

Repository by repository:

Repository	A n	B n	A avg	B avg	Change	A p50	B p50	A p95	B p95	A p99	B p99
0198ece9-d134-77f2-9338-0152fd86704e	73	71	11.9743	5.1283	-57.2%	4.77	4.314	46.283	9.471	49.133	23.658
019c2424-06b1-7ed7-8e55-b955a14f99dd	1823	1915	8.0135	4.0873	-49.0%	5.8774	3.6743	9.4581	6.6929	55.6449	7.9483
019f230c-32a3-7796-b895-8b536c1460d3	101	115	77.618	5.4872	-92.9%	11.043	4.7125	592.771	8.9344	606.5557	24.1771
019f3818-8450-710a-bbf9-0d3ccbb483f6	279	273	26.6893	4.9693	-81.4%	5.3184	4.3635	214.2871	8.4819	297.7197	11.221
019f42bb-1530-7407-92bc-104bbaa8a9ce	4	4	8.1368	5.814	-28.5%	3.543	4.412	13.508	7.78	13.508	7.78
019fd363-bd99-7e7c-8bc1-205ec94a0836	546	548	8.063	5.1063	-36.7%	4.0041	4.5008	9.4771	7.4112	120.1338	8.4819

Reading the table: B has the lower average and the lower p99 on all six repositories. B's p50 is slightly higher on two (019fd363: 4.5008 vs 4.0041, and 019f42bb: 4.412 vs 3.543, which has only 4 samples per variant). Repository 019f3d82 (1 A / 1 B on Nov 2) no longer appears in the table.

Outliers and caveats

Three repositories show B slower on average, all with very small samples:

• 019f3818 (13 A / 10 B): avg 74.3 -> 165.1, p95/p99 308 -> 1596. One slow B call accounts for about 97% of B's total; the other nine average roughly 6.1, and the medians are flat (4.44 vs 4.355). We reran the same queries many times and could not reproduce it. We suspect an unrelated circumstance but have not proven it.
• 019f3d82 (1 A / 1 B) and 019f42bb (3 A / 1 B): too few samples to conclude anything.

p99 on small per-repository samples is fragile in both directions: A also has p99 outliers (838 and 606) that B does not.

Update (Nov 4): with more samples, 019f3818 has 279 A / 273 B calls, B averages 4.9693 against 26.6893 for A, and p99 is 11.221 against 297.7197. See the Nov 4 tables above. The Nov 2 reading above is kept for transparency.

Changelog

CHANGES/+optimize-content-membership-queries.bugfix

Co-Authored-By: Claude Sonnet 5

[kdelee]

Looking again at the experiement results in cloudwatch,

=== CloudWatch repository-by-repository A/B comparison ===
Experiment  Event          Repository                            A n   B n   A avg    B avg   Change  A p50   B p50   A p95     B p95   A p99     B p99  
----------  -------------  ------------------------------------  ----  ----  -------  ------  ------  ------  ------  --------  ------  --------  -------
PULP-2346   ab_experiment  0198ece9-d134-77f2-9338-0152fd86704e  73    71    11.9743  5.1283  -57.2%  4.77    4.314   46.283    9.471   49.133    23.658 
PULP-2346   ab_experiment  019c2424-06b1-7ed7-8e55-b955a14f99dd  1823  1915  8.0135   4.0873  -49.0%  5.8774  3.6743  9.4581    6.6929  55.6449   7.9483 
PULP-2346   ab_experiment  019f230c-32a3-7796-b895-8b536c1460d3  101   115   77.618   5.4872  -92.9%  11.043  4.7125  592.771   8.9344  606.5557  24.1771
PULP-2346   ab_experiment  019f3818-8450-710a-bbf9-0d3ccbb483f6  279   273   26.6893  4.9693  -81.4%  5.3184  4.3635  214.2871  8.4819  297.7197  11.221 
PULP-2346   ab_experiment  019f42bb-1530-7407-92bc-104bbaa8a9ce  4     4     8.1368   5.814   -28.5%  3.543   4.412   13.508    7.78    13.508    7.78   
PULP-2346   ab_experiment  019fd363-bd99-7e7c-8bc1-205ec94a0836  546   548   8.063    5.1063  -36.7%  4.0041  4.5008  9.4771    7.4112  120.1338  8.4819 

=== CloudWatch overall A/B comparison ===
Experiment  Event          A n   B n   A avg    B avg   Change  A p50   B p50   A p95     B p95   A p99     B p99  
----------  -------------  ----  ----  -------  ------  ------  ------  ------  --------  ------  --------  -------
PULP-2346   ab_experiment  2824  2925  12.4624  4.4425  -64.4%  5.6697  3.9366  12.9322   7.2428  260.9212  8.9612 
PULP-2505   ab_experiment  72    83    89.6357  26.637  -70.3%  15.002  10.092  1106.772  76.689  1467.707  597.083

The outliers we saw on day one have fallen off, now p99 on all repositories is looking much better!!!

[ggainey]

The added complexity is more than justified by the performance-gain!

The S3-failure is (I believe) unrelated - likely related to #8332

[ggainey]

@larrymou9 if you rebase and repush, the tests will pick up #8332 and should run clean