Skip to content

e2e: add private registry pull/push regression test#7007

Open
lohitkolluri wants to merge 1 commit into
docker:masterfrom
lohitkolluri:e2e/private-registry-pull-push-5965
Open

e2e: add private registry pull/push regression test#7007
lohitkolluri wants to merge 1 commit into
docker:masterfrom
lohitkolluri:e2e/private-registry-pull-push-5965

Conversation

@lohitkolluri
Copy link
Copy Markdown

@lohitkolluri lohitkolluri commented May 26, 2026

Add a privateregistry service (htpasswd auth, port 5001) to the e2e
compose stack and a TestPullPushPrivateRepository test that verifies:

  • unauthenticated docker push / docker pull is rejected with an auth error
  • authenticated docker push / docker pull succeeds

The volume path in compose-env.yaml is resolved relative to the compose
file directory (e2e/), so ./testdata/registry/auth is used instead of
./e2e/testdata/registry/auth. This was the root cause of CI failures in
the earlier attempt (#6940):
the htpasswd file never mounted, the registry never started, and every
operation timed out with a DNS error.

Regression test for #5963.
Carries forward #6940 (original author: @AryanSharma9917).
Closes #5965.

@lohitkolluri
Copy link
Copy Markdown
Author

Closing in favor of #6940, which already implements the same private-registry e2e regression coverage for #5963/#5965.

Happy to help review or rebase #6940 if maintainers want to move it forward.

@lohitkolluri
Copy link
Copy Markdown
Author

Reopened and pushed a fix for the e2e failures in #6940.

Root cause: e2e/compose-env.yaml mounted ./e2e/testdata/registry/auth, but compose resolves volume paths relative to the compose file directory (e2e/), so the htpasswd file was never mounted and privateregistry never started (DNS server misbehaving for the full retry window).

Fix: use ./testdata/registry/auth, rename the service to privateregistry, and add depends_on for the dind engine.

Happy to fold this into #6940 instead if maintainers prefer a single PR.

@lohitkolluri lohitkolluri force-pushed the e2e/private-registry-pull-push-5965 branch 2 times, most recently from aa3a10d to b2e3d39 Compare May 26, 2026 13:46
@lohitkolluri lohitkolluri changed the title e2e/image: add private registry pull/push regression test e2e: add private registry pull/push regression test May 26, 2026
@lohitkolluri
Copy link
Copy Markdown
Author

@vvoland @thaJeztah — when you have a moment, could you please approve the workflow runs for this PR? Any review feedback is very welcome.

Thank you.

@codecov-commenter
Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@lohitkolluri lohitkolluri force-pushed the e2e/private-registry-pull-push-5965 branch 2 times, most recently from 3fcf48d to 2b2f60f Compare May 26, 2026 14:44
Add a privateregistry service (htpasswd auth, port 5001) to the e2e
compose stack and a TestPullPushPrivateRepository test that verifies:

  - unauthenticated push/pull is rejected with an auth error
  - authenticated push/pull succeeds

The volume path in compose-env.yaml is resolved relative to the compose
file directory (e2e/), so use ./testdata/registry/auth, not
./e2e/testdata/registry/auth.

Regression test for docker#5963.
Closes docker#5965.

Signed-off-by: aryansharma9917 <sharmaaryan9837@gmail.com>
Signed-off-by: Lohit Kolluri <lohitkolluri@gmail.com>
@lohitkolluri lohitkolluri force-pushed the e2e/private-registry-pull-push-5965 branch from 2b2f60f to ec2b6e3 Compare May 26, 2026 14:52
@lohitkolluri
Copy link
Copy Markdown
Author

CI failure root-cause analysis and fixes

Three separate issues were identified and resolved in this PR. Each is described below.


1. Private registry unreachable — DNS server misbehaving (all matrix jobs)

Root cause: The privateregistry service in e2e/compose-env.yaml used a bind-mount volume to supply the htpasswd authentication file:

volumes:
  - ./testdata/registry/auth:/auth:ro

The e2e test suite runs inside a Docker container that has the host Docker socket bind-mounted. When docker compose up is invoked from within that container, compose resolves the volume path and passes it as an absolute host-filesystem path to the host Docker daemon. On the CI runner the path /go/src/github.com/docker/cli/e2e/testdata/registry/auth does not exist, so the htpasswd file was never mounted. The registry process exited immediately on startup, its container was removed from the Docker network, and all subsequent DNS lookups for privateregistry returned SERVFAIL — the server misbehaving error that repeated throughout the 90-second retry window.

Fix: Replaced the bind-mount approach with a build: context in compose-env.yaml:

privateregistry:
  build:
    context: ./testdata/registry

A Dockerfile at e2e/testdata/registry/ extends registry:3 and copies the auth/ directory directly into the image. When compose uses a build context, the compose CLI (which runs inside the e2e container and has access to the project files) packages the context into a tar archive and sends it to the Docker daemon — so the htpasswd file is baked into the image at build time and the host-path problem is avoided entirely.


2. Incorrect bcrypt hash in htpasswd

Root cause: The previously committed e2e/testdata/registry/auth/htpasswd contained a hash generated with an unknown password, making it impossible to derive matching credentials for SetupConfigFile. The auth config in fixtures.go used ZTJlOnBhc3N3b3Jk (base64 of e2e:password), but the hash on disk corresponded to a different password, so authenticated operations would have been rejected.

Fix: Regenerated the bcrypt hash for e2e:password using htpasswd -nbB and updated both the htpasswd file and fixtures.go to be consistent.


3. assert.Check on push output always failed — tag included in assertion

Root cause: The assertion on the authenticated push output was:

assert.Check(t, strings.Contains(pushWithAuth.Combined(),
    "The push refers to repository ["+privateImage+"]"), ...)

privateImage is privateregistry:5001/private/alpine:test-private-pull-push, but Docker's push output prints the repository reference without the tag:

The push refers to repository [privateregistry:5001/private/alpine]

The strings.Contains check would never match, so this assertion would always record a test failure even on a fully successful push.

Fix: Strip the tag before constructing the expected string:

privateRepo := privateImage[:strings.LastIndex(privateImage, ":")]
assert.Check(t, strings.Contains(pushWithAuth.Combined(),
    "The push refers to repository ["+privateRepo+"]"), ...)

@lohitkolluri
Copy link
Copy Markdown
Author

Hi @vvoland @thaJeztah — gentle follow-up when you have a moment.

The CI issues called out earlier have been addressed in the latest commit on this branch (registry compose setup, auth/htpasswd alignment, and the push-output assertion). I left a root-cause summary in an earlier comment on this PR for reference.

Could you please:

  1. Approve the workflow runs for this PR (fork workflows are currently waiting on maintainer approval), and
  2. Set the milestone to 29.5.3 so validate-milestone can pass — I don’t have permission to set milestones on this repository.

Happy to make any further changes if anything else comes up in review. Thank you for your time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add e2e test for authorized pull/push

3 participants