Skip to content

fix(serve): fix two P0 resource-reuse bugs (instance-type reuse + Bedrock permission fail-open) - #6109

Open
eliseharvey wants to merge 1 commit into
aws:master-nova-follow-upsfrom
eliseharvey:fix/deploy-reuse
Open

fix(serve): fix two P0 resource-reuse bugs (instance-type reuse + Bedrock permission fail-open)#6109
eliseharvey wants to merge 1 commit into
aws:master-nova-follow-upsfrom
eliseharvey:fix/deploy-reuse

Conversation

@eliseharvey

@eliseharvey eliseharvey commented Jul 27, 2026

Copy link
Copy Markdown

This PR fixes two independent P0 resource-reuse defects found during Nova SDK
dogfooding. Both occur only under reuse_resources=True and share the reuse
code paths, so they are fixed together.

1. instance_type mismatch silently reused

deploy() trusted the endpoint candidate cached by build() in
_reused_endpoint_name without re-validating it. That cache is resolved at
build time, before instance_type (a deploy() argument) is known, so a deploy
on a different instance type reused the wrong endpoint. Separately,
_reused_endpoint_matches_config skipped the instance_type check entirely for
Inference Component (IC) endpoints -- the IC early-return (no ModelName on the
production variant) returned True before reaching the check.

Fix: deploy() now re-validates the cached endpoint against the requested
instance_type and falls back to a fresh discovery on a miss or mismatch.
_reused_endpoint_matches_config checks instance_type -- which lives on the
production variant and is available for every endpoint, including IC ones --
before the IC early-return, so an instance-type mismatch is never silently
reused.

2. Bedrock reuse_resources=True fails open on missing permission

Reuse discovery (find_existing_bedrock_model,
find_active_bedrock_deployment_for_model, find_existing_sagemaker_endpoint)
swallowed every exception and returned None. When the execution role lacked a
read permission (e.g. bedrock:ListTagsForResource), a denied discovery call
was indistinguishable from "nothing to reuse", so reuse fell through to creating
the resource -- which then failed with a confusing
ValidationException: Model with name '...' already exists because the prior
run's resource still held the deterministic name, never surfacing the real
cause.

Fix: a new _reraise_if_access_denied helper re-raises
AccessDeniedException as a PermissionError naming the missing IAM action.
All other errors still fail open (warn and return None), so transient failures
like throttling do not block a deploy.

Testing

86 unit tests pass across test_model_builder.py and test_model_reuse.py,
including new regression guards:

  • IC endpoint instance-type mismatch / match / no-instance-requested
  • deploy() re-validation of the build-time cache (mismatch ignores cache, match reuses it)
  • access-denied raises PermissionError for all three discovery functions
  • non-authorization errors still fail open to None

Issue #, if available:

Description of changes: See above.

By submitting this pull request, I confirm that you can use, modify, copy, and
redistribute this contribution, under the terms of your choice.

…rock permission fail-open)

Two independent reuse defects surfaced during Nova SDK dogfooding, both under
reuse_resources=True. Fixed together as they share the reuse code paths.

--- 1. instance_type mismatch silently reused ---

deploy() trusted the endpoint candidate cached by build() in
_reused_endpoint_name without re-validating it. That cache is resolved at build
time, before instance_type (a deploy() argument) is known, so a deploy on a
different instance type reused the wrong endpoint. Separately,
_reused_endpoint_matches_config skipped the instance_type check entirely for
Inference Component (IC) endpoints: the IC early-return (no ModelName on the
variant) returned True before reaching the check.

deploy() now re-validates the cached endpoint against the requested
instance_type and falls back to a fresh discovery on a miss or mismatch.
_reused_endpoint_matches_config checks instance_type -- which lives on the
production variant and is available for every endpoint, including IC ones --
before the IC early-return, so an instance-type mismatch is never silently
reused.

--- 2. Bedrock reuse_resources=True fails open on missing permission ---

Reuse discovery (find_existing_bedrock_model,
find_active_bedrock_deployment_for_model, find_existing_sagemaker_endpoint)
swallowed every exception and returned None. When the execution role lacked a
read permission (e.g. bedrock:ListTagsForResource), a denied discovery call was
indistinguishable from "nothing to reuse", so reuse fell through to creating
the resource -- which then failed with a confusing
"ValidationException: Model with name '...' already exists" because the prior
run's resource still held the deterministic name, never surfacing the real
cause.

A new _reraise_if_access_denied helper re-raises AccessDeniedException as a
PermissionError naming the missing IAM action; all other errors still fail open
(warn and return None) so transient failures like throttling do not block a
deploy.

Tests: 86 unit tests pass across test_model_builder.py and test_model_reuse.py,
including regression guards for both fixes (IC instance-type mismatch,
build-time cache re-validation, access-denied-raises, and fail-open-on-other-error).
@eliseharvey
eliseharvey marked this pull request as ready for review July 28, 2026 01:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant