[agw-cuj-arun-ingress-modar]

[bt-zdunning]

Summary

Two issues block this codelab, and in both cases the visible error does not point at the cause. Both are now diagnosed with fixes verified end to end.

#	Step	Problem
1	8 — IAM permissions	${RE_AGENT_IDENTITY} resolves empty, so eight add-iam-policy-binding calls fail
2	7 — bucket creation	Buckets are created without Uniform Bucket Level Access, which agent identity tokens require

Environment: us-central1, ADK agent agent-crm deployed successfully with --enable-agent-identity, all APIs enabled, followed the codelab exactly.

---

Issue 1 — Agent identity is read from Agent Registry, but is only present on the reasoning engine

What happens

All eight bindings that use ${RE_AGENT_IDENTITY} fail:

ERROR: (gcloud.projects.add-iam-policy-binding) INVALID_ARGUMENT:
  Policy members must be of the form "<type>:<value>".
  reason: PROJECT_SET_IAM_DISALLOWED_MEMBER_TYPE

The bindings in the same section that use other principals (including ${RE_AGENT_ID_SET} for roles/mcp.toolUser) all succeed.

Cause

${RE_AGENT_IDENTITY} is an empty string, so gcloud sends --member="". It is empty because the lookup command returns nothing:

gcloud agent-registry agents list \
  --project=${PROJ_ID} --location=${REGION} --filter="displayName=${RE_AGENT_NAME}" \
  --format="value(attributes.'agentregistry.googleapis.com/system/RuntimeIdentity'.principal)"

Listing the registry raw shows why — there is no entry for the deployed agent, only the built-in Workspace Agent, which has no attributes field at all:

[
  {
    "displayName": "Workspace Agent",
    "location": "global",
    "uid": "agentregistry-00000000-0000-0000-XXXX-XXXXXXXXXXXX",
    "protocols": [ { "interfaces": [ { "url": "https://workspaceagent.googleapis.com/a2a" } ] } ]
  }
]

The agent identity does exist — it is on the reasoningEngines resource, not in the registry:

curl -s -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://${REGION}-aiplatform.googleapis.com/v1beta1/projects/${PROJ_ID}/locations/${REGION}/reasoningEngines/${RE_ENGINE_ID}"

{
  "displayName": "agent-crm",
  "spec": {
    "identityType": "AGENT_IDENTITY",
    "effectiveIdentity": "agents.global.org-<ORG_ID>.system.id.goog/resources/aiplatform/projects/<PROJ_NO>/locations/us-central1/reasoningEngines/<ENGINE_ID>"
  }
}

So the deploy provisioned agent identity correctly; only the lookup path is wrong.

This fails silently: when a --format=value(...) projection resolves to nothing, gcloud prints an empty line and exits 0. The failure surfaces eight commands later as an IAM error with nothing linking back to the lookup.

Workaround

Read the principal from the engine and prepend principal:// (the stored value has no type prefix, mirroring how principalSet:// fronts the attribute path in ${RE_AGENT_ID_SET}):

export RE_AGENT_IDENTITY="principal://$(curl -s \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://${REGION}-aiplatform.googleapis.com/v1beta1/projects/${PROJ_ID}/locations/${REGION}/reasoningEngines/${RE_ENGINE_ID}" \
  | jq -r '.spec.effectiveIdentity')"

[[ "${RE_AGENT_IDENTITY}" == principal://* ]] || { echo "BAD IDENTITY"; return 1; }

for R in storage.objectViewer aiplatform.user cloudtrace.agent \
         monitoring.metricWriter logging.logWriter telemetry.writer \
         serviceusage.serviceUsageConsumer browser ; do
  gcloud projects add-iam-policy-binding ${PROJ_ID} \
    --member="${RE_AGENT_IDENTITY}" --role="roles/${R}" --condition=None
done

Suggested fixes

• Source the principal from spec.effectiveIdentity on the reasoning engine rather than from Agent Registry. It is written by the same deploy that provisions the identity, so it cannot be missing the way the registry entry was here.
• Add a non-empty guard on ${RE_AGENT_IDENTITY} before the bindings run, so the failure reports at the lookup instead of eight commands downstream.
• If the registry lookup is kept, document that the registry entry is written implicitly by --enable-agent-identity. No registration command appears anywhere in the codelab, so when the entry is absent there is nothing obvious to re-run.
• Note that identityType / effectiveIdentity were read from v1beta1.

Open question

Why was no Agent Registry entry created? The deploy ran with --enable-agent-identity and succeeded, the identity was provisioned on the engine, and agentregistry.googleapis.com was enabled before the IAM step ran. If the registry is meant to be the canonical lookup surface for agent principals, the implicit write may be worth a look.

---

Issue 2 — Buckets are created without UBLA, which agent identity tokens require

What happens

With issue 1 worked around, the first step 9 test prints nothing:

curl --no-buffer -s -X POST ".../reasoningEngines/${RE_ENGINE_ID}:streamQuery" ... \
  -d '{"input":{"message":"what are the names of our west customers?","user_id":"test-user"}}' \
  | jq -r --unbuffered '... .content.parts[].text // empty'

Expected output is the customer names. Actual output is nothing at all.

Cause

Visible only in reasoning_engine_stderr — 12 occurrences across both data files:

File "/code/agent/agent.py", line 48, in read_customer_file
  if not blob.exists():
google.api_core.exceptions.PreconditionFailed: 412 GET
  https://storage.mtls.googleapis.com/storage/v1/b/customer-data-<PROJ_NO>/o/customers_west.csv
  The type of authentication token used for this request requires that
  Uniform Bucket Level Access be enabled.

Agent identity tokens cannot be used against a bucket that still has object ACLs enabled. GCS rejects this at the bucket-configuration layer before IAM is consulted — hence a 412 rather than a 403, and it persists regardless of how the IAM bindings are set.

The codelab creates both buckets with no UBLA flag, and there is no follow-up buckets update anywhere:

gcloud storage buckets create gs://${STAGING_BUCKET} --location=${REGION}
gcloud storage buckets create gs://${DATA_BUCKET} --location=${REGION}

This is latent rather than always-broken: GCS has defaulted new buckets to UBLA-on for some time, so on most projects these commands happen to produce a compliant bucket and the codelab passes. Where a project's default leaves ACLs enabled, it fails. The codelab depends on an environment default it never states.

Why it is hard to diagnose

Five steps pass before the failure, and the failure itself is invisible:

1. Bucket creates without UBLA — no warning
2. CRM data uploads successfully
3. Agent deploys, agent identity provisioned
4. IAM bindings applied to the correct principal
5. Query accepted, agent runs, calls its read tool
6. GCS returns 412 — only in stderr logs
7. jq filter drops the event — terminal prints nothing

The test command combines curl -s with jq '... // empty', so a blocked response, an HTTP error body, and an agent-side crash all render identically as no output. Note also that empty output is the expected pass for the redaction test in the same step, which makes silence especially ambiguous here.

Fix

gcloud storage buckets update gs://${DATA_BUCKET}    --uniform-bucket-level-access
gcloud storage buckets update gs://${STAGING_BUCKET} --uniform-bucket-level-access

# verify — the field is a plain boolean, NOT .enabled
gcloud storage buckets describe gs://${DATA_BUCKET} --format=json \
  | jq '.uniform_bucket_level_access'   # → true

After this, the safe prompt returns as documented:

The names of our west customers are Bob Johnson and Alice Brown.

Suggested fixes

• Add --uniform-bucket-level-access to both gcloud storage buckets create calls in step 7, with a note that agent identity tokens require it. One flag, and the codelab no longer depends on an unstated project default.
• Add a troubleshooting note to step 9 pointing at reasoning_engine_stdout / reasoning_engine_stderr when a test returns nothing. The 412 was plainly stated there and would have collapsed the whole investigation into one step.

---

Notes

• Identifiers redacted; all work was on a non-production POC project.
• Both fixes verified end to end through the step 9 safe-prompt test. The redaction and audit-log tests in step 9 had not yet been run when this was written.
• Diagnosed with Claude Opus 5 in Claude Code, driving a live Cloud Shell session.