Skip to content

Evals fail with VertexAiSessionService because generated session IDs violate Agent Engine constraints #6683

Description

@ftnext

🔴 Required Information

Describe the Bug:

Running an evaluation with VertexAiSessionService fails because ADK generates evaluation session IDs using the following format:

___eval___session___<UUID>

The underscores in this ID violate the Vertex AI Agent Engine Sessions restrictions for custom session IDs. Agent Engine only accepts lowercase letters, digits, and hyphens ([a-z0-9-]), with a letter or digit as the first and last character.

The eval session ID is generated in google/adk/evaluation/local_eval_service.py:

EVAL_SESSION_ID_PREFIX = "___eval___session___"
def _get_session_id() -> str:
return f"{EVAL_SESSION_ID_PREFIX}{str(uuid.uuid4())}"

It is then passed to the configured session service by _get_or_create_eval_session() in google/adk/evaluation/evaluation_generator.py:

return await session_service.create_session(
app_name=app_name,
user_id=user_id,
state=initial_session.state if initial_session else {},
session_id=pinned_session_id or fallback_session_id or str(uuid.uuid4()),
)

When the configured session service is VertexAiSessionService, Agent Engine rejects the generated ID and the evaluation cannot start.

There is also a validation mismatch in VertexAiSessionService. Its current local pattern accepts underscores and uppercase letters:
ref: https://docs.cloud.google.com/gemini-enterprise-agent-platform/scale/sessions/manage-with-api?hl=en#create_a_session

_SESSION_ID_PATTERN = re.compile(r'^[A-Za-z0-9_-]+$')

Therefore, the invalid eval session ID passes ADK-side validation and is rejected only by the remote Agent Engine API.

Steps to Reproduce:

  1. Install ADK with the required extras:

    pip install "google-adk[gcp,eval]==2.6.3"
  2. Configure an Agent Engine-backed session service:

    adk web \
      --session_service_uri="agentengine://projects/<PROJECT>/locations/<LOCATION>/reasoningEngines/<ENGINE_ID>" \
      .
  3. Open the ADK Web UI and create or select an eval set.

  4. Run an evaluation.

  5. Observe that creation of the generated ___eval___session___<UUID> session fails.

The problem can also be isolated through the session creation endpoint:

Request Result
No custom session ID 200 OK
eval-<UUID> 200 OK
___eval___session___<UUID> 500 Internal Server Error

Expected Behavior:

Evaluations should be able to create their temporary sessions when VertexAiSessionService is configured.

Observed Behavior:

Agent Engine rejects the generated session ID:

400 INVALID_ARGUMENT

session_id can only contain lowercase letters, digits and hyphens.
The first and last characters must be a letter or number.

The exception is subsequently exposed by the ADK Web/API server as an HTTP 500, and the evaluation fails before inference begins.

Environment Details:

  • ADK Library Version (pip show google-adk):
    • google-adk==1.38.0
    • google-adk==2.6.3
  • Desktop OS: macOS 26.4.1
  • Python Version: Python 3.13.12

Model Information:

  • Are you using LiteLLM: No
  • Which model is being used: N/A — the failure occurs before model inference

🟡 Optional Information

Regression:

This is related to #2731, which reported that VertexAiSessionService did not support user-provided session IDs.

Current ADK releases now pass custom IDs to Agent Engine. However, evals still fail because the ADK-generated ___eval___session___ prefix does not satisfy the Agent Engine ID format.

Issue #2731 was closed as stale rather than by a merged fix, so this appears to be the remaining/current form of the compatibility problem.

Logs:

400 INVALID_ARGUMENT. {
  'error': {
    'code': 400,
    'message': (
      'List of found errors: '
      '1.Field: session_id; Message: session_id can only contain '
      'lowercase letters, digits and hyphens. The first and last '
      'characters must be a letter or number.'
    ),
    'status': 'INVALID_ARGUMENT'
  }
}

Screenshots / Video:

N/A

Additional Context:

I developed adk eval <api_server_url> version, and I found this issue.
https://github.com/ftnext/pytest-adk/releases/tag/0.0.7

Minimal Reproduction Code:

import asyncio

from google.adk.sessions import VertexAiSessionService


async def main() -> None:
  service = VertexAiSessionService(
      project="<PROJECT>",
      location="<LOCATION>",
      agent_engine_id="<ENGINE_ID>",
  )

  await service.create_session(
      app_name="<APP_NAME>",
      user_id="test-user",
      session_id="___eval___session___550e8400-e29b-41d4-a716-446655440000",
  )


asyncio.run(main())

How often has this issue occurred?:

  • Always (100%)

Metadata

Metadata

Assignees

Labels

agent engine[Component] This issue is related to Vertex AI Agent Engineservices[Component] This issue is related to runtime services, e.g. sessions, memory, artifacts, etc

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions