diff --git a/AGENTS.md b/AGENTS.md index dba3d8ca..57f012f5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -45,8 +45,9 @@ or one of them only where it is used. | `make int-test-slow` | the `int_test_slow` tests, serially (they build the same Docker tags from module fixtures and race under xdist) | | `make installer-test` | `tst/installer`: `plugin add` / `remove` through the real pip, uv and pipx, offline against wheels built from the checkout, and the container user journey (`python:3.12-slim`, `--network none`) with the PEP 668 refusal; needs uv, pipx and Docker | | `make clean-install-test` | the `clean-install` CI gate: both distributions built as the release builds them, installed into a fresh venv from public PyPI with an allowlisted environment, and `agent-env run hello` run twice by name and checked through the local store; needs Python 3.11, uv and git | +| `npm ci && npm run typecheck && npm run lint && npm run test:smoke` | in `src/agent_env/explorer/ui`: the explorer UI checks the `ui` job runs (it also runs `npm run build:static`); needs Node 22 | | `pytest tst/unit/env/store_test.py -v` | one file; add `--log-cli-level=DEBUG` for debug logs | -| `uv build` | wheel `agentenv_framework--py3-none-any.whl` (`src/agent_env` only) and sdist `agentenv_framework-.tar.gz` (`src`, `tst`, the protocol package) | +| `uv build` | wheel `agentenv_framework--py3-none-any.whl` (`src/agent_env` only, without the UI source) and sdist `agentenv_framework-.tar.gz` (`src`, `tst`, the protocol package, `.github/scripts`) | | `agent-env`, `python -m agent_env.cli` | the CLI; `agent-env config show` prints which config file is in effect and where each section came from | ## Configuration @@ -60,9 +61,10 @@ or one of them only where it is used. `AGENT_ENV_SECRET_STORE`, `AGENT_ENV_RUNNER`) < `configure(...)` in code. - Defaults are local and carry no external coordinates: SQLite document store, filesystem object store, an OCI registry at `localhost:5000`, env-var secret store, `LocalRunner`, `local` Docker - sandboxes for envs and agents. MongoDB, S3, Cloud Storage, ECR, AWS Secrets Manager, Google Cloud - Secret Manager, Modal, E2B, Tensorlake and Tenki exist as implementations and are selected by config - (Tensorlake through the `tensorlake` extra, Tenki through the `tenki` extra). + sandboxes for envs and agents. MongoDB, DynamoDB, Firestore, S3, Cloud Storage, ECR, AWS Secrets + Manager, Google Cloud Secret Manager, and the Modal, E2B, Sail, SmolVM, Tensorlake, Tenki and Vercel + sandboxes exist as implementations and are selected by config (the last five through their extras: + `sail`, `smol`, `tensorlake`, `tenki`, `vercel`; every extra but `smol` is in `dev`). - The model endpoint is unset until `[model] base_url` / `api_key` (or `LITELLM_BASE_URL` / `LITELLM_API_KEY`) is configured. A loopback URL handed into a `local` container (the model endpoint, an agent, peer or gateway on this machine) becomes `host.docker.internal`. On Linux that @@ -96,21 +98,24 @@ and deserialized through a registry. | Package | Contents | |---|---| | `env/` | `Env` base class: a subclass declares `type`, implements `from_dict` and `async deploy(...) -> DeployedEnv`, and may override `reset` and `load_file_artifact_universe`. Built-ins in `env/envs/`: `MCPServerEnv` (`mcp_server`), `MultiEnv` (`multi`, MCP servers plus websites), `ServiceDBEnv` (`service_db`, PostgreSQL), `WebsiteEnv` (`website`), `GatewayEnv` (`gateway_server`). `env/gateway/gateway.py` is the env-side gateway server, container-only code. | -| `artifact/` | Immutable versioned artifacts; every `put()` creates a new version. The document goes to the document store, binary payloads to the object store. Built-ins: `CliArtifact` (`cli`), `FileArtifact` (`file`), `FileArtifactUniverse` (`file_artifact_universe`), `DockerImageArtifact` (`docker_image`), `EnvironmentArtifact` (`environment`; legacy `service`), `EnvironmentUniverseArtifact` (`environment_universe`; legacy `service_universe`), `SkillArtifact` (`skill`). A `DockerImageArtifact` holds a tar.gz of its image, or, written by `put_ref` (`--image-ref` on `env mcp-server put` and `a2a-agent put`), only a registry reference with its tag pinned to a digest, which sandboxes pull, or, written by `put_context`, only its build context, which a VM sandbox builds before running it (`VmSandbox.load_docker_images`) and Modal builds before creating its container (`ModalSandboxProvider.prepare_image`, outside a fallback chain's deadline); anything else that runs images by name (a container on the local provider, a lone server on any provider but Modal) refuses such an image. A build context (`utils/build_context.py`) is what `docker build` would send, `.dockerignore` applied, written so equal files give equal bytes and an equal `source_digest`. | +| `artifact/` | Immutable versioned artifacts; every `put()` creates a new version. The document goes to the document store, binary payloads to the object store. Built-ins: `CliArtifact` (`cli`), `FileArtifact` (`file`), `FileArtifactUniverse` (`file_artifact_universe`), `DockerImageArtifact` (`docker_image`), `EnvironmentArtifact` (`environment`), `EnvironmentUniverseArtifact` (`environment_universe`), `SkillArtifact` (`skill`); `[artifacts] type_aliases` maps another stored spelling of a type onto one of them. A `DockerImageArtifact` holds a tar.gz of its image, or, written by `put_ref` (`--image-ref` on `env mcp-server put` and `a2a-agent put`), only a registry reference with its tag pinned to a digest, which sandboxes pull, or, written by `put_context`, only its build context, which a VM sandbox builds before running it (`VmSandbox.load_docker_images`) and Modal builds before creating its container (`ModalSandboxProvider.prepare_image`, outside a fallback chain's deadline); anything else that runs images by name (a container on the local provider, a lone server on any provider but Modal) refuses such an image. A build context (`utils/build_context.py`) is what `docker build` would send, `.dockerignore` applied, written so equal files give equal bytes and an equal `source_digest`. | | `task/`, `task_step/` | A `Task` holds its `TaskStep`s inline; `Task.run()` executes them as a DAG. `depends_on` (None means all prior steps) gates a step, independent steps run concurrently, `fail_task_on_error` makes a failure fatal or tolerated, `retry_config` rolls a failed span back through the step journal and re-dispatches it. Built-in steps live in `task_step/task_steps/` (`deploy_env`, `deploy_agent`, `prompt_agent`, the verifiers under `verifiers/`, and more). `task/episode/` runs a task with some of its agents played from outside, as a training loop does (`EpisodeSpec`). | -| `store/` | Four store ABCs with local and cloud implementations: `DocumentStore` (SQLite, MongoDB), `ObjectStore` (filesystem, S3, Cloud Storage), `ImageStore` (local OCI registry, ECR), `SecretStore` (env vars or file, AWS Secrets Manager, Google Cloud Secret Manager). `VersionedEntityStore` implements the shared versioned get/put logic, `QueryBuilder` is the immutable chained query API, `store/base.py` holds the error types. A new backend must pass the conformance kits in `tst/store/`. | +| `store/` | Four store ABCs with local and cloud implementations: `DocumentStore` (SQLite, MongoDB, DynamoDB, Firestore with MongoDB compatibility), `ObjectStore` (filesystem, S3, Cloud Storage), `ImageStore` (local OCI registry, ECR), `SecretStore` (env vars or file, AWS Secrets Manager, Google Cloud Secret Manager). `VersionedEntityStore` implements the shared versioned get/put logic, `QueryBuilder` is the immutable chained query API, `store/base.py` holds the error types. `store/routing.py` is the `@local` namespace: while the CLI runs, an `@local/…` id and what an `@local` task's run records go to local stores beside the configured ones, and that run's writes to the configured stores are refused. A new backend must pass the conformance kits in `tst/store/`. | | `config/` | The `Config` singleton (`get_config`, `configure`, `reset_config`) in `config/runtime.py`, file discovery in `config/loader.py`, and `load_impl`, which resolves `module:Class` pointers. `agent_env.store` re-exports the config names for compatibility. | -| `providers/` | `providers/sandbox_providers/` holds the sandbox providers `local`, `modal`, `modal_vm`, `e2b`, `sail_vm` (the `sail` extra), `tensorlake` (the `tensorlake` extra), `tenki` (the `tenki` extra); `[sandbox] default` and `agent_default` accept a comma-separated fallback chain, whose deploys skip a provider that can't run them. A sandbox provider declares what it runs (`ON_THIS_MACHINE`, and for VMs on this machine the address their requests to it arrive at, `REACHES_HOST_AT`; `CONTAINER_ACCEPTS`, `PRIVATE_NETWORK`; `SUPPORTED_SANDBOX_MODES`, derived from whether it implements `create_sandbox` and `create_vm`; and `SANDBOX_ACCEPTS`, the images an agent's sandbox runs, which defaults to those a VM loads when it implements `create_vm` and to `CONTAINER_ACCEPTS` otherwise, so only a provider that implements `create_vm` yet gives an agent a container, like the local one, declares it), each sandbox carries the `platform` of the images it runs (linux/amd64 unless its provider sets another when it creates it; None, any, unchecked, for a local sandbox), and code outside `providers/sandbox_providers/` reads those, never which provider it has. `providers/env_providers/` holds the environment providers: `EnvironmentProvider` (an env's containers and state store) and `EnvironmentGatewayProvider`, which renders a docker-compose for the gateway and its MCP servers inside the sandbox; `providers/env_state/` holds env-state providers (`local_postgres` built in). | +| `providers/` | `providers/sandbox_providers/` holds the sandbox providers `local`, `modal`, `modal_vm`, `e2b`, `sail_vm` (the `sail` extra), `smol_vm` (the `smol` extra), `tensorlake` (the `tensorlake` extra), `tenki` (the `tenki` extra), `vercel` (the `vercel` extra); `[sandbox] default` and `agent_default` accept a comma-separated fallback chain, whose deploys skip a provider that can't run them. A sandbox provider declares what it runs (`ON_THIS_MACHINE`, and for VMs on this machine the address their requests to it arrive at, `REACHES_HOST_AT`; `CONTAINER_ACCEPTS`, `PRIVATE_NETWORK`; `SUPPORTED_SANDBOX_MODES`, derived from whether it implements `create_sandbox` and `create_vm`; and `SANDBOX_ACCEPTS`, the images an agent's sandbox runs, which defaults to those a VM loads when it implements `create_vm` and to `CONTAINER_ACCEPTS` otherwise, so only a provider that implements `create_vm` yet gives an agent a container, like the local one, declares it), each sandbox carries the `platform` of the images it runs (linux/amd64 unless its provider sets another when it creates it; None, any, unchecked, for a local sandbox), and code outside `providers/sandbox_providers/` reads those, never which provider it has. `providers/env_providers/` holds the environment providers: `EnvironmentProvider` (an env's containers and state store), `EnvironmentGatewayProvider` (`gateway`, an env's default), which renders a docker-compose for the gateway and its MCP servers inside the sandbox, and `EnvironmentServerProvider` (`server`), one MCP server in its own container with no gateway in front and no state store; `providers/env_state/` holds env-state providers (`local_postgres` built in). | | `a2a_agent/` | The `A2AAgent` entity (`a2a_agent`), its stores and the validator steps. The protocol package provides the agent-side framework. | -| `runner/` | The `[runner]` seam: `Runner.submit()` returns `(run_id, instance_id)`; `LocalRunner` is built in. | +| `runner/` | The `[runner]` seam: `Runner.submit()` returns a `RunHandle` (`run_id`, `instance_id`) before the run starts; `LocalRunner` is built in. | +| `bundle/` | A bundle is a folder of entities (`envs/`, `agents/`, `artifacts/`, `skills/`, `tasks/.json`, `evals/.toml`) that runs without a registry: `agent-env run` plans it, writes it into the `@local` namespace (reusing each version whose inputs haven't changed) and runs its evals or tasks; `agent-env bundle init` writes a stored task or eval as one, with its references pinned. | +| `eval/` | `Eval` (`eval`), a versioned list of task references, and its store. | +| `plugins/` | Entry-point plugins (see Extension points): the group names, `inventory()`, which `agent-env plugin list/show/check` report, `load_failures` and `settings`. The top level is public; the underscored modules are internal. | | `explorer/` | Optional local web UI: `agent-env up`, needs the `explorer` extra, binds loopback `:8234`. | | `cli/` | Click CLI with the groups `a2a-agent`, `artifact`, `bundle`, `config`, `env`, `eval`, `plugin`, `run`, `task`, `up`. | | `examples/` | The bundles agent-env ships (`hello`, and `echo-mcp`, which needs Docker), registered under `agent_env.bundles` in `pyproject.toml`. Tests run them, but they are examples for users, not fixtures. | ## Extension points -Everything out-of-tree is declared in `config.toml` as a `module:Class` pointer and constructed with -`from_config(**config)`: +Out-of-tree code plugs in either through `config.toml`, as a `module:Class` pointer constructed with +`from_config(**config)`, or as an entry point of an installed package: - `[stores.document|object|image|secret]`, `[runner]`, `[sandbox.providers.]`, `[state.providers.]`, `[explorer.plugins]`: one implementation each. @@ -120,6 +125,16 @@ Everything out-of-tree is declared in `config.toml` as a `module:Class` pointer leaves a method its base requires unimplemented fails at registry build with `ConfigError`. - For sandbox and state providers the table name must equal the `type` the provider produces, because that string is persisted and used to reconnect. +- Entry points register the same kinds from an installed package, with the entry-point name as the + registry key: `agent_env.envs`, `agent_env.artifacts`, `agent_env.task_steps`, + `agent_env.sandbox_providers`, `agent_env.state_providers` and `agent_env.explorer_plugins`. A + plugin whose name clashes with a built-in is skipped, a name two plugins claim is left out and + config cannot take it, and config that registers a plugin's name takes it over. + `agent-env plugin check` reports each plugin that did not take effect. +- Bundles install through `agent_env.bundles` entry points (` = ""`, the folder at + `//`), with ids rooted at `@local//`. A shared name is not left out: + a bundle runs by its bare name unless another distribution installs that name too, and then by + `/`, except that agent-env's own bundles keep their bare names. - Environment providers have no table: an installed package registers one as an `agent_env.env_providers` entry point named its `type`, which records carry as `env_provider_type`. - `[plugins.]` is reserved for an installed plugin's own settings, read with @@ -150,8 +165,9 @@ orchestrators call it the same way. What callers rely on: - The runner writes its run identifier to `context.metadata["workflow_id"]` before calling `run()`; core stores it opaquely with the instance. - Per-run overrides travel in `context.metadata["user_overrides"]` (`litellm_api_key`, - `agent_sandbox`, `env_state_type`, `step_params`), never in `os.environ`; steps fall back to the - configured `[model]` key. + `a2a_agent_id`, `agent_sandbox`, `env_sandbox`, `env_state_type`, and more), never in `os.environ`; + steps fall back to the configured `[model]` key. `step_params` maps a step id to overrides of that + step's params, which the step reads through `TaskStep.step_param_overrides()`. - Every persisted or transmitted context snapshot passes through `TaskStepContext.to_safe_dict()` or the context-ops diff, which strip `_REDACTED_KEYS` (`task_step/context.py`); `regraft_redacted_keys` restores live values when a context is rebuilt from a stored document. @@ -177,6 +193,8 @@ orchestrators call it the same way. What callers rely on: - `tst/integration/` needs Docker and the local registry on `:5000` and runs on the local default backends. Minutes-long tests (real image builds, sandbox VMs, gateways, agents) carry `@pytest.mark.int_test_slow` (or module-level `pytestmark`) and run serially. +- `tst/manual/` and `tst/benchmarks/` run only by hand, as each module's docstring says; no CI job + collects them. - Skips are declared capability gaps, decided at collection time: the gates in `tst/util/capabilities.py` produce `skipif` marks with the reason `agentenv-capability-missing: ` (`model_endpoint_configured`, `remote_sandbox`, @@ -192,18 +210,26 @@ orchestrators call it the same way. What callers rely on: ## CI -The required checks on pull requests and `main` are `unit`, `integration-local`, -`integration-local-slow`, `integration-local-gateway`, `installer`, `plugin-api` and `clean-install`. -`.github/workflows/local-backends.yml` runs the first five (`integration-local-gateway` is the slow tier's -`gateway_test.py`, run beside the rest of it), all installed from `uv.lock` (public PyPI only) with no secrets and no external services (a `registry:2` service container for the integration jobs). The -`unit` job also fails if `uv.lock` resolves anything from a registry other than PyPI or if -`agentenv-framework-protocol` is not the editable workspace member, or if `ruff check` reports a -problem. The integration jobs run +A merge into `main` needs an approving review from a code owner (`CODEOWNERS`), every review thread +resolved, and the checks the `main` ruleset requires: `unit`, `integration-local` and +`integration-local-slow`. The other checks below run on every pull request too and are expected to +pass, but a failing one does not block the merge. +`.github/workflows/local-backends.yml` runs `unit`, `ui`, `integration-local`, `integration-local-slow`, +`integration-local-gateway` (the slow tier's `gateway_test.py`, run beside the rest of it) and +`installer`. The Python jobs install from `uv.lock` (public PyPI only), with the uv that +`.github/ci-requirements.txt` pins, and use no secrets and no external services (a `registry:2` +container for the integration jobs). The `unit` job also fails if `uv.lock` resolves anything from a +registry other than PyPI, if `agentenv-framework-protocol` is not the editable workspace member, if +`uv lock --check` fails before or after a release's version bump (`scripts/bump_version.py`), or if +`ruff check` reports a problem. The `ui` job type-checks, lints, smoke-tests and statically exports +the explorer UI. The integration jobs run `.github/scripts/check_skip_policy.py` over the JUnit report: only `agentenv-capability-missing` skips for the allowed capabilities pass, and the fast job allows no skips under `tst/integration/store/`. The `installer` job runs `tst/installer` when a pull request or push -touches the plugin CLI, `agent_env/plugins`, the tier itself or the lockfile, and allows no skips; a -`changes` job decides. `.github/workflows/plugin-api.yml` runs the `plugin-api` job on every pull +touches the plugin CLI, `agent_env/plugins` or `agent_env/config`, the protocol package, the tier +itself, `pyproject.toml`, `uv.lock`, the `Makefile` or the CI that runs the tier, and allows no +skips; a `changes` job decides, and a skipped `installer` counts as passed. +`.github/workflows/plugin-api.yml` runs the `plugin-api` job on every pull request, title edits included, and on `main`: `.github/scripts/check_plugin_api.py` compares the plugin surface (its `BASES` and `USED`) between `HEAD^1` and `HEAD` with griffe, pinned in the `dev` extra, and fails on a break the title does not mark with `!`. `.github/workflows/clean-install.yml` runs the `clean-install` job on @@ -213,7 +239,11 @@ release does, fails if the wheel leaves out a file tracked under `src/agent_env/ allowlisted environment (no AWS, no config, no plugin), and runs `agent-env run hello` twice by name; `.github/scripts/check_clean_install.py`, run by that venv, checks the local store and that the runs left no sandbox work folder. -Actions are pinned to commit SHAs. +`.github/workflows/publish-pypi.yml` runs on the `v*` tag the release automation pushes: it waits +for `main`'s CI to pass on the tagged commit, publishes `agentenv-framework-protocol` and then +`agentenv-framework` to PyPI, and creates the GitHub release, whose notes are the titles of the pull +requests merged since the previous release. +Actions are pinned to commit SHAs; Dependabot updates them, the pinned uv, `uv.lock` and the UI's npm packages. ## Conventions diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 191d21b3..f9ab2dd3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -54,7 +54,8 @@ from sources outside this repository, found through `AGENT_ENV_TEST_MCP_SERVERS_ clock, run `make int-test-slow` with it set, and say so in the pull request. A test may skip only when a capability is genuinely unavailable, and the skip reason must name it (`agentenv-capability-missing: `, see `tst/util/capabilities.py`); the `check_skip_policy` step fails the run otherwise. The `clean-install` job runs the same script as -`make clean-install-test` on every pull request, and is one of the required checks. +`make clean-install-test` on every pull request, and must pass too, as must the `ui` job, which +type-checks, lints and smoke-tests the explorer UI. ## Making changes @@ -74,7 +75,8 @@ must name it (`agentenv-capability-missing: `, see `tst/util/capabilities. - Commit with an email address you are comfortable publishing; the history of this repository is public. GitHub's `noreply` address is a good default. -Every pull request needs an approving review from a code owner (see `CODEOWNERS`) and green CI. +Every pull request needs an approving review from a code owner (see `CODEOWNERS`), every review +thread resolved, and green CI. ## Releases