Skip to content

feat: connect orchestrator and frontend#1564

Open
Scra3 wants to merge 176 commits into
mainfrom
feat/prd-214-server-step-mapper
Open

feat: connect orchestrator and frontend#1564
Scra3 wants to merge 176 commits into
mainfrom
feat/prd-214-server-step-mapper

Conversation

@Scra3
Copy link
Copy Markdown
Member

@Scra3 Scra3 commented Apr 20, 2026

MAIN BRANCH TO INTRODUCE WORKFLOW EXECUTOR.

image

Note

Add @forestadmin/workflow-executor package and connect it to the agent frontend

  • Introduces a new @forestadmin/workflow-executor package: a long-lived polling service that fetches available workflow runs from the Forest server, executes AI-driven steps (read, update, trigger-action, load-related-record, MCP, condition, guidance), and persists results via in-memory or database-backed stores.
  • Adds concrete step executors for each step type with idempotency guards, AI tool selection, activity log integration, and optional user-confirmation flows (awaiting-input).
  • Adds adapters for agent client, Forest server workflow port, AI model (via AiClientAdapter or ServerAiAdapter), activity log port with retry and draining, and structured loggers.
  • Exposes an authenticated Koa HTTP server (ExecutorHttpServer) with /health, GET /runs/:runId, and POST /runs/:runId/trigger endpoints protected by JWT.
  • Adds WorkflowExecutorProxyRoute to the agent, proxying /_internal/workflow-executions/:runId requests to the configured executor URL when workflowExecutorUrl is set in AgentOptions.
  • Extends @forestadmin/ai-proxy with AiClient, createBaseChatModel, getAiConfiguration, validateAiConfigurations, and mcpServerId propagation through MCP and integration tool factories.
  • Risk: the new executor process requires FOREST_ENV_SECRET, FOREST_AUTH_SECRET, and AGENT_URL at startup; missing or malformed values cause an immediate ConfigurationError.

Macroscope summarized 0cac172.

matthv and others added 30 commits March 17, 2026 15:00
…premature deps, add smoke test

- Rewrite CLAUDE.md with project overview and architecture principles, remove changelog
- Remove unused dependencies (ai-proxy, sequelize, zod) per YAGNI
- Add smoke test so CI passes

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… document system architecture

- Lint now covers src and test directories
- Replace require() with import, use stronger assertion (toHaveLength)
- Add System Architecture section describing Front/Orchestrator/Executor/Agent
- Mark Architecture Principles as planned (not yet implemented)
- Remove redundant test/.gitkeep
- Make index.ts a valid module with export {}

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…erver (#1504)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: alban bertolini <albanb@forestadmin.com>
…ain (#1512)

Co-authored-by: alban bertolini <albanb@forestadmin.com>
- Remove McpClient.tools property, loadTools() returns local array
- Rename closeConnections() → dispose()
- Rename testConnections() → checkConnection()
- Add McpServers type export
- Rename mcpServerConfigs → toolConfigs in create-ai-provider
- Update all tests accordingly

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…eability

Add sourceId to McpToolRef so that persisted execution data (pendingData,
executionParams) tracks which MCP server provided the tool. This fixes
tool lookup on re-entry (confirmation flow) when multiple servers expose
tools with the same name.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… collection schema

- Replace non-null assertion with explicit McpToolNotFoundError when AI
  selects a tool name that doesn't match any available tool
- Resolve related collection name from parent schema before looking up
  the related schema in cache, fixing cases where relation name differs
  from target collection name

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Merge main into feature branch, resolve conflicts by taking main's
versions of router.ts, create-ai-provider.ts and their tests.
Remove deleted mcp-config-checker.ts. Bump workflow-executor internal
deps to match main.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Comment thread packages/workflow-executor/src/adapters/agent-client-agent-port.ts Outdated
alban bertolini and others added 16 commits April 30, 2026 16:41
…d and values

Same camelCase deserialization issue as getRecord: apply restoreFieldNames to
related records using both primaryKeyFields and requested fields so that
extractRecordId finds the correct PK values and callers receive snake_case keys.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…lues

Same camelCase deserialization issue as getRecord/getRelatedData: agent-client's
JSON:API deserializer converts response keys to camelCase. Restore original names
using the input values keys, which are already in the caller's original format.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…client to 1.39.5

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…pending data

When the executor picks up a guidance step from the polling loop (before
the user has submitted anything), it now returns awaiting-input instead of
throwing StepStateError. This keeps the step pending so the user trigger
can process it with the submitted data.

Also makes userInput optional so users can submit a guidance step without
providing any text input.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…restoreFieldNames

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… rethrow path

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ect to fix OpenAI rejection

OpenAI requires type: "object" at the root of a tool schema. Using
z.discriminatedUnion directly as schema serialized to anyOf, causing
a 400 "got type: None" error on update-data steps with ≥2 fields.

Wrap in z.object({ input: ... }) (same pattern as the frontend) so
the root is always type: "object". Switch to z.union for the field
variants — discriminatedUnion brought no benefit for LLM selection.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… root schema shape

Add two tests to prevent the OpenAI 400 regression from silently
re-appearing: one asserting the root schema is a ZodObject (not a
union), and one covering the multi-field z.union path with a flat
payload rejection check.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…apper for update-record-field

The integration tests were still passing the old flat args shape
{ fieldName, value, reasoning } to the mock model. After wrapping
the tool schema in z.object({ input: ... }), the executor now
destructures result.input — so the mocks need the wrapper too.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…r) when finding pending step

After a back change, errored steps are stored as done:false + context.error instead of done:true.
The find() must exclude them so the executor picks the next genuinely pending step and not the
already-failed one.

Also includes a temporary throw in ReadRecordStepExecutor for manual front-end error testing.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… first non-done

The orchestrator is the source of truth for which step to execute next.
Always pick the last entry in workflowHistory rather than scanning for
the first non-done/non-cancelled/non-errored step.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
run: ServerHydratedWorkflowRun,
err: WorkflowExecutorError,
): MalformedRunInfo {
const pending = run.workflowHistory.at(-1) ?? null;
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium adapters/forest-server-workflow-port.ts:134

In toMalformedInfo, accessing run.workflowHistory.at(-1) throws TypeError: Cannot read properties of undefined (reading 'at') when workflowHistory is null or undefined. Since this method is only called from error handlers during malformed run recovery, a missing workflowHistory causes the error handling itself to crash. Consider guarding the access with optional chaining: run.workflowHistory?.at(-1).

Suggested change
const pending = run.workflowHistory.at(-1) ?? null;
const pending = run.workflowHistory?.at(-1) ?? null;
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file packages/workflow-executor/src/adapters/forest-server-workflow-port.ts around line 134:

In `toMalformedInfo`, accessing `run.workflowHistory.at(-1)` throws `TypeError: Cannot read properties of undefined (reading 'at')` when `workflowHistory` is `null` or `undefined`. Since this method is only called from error handlers during malformed run recovery, a missing `workflowHistory` causes the error handling itself to crash. Consider guarding the access with optional chaining: `run.workflowHistory?.at(-1)`.

Evidence trail:
packages/workflow-executor/src/adapters/forest-server-workflow-port.ts line 134: `run.workflowHistory.at(-1)` without optional chaining
packages/workflow-executor/src/adapters/server-types.ts line 120: `workflowHistory: ServerStepHistory[]` (non-nullable type, but data from network)
packages/workflow-executor/src/adapters/forest-server-workflow-port.ts lines 63-76: catch block calling `toMalformedInfo` at line 69 when error is `WorkflowExecutorError`
packages/workflow-executor/src/adapters/forest-server-workflow-port.ts lines 107-128: `toDispatch` throws `InvalidStepDefinitionError` at lines 109 and 118 BEFORE `workflowHistory` is accessed
packages/workflow-executor/src/adapters/forest-server-workflow-port.ts lines 55-57: data comes from server HTTP response with no runtime type validation

alban bertolini and others added 2 commits May 18, 2026 12:08
… run)

at(-1) picks the orchestrator's current step but must still return null
when that step is already done — the run is complete and nothing to execute.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Scra3 Scra3 changed the base branch from main to feat/prd-214-setup-workflow-executor-package May 18, 2026 13:00
@Scra3 Scra3 changed the base branch from feat/prd-214-setup-workflow-executor-package to main May 18, 2026 13:03
alban bertolini and others added 6 commits May 19, 2026 18:37
The model sometimes returns snake_case (first_name) when the actual
displayName is camelCase or unseparated (firstname). Add a normalized
fallback that strips separators and lowercases before matching, so a
hallucinated variation doesn't kill an otherwise correct step.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…allback

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ape (#1583)

fix(workflow-executor): fix MCP step path — type shape and id matching

MCP-typed workflow steps failed end-to-end before this change. Two
independent bugs on the path getMcpServerConfigs → loadRemoteTools →
getFilteredTools:

1. Port type mismatch (PRD-357). The orchestrator's
   /liana/mcp-server-configs-with-details endpoint returns
   Record<string, ToolConfig>, but the executor port was typed
   McpConfiguration[] and runner.fetchRemoteTools called .map on it —
   every MCP step crashed with "TypeError: configs.map is not a
   function" before reaching loadRemoteTools. Tests masked the bug
   by mocking mockResolvedValue([]) at 8 call sites, which matched
   the wrong type and short-circuited the buggy branch.

   The port now returns McpServers. The { configs } wrap to
   langchain's McpConfiguration shape lives in runner.fetchRemoteTools,
   one site, after the empty-record short-circuit.

2. Tool id matching (PRD-362). The executor's filter compared
   tool.sourceId (server display name) against step.mcpServerId (DB
   id written by the frontend), so any workflow that specified an
   MCP server failed with NoMcpToolsError regardless of
   configuration.

   The stable DB id from each ToolConfig entry is now threaded
   through ai-proxy (McpClient, ForestIntegrationClient, and the
   integration factories) onto RemoteTool.mcpServerId, and the
   executor matches by id. NoMcpToolsError's technical message now
   includes the requested id and the loaded id list so
   misconfigurations are diagnosable from structured logs; the
   user-facing message stays generic per the dual-message convention.

   The new tool-side field is named mcpServerId (not id) to read
   honestly at the consumer site — tool.mcpServerId === step.mcpServerId
   expresses the FK relationship plainly. McpServerConfig.id and
   ForestIntegrationConfig.id keep id since those mirror the wire
   shape (which itself mirrors the ai_mcp_configs PK column).

Also drops the now-unused McpConfiguration re-export from the port
and barrel (used only internally by AiModelPort/adapters, imported
directly from @forestadmin/ai-proxy), drops a wire-shape comment
that the type and route URL already document, and renames a test
config key from "data-gouv" to "mcp-server-1" to keep the test free
of real-world references.

fixes PRD-357
fixes PRD-362
Comment on lines +209 to +217
if (pendingData !== undefined && execution) {
const schema = patchBodySchemas[execution.type]!;
const parsed = schema.safeParse(pendingData);

if (!parsed.success) {
throw new StepStateError(
`Invalid pending data: ${parsed.error.issues.map(i => i.message).join(', ')}`,
);
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium executors/base-step-executor.ts:209

The non-null assertion patchBodySchemas[execution.type]! at line 210 crashes when execution.type is 'read-record' or 'record' — these types have no schema in patchBodySchemas, so schema.safeParse throws TypeError: Cannot read properties of undefined. Consider adding schemas for these types or throwing a descriptive StepStateError when no schema exists.

      const schema = patchBodySchemas[execution.type];
-
-      if (!parsed.success) {
+      if (!schema) {
+        throw new StepStateError(`No pending data validator for step type "${execution.type}"`);
+      }
+
+      const parsed = schema.safeParse(pendingData);
+
+      if (!parsed.success) {
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file packages/workflow-executor/src/executors/base-step-executor.ts around lines 209-217:

The non-null assertion `patchBodySchemas[execution.type]!` at line 210 crashes when `execution.type` is `'read-record'` or `'record'` — these types have no schema in `patchBodySchemas`, so `schema.safeParse` throws `TypeError: Cannot read properties of undefined`. Consider adding schemas for these types or throwing a descriptive `StepStateError` when no schema exists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants