diff --git a/workspaces/ai-integrations/openspec/changes/airesource-agent-typed-schema/design.md b/workspaces/ai-integrations/openspec/changes/airesource-agent-typed-schema/design.md index aa5ae843e37..88a45ff1f9b 100644 --- a/workspaces/ai-integrations/openspec/changes/airesource-agent-typed-schema/design.md +++ b/workspaces/ai-integrations/openspec/changes/airesource-agent-typed-schema/design.md @@ -94,9 +94,9 @@ The “Research reference” column records where the catalog field was derived ### D5 — Schema + processor in this OpenSpec; agent-only processor rules -**Choice**: Deliver types, examples, schema tests, **and** catalog processor validation for agent-specific fields. Processor does **not** re-validate core entity fields (`owner`, `lifecycle`, etc.) beyond existing catalog behavior. +**Choice**: Deliver types, examples, schema tests, **and** catalog processor validation for agent-specific fields. The agent processor (`AiResourceAgentProcessor`) lives in `catalog-backend-module-ai-resource-agent`, not in `AIResourceExtensionsProcessor`. The extensions processor remains scope/OCI only. Processor does **not** re-validate core entity fields (`owner`, `lifecycle`, etc.) beyond existing catalog behavior. -**Rationale**: Epic cohesion (15867 + 15868 share one field set). Keep processor focused on agent fields. +**Rationale**: Epic cohesion (15867 + 15868 share one field set). Keep processor focused on agent fields. Agent validation belongs with the agent packages, matching the packaging approach from #4128. ### D6 — Dual-track documentation (rhdh-plugins + upstream) @@ -132,14 +132,14 @@ The “Research reference” column records where the catalog field was derived - Per-type validators / guards: `skillAiResourceEntityV1alpha1Validator`, `ruleAiResourceEntityV1alpha1Validator`, `isSkillAiResourceEntity`, `isRuleAiResourceEntity` - Kind registration via `aiResourceEntityModel` / `catalogModuleAiResourceEntityModel` -Agent should follow that pattern: add an `AgentAiResourceEntity…` (name TBD) member of the AiResource union (or an RHDH-local extension layer that mirrors it until upstream accepts agent), with a `KindValidator` + type guard keyed on `spec.type: 'agent'`. +Agent should follow that pattern: add an `AgentAiResourceEntity…` (name TBD) member of the AiResource union (or an RHDH-local extension layer that mirrors it until upstream accepts agent), with a `KindValidator` + type guard keyed on `spec.type: ‘agent’`. **Secondary reference — MCP server API discriminated extension** (same catalog-model alpha surface): -- `McpServerApiEntity` with `spec.type: 'mcp-server'`, `mcpServerApiEntityValidator`, `isMcpServerApiEntity`, `mcpServerApiEntityModel` +- `McpServerApiEntity` with `spec.type: ‘mcp-server’`, `mcpServerApiEntityValidator`, `isMcpServerApiEntity`, `mcpServerApiEntityModel` - Shows how Backstage extends an existing kind with a typed `spec.type` branch (useful precedent for dual-track / upstream PR work) -**Local RHDH extension precedent**: this workspace’s `AIResourceExtensionsProcessor` for `spec.scope` / OCI checks—agent **field** validation (RHIDP-15868) should extend that processor path for agent-specific rules, while the **typed schema** itself follows the catalog-model validator pattern above. +**Agent packaging**: Agent schema, `KindValidator`, type guard, and `CatalogModelLayer` live in `catalog-model-ai-resource-agent`. Agent-specific field validation (RHIDP-15868) lives in `catalog-backend-module-ai-resource-agent` as `AiResourceAgentProcessor`, registered alongside the model source in the same backend module. The `AIResourceExtensionsProcessor` in `catalog-backend-module-ai-resource-extensions` remains scope/OCI only and does not contain agent-specific rules. Kind spelling is `AiResource` (matching #4128 / the agent model packages). **Rationale**: Without these pointers, implementers (human or coding agent) will invent ad-hoc types that diverge from skill/rule and force manual rework. Gabe’s review feedback on this PR. diff --git a/workspaces/ai-integrations/openspec/changes/airesource-agent-typed-schema/proposal.md b/workspaces/ai-integrations/openspec/changes/airesource-agent-typed-schema/proposal.md index 5c5e4de758f..b2ec76ca837 100644 --- a/workspaces/ai-integrations/openspec/changes/airesource-agent-typed-schema/proposal.md +++ b/workspaces/ai-integrations/openspec/changes/airesource-agent-typed-schema/proposal.md @@ -9,7 +9,7 @@ Platform engineers need a first-class way to register AI agents in the Software - Add a typed agent schema for `kind: AiResource` with `spec.type: agent` (TypeScript types / schema-style validation; no OpenAI Agents SDK package dependency). - Encode the decided field mapping (RHIDP-15866) in OpenSpec design; only agent-specific required field is non-empty `spec.instructions`. - Add example `catalog-info.yaml` and/or test fixtures covering a representative agent (required + optional fields). -- Extend catalog processor validation for agent entities (RHIDP-15868): reject missing/invalid agent-specific fields with actionable errors. +- Add agent-specific catalog processor validation in the agent backend module (`catalog-backend-module-ai-resource-agent`) via `AiResourceAgentProcessor` (RHIDP-15868): reject missing/invalid agent-specific fields with actionable errors. `AIResourceExtensionsProcessor` remains scope/OCI only. - Update in-repo OpenSpec/design docs for `AiResource` + agent ownership under RHDHPLAN-1507; remove “pending 1113” language for the agent type where it appears. - Add unit/schema and processor tests for accept/reject paths. - Align naming with upstream skill/rule style: singular `spec.type: agent` (not `agents`); correct sibling discovery examples accordingly. @@ -49,7 +49,7 @@ _(none promoted under `openspec/specs/` yet. Sibling change-local discovery exam ## Impact - **Schema / types**: New or extended TypeScript for agent-shaped `AiResource` in this workspace. -- **Catalog processor**: Agent-specific validation in the AiResource extensions processor path. +- **Catalog processor**: Agent-specific validation in the agent backend module (`AiResourceAgentProcessor`), not the extensions processor. - **Examples / fixtures**: New catalog YAML under `examples/` (and/or test fixtures). - **Tests**: Schema/unit and processor accept/reject coverage. - **Docs / OpenSpec**: Dual-track (rhdh-plugins + upstream) and singular type discriminator docs. diff --git a/workspaces/ai-integrations/openspec/changes/airesource-agent-typed-schema/tasks.md b/workspaces/ai-integrations/openspec/changes/airesource-agent-typed-schema/tasks.md index 2d41bdc9fbc..7de430bbf52 100644 --- a/workspaces/ai-integrations/openspec/changes/airesource-agent-typed-schema/tasks.md +++ b/workspaces/ai-integrations/openspec/changes/airesource-agent-typed-schema/tasks.md @@ -26,10 +26,11 @@ ## 5. Catalog processor validation (RHIDP-15868) -- [ ] 5.1 Extend the AiResource extensions processor to validate agent-specific fields when `spec.type: agent` +- [ ] 5.1 Add `AiResourceAgentProcessor` in `catalog-backend-module-ai-resource-agent` to validate agent-specific fields when `spec.type: agent`; do **not** extend `AIResourceExtensionsProcessor` (which remains scope/OCI only) - [ ] 5.2 Reject missing/empty `spec.instructions` and wrong optional agent field shapes with actionable errors - [ ] 5.3 Do not enforce entity-ref format on `handoffs` / `tools`; do not add new owner/lifecycle processor rules - [ ] 5.4 Add processor tests for accept and reject paths; ensure non-agent AiResources are unaffected +- [ ] 5.5 Register `AiResourceAgentProcessor` in the agent backend module alongside the model source ## 6. Docs and OpenSpec DoD diff --git a/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/package.json b/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/package.json index 3d32a4c9250..79502c93c9c 100644 --- a/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/package.json +++ b/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/package.json @@ -32,10 +32,12 @@ "dependencies": { "@backstage/backend-plugin-api": "^1.9.2", "@backstage/catalog-model": "^1.9.0", + "@backstage/plugin-catalog-common": "^1.1.10", "@backstage/plugin-catalog-node": "^2.2.2", "@red-hat-developer-hub/backstage-plugin-catalog-model-ai-resource-agent": "workspace:^" }, "devDependencies": { + "@backstage/backend-test-utils": "^1.11.4", "@backstage/cli": "^0.36.3" }, "files": [ diff --git a/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/report.api.md b/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/report.api.md index e3b7b93b814..f3f8942a23a 100644 --- a/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/report.api.md +++ b/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/report.api.md @@ -4,6 +4,22 @@ ```ts import { BackendFeature } from '@backstage/backend-plugin-api'; +import { CatalogProcessor } from '@backstage/plugin-catalog-node'; +import { CatalogProcessorEmit } from '@backstage/plugin-catalog-node'; +import { Entity } from '@backstage/catalog-model'; +import { LocationSpec } from '@backstage/plugin-catalog-common'; + +// @public +export class AiResourceAgentProcessor implements CatalogProcessor { + // (undocumented) + getProcessorName(): string; + // (undocumented) + preProcessEntity( + entity: Entity, + _location: LocationSpec, + _emit: CatalogProcessorEmit, + ): Promise; +} // @public const catalogModuleAiResourceAgent: BackendFeature; diff --git a/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/src/AiResourceAgentProcessor.test.ts b/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/src/AiResourceAgentProcessor.test.ts new file mode 100644 index 00000000000..20d5f07b939 --- /dev/null +++ b/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/src/AiResourceAgentProcessor.test.ts @@ -0,0 +1,236 @@ +/* + * Copyright Red Hat, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { Entity } from '@backstage/catalog-model'; +import { AiResourceAgentProcessor } from './AiResourceAgentProcessor'; + +function makeAiResource( + spec: Entity['spec'] = {}, + annotations?: Record, +): Entity { + return { + apiVersion: 'backstage.io/v1alpha1', + kind: 'AiResource', + metadata: { + name: 'test-resource', + ...(annotations ? { annotations } : {}), + }, + spec, + }; +} + +describe('AiResourceAgentProcessor', () => { + let processor: AiResourceAgentProcessor; + const location = { type: 'url', target: 'https://example.com' }; + const emit = jest.fn(); + + beforeEach(() => { + processor = new AiResourceAgentProcessor(); + emit.mockClear(); + }); + + it('should return processor name', () => { + expect(processor.getProcessorName()).toBe('AiResourceAgentProcessor'); + }); + + describe('agent validation (spec.type: agent)', () => { + it('should accept a valid agent entity with required fields', async () => { + const entity = makeAiResource({ + type: 'agent', + lifecycle: 'production', + owner: 'ai-platform-team', + instructions: 'You are a test agent.', + }); + + const result = await processor.preProcessEntity(entity, location, emit); + + expect(result).toEqual(entity); + }); + + it('should accept a valid agent with all optional fields', async () => { + const entity = makeAiResource({ + type: 'agent', + lifecycle: 'production', + owner: 'ai-platform-team', + instructions: 'You are a fully configured agent.', + handoffDescription: 'Handles everything.', + model: 'gpt-4o', + handoffs: ['agent-a', 'agent-b'], + tools: ['tool-x', 'tool-y'], + toolUseBehavior: 'run_llm_again', + resetToolChoice: true, + modelSettings: { temperature: 0.5 }, + outputSchema: { type: 'object' }, + }); + + const result = await processor.preProcessEntity(entity, location, emit); + + expect(result).toEqual(entity); + }); + + it('should accept opaque handoffs and tools strings', async () => { + const entity = makeAiResource({ + type: 'agent', + instructions: 'Agent with opaque refs.', + handoffs: ['some-arbitrary-string', 'another-ref'], + tools: ['my-custom-tool'], + }); + + const result = await processor.preProcessEntity(entity, location, emit); + + expect(result).toEqual(entity); + }); + + it('should reject agent with missing instructions', async () => { + const entity = makeAiResource({ + type: 'agent', + lifecycle: 'production', + owner: 'team', + }); + + await expect( + processor.preProcessEntity(entity, location, emit), + ).rejects.toThrow('spec.instructions'); + }); + + it('should reject agent with empty instructions', async () => { + const entity = makeAiResource({ + type: 'agent', + instructions: '', + }); + + await expect( + processor.preProcessEntity(entity, location, emit), + ).rejects.toThrow('spec.instructions'); + }); + + it('should reject agent with wrong-type instructions', async () => { + const entity = makeAiResource({ + type: 'agent', + instructions: 42, + }); + + await expect( + processor.preProcessEntity(entity, location, emit), + ).rejects.toThrow('spec.instructions'); + }); + + it('should reject agent with handoffs as non-array', async () => { + const entity = makeAiResource({ + type: 'agent', + instructions: 'Valid instructions.', + handoffs: 'not-an-array', + }); + + await expect( + processor.preProcessEntity(entity, location, emit), + ).rejects.toThrow('spec.handoffs'); + }); + + it('should reject agent with resetToolChoice as non-boolean', async () => { + const entity = makeAiResource({ + type: 'agent', + instructions: 'Valid instructions.', + resetToolChoice: 'yes', + }); + + await expect( + processor.preProcessEntity(entity, location, emit), + ).rejects.toThrow('spec.resetToolChoice'); + }); + + it('should not apply agent instructions rule to skill entities', async () => { + const entity = makeAiResource({ + type: 'skill', + lifecycle: 'production', + owner: 'team', + }); + + const result = await processor.preProcessEntity(entity, location, emit); + + expect(result).toEqual(entity); + }); + + it('should not apply agent instructions rule to entities without spec.type', async () => { + const entity = makeAiResource({ + lifecycle: 'production', + owner: 'team', + }); + + const result = await processor.preProcessEntity(entity, location, emit); + + expect(result).toEqual(entity); + }); + + it('should report multiple agent errors together', async () => { + const entity = makeAiResource({ + type: 'agent', + handoffs: 'not-an-array', + }); + + const error = await processor + .preProcessEntity(entity, location, emit) + .catch((e: Error) => e); + + expect(error).toBeInstanceOf(Error); + const message = (error as Error).message; + expect(message).toContain('spec.instructions'); + expect(message).toContain('spec.handoffs'); + }); + + it('should not expose internal class names in agent errors', async () => { + const entity = makeAiResource({ + type: 'agent', + }); + + const error = await processor + .preProcessEntity(entity, location, emit) + .catch((e: Error) => e); + + expect(error).toBeInstanceOf(Error); + expect((error as Error).message).not.toMatch(/AiResourceAgentProcessor/); + expect((error as Error).message).not.toMatch(/at\s+\w+\.\w+\s+\(/); + }); + }); + + describe('non-AiResource entities', () => { + it('should pass through Component entities unchanged', async () => { + const entity: Entity = { + apiVersion: 'backstage.io/v1alpha1', + kind: 'Component', + metadata: { name: 'my-component' }, + spec: { type: 'service', lifecycle: 'production', owner: 'team-a' }, + }; + + const result = await processor.preProcessEntity(entity, location, emit); + + expect(result).toEqual(entity); + }); + + it('should not validate agent fields on non-AiResource kinds', async () => { + const entity: Entity = { + apiVersion: 'backstage.io/v1alpha1', + kind: 'Resource', + metadata: { name: 'my-resource' }, + spec: { type: 'agent', owner: 'team-a' }, + }; + + const result = await processor.preProcessEntity(entity, location, emit); + + expect(result).toEqual(entity); + }); + }); +}); diff --git a/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/src/AiResourceAgentProcessor.ts b/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/src/AiResourceAgentProcessor.ts new file mode 100644 index 00000000000..4f6786f3e4a --- /dev/null +++ b/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/src/AiResourceAgentProcessor.ts @@ -0,0 +1,68 @@ +/* + * Copyright Red Hat, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { + CatalogProcessor, + CatalogProcessorEmit, +} from '@backstage/plugin-catalog-node'; +import { Entity } from '@backstage/catalog-model'; +import { LocationSpec } from '@backstage/plugin-catalog-common'; +import { collectAgentErrors } from './collectAgentErrors'; + +/** + * A CatalogProcessor that validates agent-specific fields on + * AiResource entities with `spec.type: 'agent'`. + * + * Validates: + * - `spec.instructions`: required, non-empty string + * - `spec.handoffs` / `spec.tools`: must be arrays if present + * - `spec.resetToolChoice`: must be boolean if present + * - `spec.modelSettings`: must be plain object if present + * - `spec.toolUseBehavior`: must be string or string array if present + * - `spec.outputSchema`: must be string or object if present + * - `spec.handoffDescription` / `spec.model`: must be strings if present + * + * Non-agent AiResource entities (skill, rule, model) are unaffected. + * All constraint violations are collected and reported in a single + * error rather than stopping at the first failure. + * + * @public + */ +export class AiResourceAgentProcessor implements CatalogProcessor { + getProcessorName(): string { + return 'AiResourceAgentProcessor'; + } + + async preProcessEntity( + entity: Entity, + _location: LocationSpec, + _emit: CatalogProcessorEmit, + ): Promise { + if (entity.kind !== 'AiResource') { + return entity; + } + + const errors = collectAgentErrors(entity); + + if (errors.length > 0) { + throw new Error( + `Validation failed for AiResource agent entity: ${errors.join('; ')}`, + ); + } + + return entity; + } +} diff --git a/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/src/collectAgentErrors.test.ts b/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/src/collectAgentErrors.test.ts new file mode 100644 index 00000000000..bc8699d3642 --- /dev/null +++ b/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/src/collectAgentErrors.test.ts @@ -0,0 +1,423 @@ +/* + * Copyright Red Hat, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { Entity } from '@backstage/catalog-model'; +import { collectAgentErrors } from './collectAgentErrors'; + +function makeAgent(spec: Entity['spec'] = {}): Entity { + return { + apiVersion: 'backstage.io/v1alpha1', + kind: 'AiResource', + metadata: { name: 'test-agent' }, + spec: { + type: 'agent', + lifecycle: 'production', + owner: 'ai-platform-team', + instructions: 'You are a test agent.', + ...spec, + }, + }; +} + +describe('collectAgentErrors', () => { + describe('non-agent entities are skipped', () => { + it('returns no errors for spec.type: skill', () => { + const entity = makeAgent({ type: 'skill' }); + delete (entity as any).spec.instructions; + + expect(collectAgentErrors(entity)).toEqual([]); + }); + + it('returns no errors for spec.type: rule', () => { + const entity: Entity = { + apiVersion: 'backstage.io/v1alpha1', + kind: 'AiResource', + metadata: { name: 'test-rule' }, + spec: { type: 'rule', lifecycle: 'production', owner: 'team' }, + }; + + expect(collectAgentErrors(entity)).toEqual([]); + }); + + it('returns no errors for spec.type: model', () => { + const entity: Entity = { + apiVersion: 'backstage.io/v1alpha1', + kind: 'AiResource', + metadata: { name: 'test-model' }, + spec: { type: 'model', lifecycle: 'production', owner: 'team' }, + }; + + expect(collectAgentErrors(entity)).toEqual([]); + }); + + it('returns no errors when spec is undefined', () => { + const entity: Entity = { + apiVersion: 'backstage.io/v1alpha1', + kind: 'AiResource', + metadata: { name: 'test' }, + }; + + expect(collectAgentErrors(entity)).toEqual([]); + }); + }); + + describe('spec.instructions (required)', () => { + it('accepts valid non-empty instructions', () => { + const entity = makeAgent({ instructions: 'You are a helpful agent.' }); + + expect(collectAgentErrors(entity)).toEqual([]); + }); + + it('rejects missing instructions', () => { + const entity = makeAgent(); + delete (entity as any).spec.instructions; + + const errors = collectAgentErrors(entity); + expect(errors).toHaveLength(1); + expect(errors[0]).toContain('spec.instructions'); + expect(errors[0]).toContain('required'); + }); + + it('rejects empty string instructions', () => { + const entity = makeAgent({ instructions: '' }); + + const errors = collectAgentErrors(entity); + expect(errors).toHaveLength(1); + expect(errors[0]).toContain('spec.instructions'); + expect(errors[0]).toContain('empty'); + }); + + it('rejects null instructions', () => { + const entity = makeAgent({ instructions: null }); + + const errors = collectAgentErrors(entity); + expect(errors).toHaveLength(1); + expect(errors[0]).toContain('spec.instructions'); + expect(errors[0]).toContain('required'); + }); + + it('rejects numeric instructions', () => { + const entity = makeAgent({ instructions: 42 }); + + const errors = collectAgentErrors(entity); + expect(errors).toHaveLength(1); + expect(errors[0]).toContain('spec.instructions'); + expect(errors[0]).toContain('string'); + }); + + it('rejects array instructions', () => { + const entity = makeAgent({ instructions: ['step 1', 'step 2'] }); + + const errors = collectAgentErrors(entity); + expect(errors).toHaveLength(1); + expect(errors[0]).toContain('spec.instructions'); + }); + }); + + describe('spec.handoffs (optional, must be array)', () => { + it('accepts undefined handoffs', () => { + const entity = makeAgent(); + expect(collectAgentErrors(entity)).toEqual([]); + }); + + it('accepts handoffs as string array', () => { + const entity = makeAgent({ + handoffs: ['agent-a', 'agent-b'], + }); + + expect(collectAgentErrors(entity)).toEqual([]); + }); + + it('accepts opaque handoff strings (no entity-ref format)', () => { + const entity = makeAgent({ + handoffs: ['some-arbitrary-string', 'another-ref'], + }); + + expect(collectAgentErrors(entity)).toEqual([]); + }); + + it('rejects handoffs as a string', () => { + const entity = makeAgent({ handoffs: 'not-an-array' }); + + const errors = collectAgentErrors(entity); + expect(errors).toHaveLength(1); + expect(errors[0]).toContain('spec.handoffs'); + expect(errors[0]).toContain('array'); + }); + + it('rejects handoffs as a number', () => { + const entity = makeAgent({ handoffs: 123 }); + + const errors = collectAgentErrors(entity); + expect(errors).toHaveLength(1); + expect(errors[0]).toContain('spec.handoffs'); + }); + + it('rejects handoffs as an object', () => { + const entity = makeAgent({ handoffs: { a: 'b' } }); + + const errors = collectAgentErrors(entity); + expect(errors).toHaveLength(1); + expect(errors[0]).toContain('spec.handoffs'); + }); + }); + + describe('spec.tools (optional, must be array)', () => { + it('accepts tools as string array', () => { + const entity = makeAgent({ + tools: ['tool-x', 'tool-y'], + }); + + expect(collectAgentErrors(entity)).toEqual([]); + }); + + it('accepts opaque tool strings (no entity-ref format)', () => { + const entity = makeAgent({ + tools: ['my-custom-tool'], + }); + + expect(collectAgentErrors(entity)).toEqual([]); + }); + + it('rejects tools as a string', () => { + const entity = makeAgent({ tools: 'not-an-array' }); + + const errors = collectAgentErrors(entity); + expect(errors).toHaveLength(1); + expect(errors[0]).toContain('spec.tools'); + expect(errors[0]).toContain('array'); + }); + }); + + describe('spec.resetToolChoice (optional, must be boolean)', () => { + it('accepts resetToolChoice as true', () => { + const entity = makeAgent({ resetToolChoice: true }); + expect(collectAgentErrors(entity)).toEqual([]); + }); + + it('accepts resetToolChoice as false', () => { + const entity = makeAgent({ resetToolChoice: false }); + expect(collectAgentErrors(entity)).toEqual([]); + }); + + it('rejects resetToolChoice as a string', () => { + const entity = makeAgent({ resetToolChoice: 'yes' }); + + const errors = collectAgentErrors(entity); + expect(errors).toHaveLength(1); + expect(errors[0]).toContain('spec.resetToolChoice'); + expect(errors[0]).toContain('boolean'); + }); + + it('rejects resetToolChoice as a number', () => { + const entity = makeAgent({ resetToolChoice: 1 }); + + const errors = collectAgentErrors(entity); + expect(errors).toHaveLength(1); + expect(errors[0]).toContain('spec.resetToolChoice'); + }); + }); + + describe('spec.modelSettings (optional, must be object)', () => { + it('accepts modelSettings as a plain object', () => { + const entity = makeAgent({ + modelSettings: { temperature: 0.5, maxTokens: 2048 }, + }); + + expect(collectAgentErrors(entity)).toEqual([]); + }); + + it('rejects modelSettings as an array', () => { + const entity = makeAgent({ modelSettings: [1, 2, 3] }); + + const errors = collectAgentErrors(entity); + expect(errors).toHaveLength(1); + expect(errors[0]).toContain('spec.modelSettings'); + expect(errors[0]).toContain('object'); + }); + + it('rejects modelSettings as a string', () => { + const entity = makeAgent({ modelSettings: 'high' }); + + const errors = collectAgentErrors(entity); + expect(errors).toHaveLength(1); + expect(errors[0]).toContain('spec.modelSettings'); + }); + + it('rejects modelSettings as null', () => { + const entity = makeAgent({ modelSettings: null }); + + const errors = collectAgentErrors(entity); + expect(errors).toHaveLength(1); + expect(errors[0]).toContain('spec.modelSettings'); + }); + }); + + describe('spec.toolUseBehavior (optional, string or string[])', () => { + it('accepts toolUseBehavior as a string', () => { + const entity = makeAgent({ toolUseBehavior: 'run_llm_again' }); + expect(collectAgentErrors(entity)).toEqual([]); + }); + + it('accepts toolUseBehavior as a string array', () => { + const entity = makeAgent({ + toolUseBehavior: ['tool-a', 'tool-b'], + }); + expect(collectAgentErrors(entity)).toEqual([]); + }); + + it('rejects toolUseBehavior as a number', () => { + const entity = makeAgent({ toolUseBehavior: 42 }); + + const errors = collectAgentErrors(entity); + expect(errors).toHaveLength(1); + expect(errors[0]).toContain('spec.toolUseBehavior'); + }); + + it('rejects toolUseBehavior as a boolean', () => { + const entity = makeAgent({ toolUseBehavior: true }); + + const errors = collectAgentErrors(entity); + expect(errors).toHaveLength(1); + expect(errors[0]).toContain('spec.toolUseBehavior'); + }); + }); + + describe('spec.outputSchema (optional, string or object)', () => { + it('accepts outputSchema as a string', () => { + const entity = makeAgent({ outputSchema: 'text' }); + expect(collectAgentErrors(entity)).toEqual([]); + }); + + it('accepts outputSchema as an object', () => { + const entity = makeAgent({ + outputSchema: { type: 'object', properties: {} }, + }); + expect(collectAgentErrors(entity)).toEqual([]); + }); + + it('rejects outputSchema as a number', () => { + const entity = makeAgent({ outputSchema: 123 }); + + const errors = collectAgentErrors(entity); + expect(errors).toHaveLength(1); + expect(errors[0]).toContain('spec.outputSchema'); + }); + + it('rejects outputSchema as an array', () => { + const entity = makeAgent({ outputSchema: ['a', 'b'] }); + + const errors = collectAgentErrors(entity); + expect(errors).toHaveLength(1); + expect(errors[0]).toContain('spec.outputSchema'); + }); + }); + + describe('spec.handoffDescription (optional, must be string)', () => { + it('accepts handoffDescription as a string', () => { + const entity = makeAgent({ + handoffDescription: 'Handles routing.', + }); + expect(collectAgentErrors(entity)).toEqual([]); + }); + + it('rejects handoffDescription as a number', () => { + const entity = makeAgent({ handoffDescription: 42 }); + + const errors = collectAgentErrors(entity); + expect(errors).toHaveLength(1); + expect(errors[0]).toContain('spec.handoffDescription'); + expect(errors[0]).toContain('string'); + }); + }); + + describe('spec.model (optional, must be string)', () => { + it('accepts model as a string', () => { + const entity = makeAgent({ model: 'gpt-4o' }); + expect(collectAgentErrors(entity)).toEqual([]); + }); + + it('rejects model as a number', () => { + const entity = makeAgent({ model: 42 }); + + const errors = collectAgentErrors(entity); + expect(errors).toHaveLength(1); + expect(errors[0]).toContain('spec.model'); + expect(errors[0]).toContain('string'); + }); + }); + + describe('multiple errors reported together', () => { + it('collects all agent field errors in a single array', () => { + const entity = makeAgent({ + instructions: 42, + handoffs: 'not-an-array', + resetToolChoice: 'yes', + }); + + const errors = collectAgentErrors(entity); + expect(errors.length).toBeGreaterThanOrEqual(3); + expect(errors.some(e => e.includes('spec.instructions'))).toBe(true); + expect(errors.some(e => e.includes('spec.handoffs'))).toBe(true); + expect(errors.some(e => e.includes('spec.resetToolChoice'))).toBe(true); + }); + }); + + describe('error quality', () => { + it('does not expose internal class names', () => { + const entity = makeAgent(); + delete (entity as any).spec.instructions; + + const errors = collectAgentErrors(entity); + for (const err of errors) { + expect(err).not.toMatch(/Processor/); + expect(err).not.toMatch(/at\s+\w+\.\w+\s+\(/); + } + }); + + it('names the field path in the error', () => { + const entity = makeAgent({ instructions: '' }); + + const errors = collectAgentErrors(entity); + expect(errors[0]).toContain('spec.instructions'); + }); + }); + + describe('valid agent with all optional fields', () => { + it('returns no errors for a fully populated agent', () => { + const entity = makeAgent({ + instructions: 'You are a fully configured agent.', + handoffDescription: 'Handles everything.', + model: 'gpt-4o', + handoffs: ['agent-a', 'agent-b'], + tools: ['tool-x', 'tool-y'], + toolUseBehavior: 'run_llm_again', + resetToolChoice: true, + modelSettings: { + temperature: 0.5, + maxTokens: 2048, + toolChoice: 'auto', + }, + outputSchema: { + type: 'object', + properties: { result: { type: 'string' } }, + }, + }); + + expect(collectAgentErrors(entity)).toEqual([]); + }); + }); +}); diff --git a/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/src/collectAgentErrors.ts b/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/src/collectAgentErrors.ts new file mode 100644 index 00000000000..843f23bc580 --- /dev/null +++ b/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/src/collectAgentErrors.ts @@ -0,0 +1,150 @@ +/* + * Copyright Red Hat, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { Entity } from '@backstage/catalog-model'; + +/** + * Collects agent-specific validation errors for an AiResource entity + * with `spec.type: 'agent'` without throwing. Returns an array of + * error messages (empty if valid or if `spec.type` is not `'agent'`). + * + * Validates agent-specific fields only; does NOT re-validate core + * entity fields such as `spec.owner` or `spec.lifecycle`. + * + * @internal + */ +export function collectAgentErrors(entity: Entity): string[] { + const spec = entity.spec; + if (spec?.type !== 'agent') { + return []; + } + + const errors: string[] = []; + + // Required: spec.instructions — non-empty string + const instructions = spec.instructions; + if (instructions === undefined || instructions === null) { + errors.push('spec.instructions is required for agent AiResource entities'); + } else if (typeof instructions !== 'string') { + errors.push( + `spec.instructions must be a string; got ${typeLabel(instructions)}`, + ); + } else if (instructions === '') { + errors.push('spec.instructions must not be empty'); + } + + // Optional: spec.handoffs — must be an array if present + if (spec.handoffs !== undefined && !Array.isArray(spec.handoffs)) { + errors.push( + `spec.handoffs must be an array; got ${typeLabel(spec.handoffs)}`, + ); + } + + // Optional: spec.tools — must be an array if present + if (spec.tools !== undefined && !Array.isArray(spec.tools)) { + errors.push(`spec.tools must be an array; got ${typeLabel(spec.tools)}`); + } + + // Optional: spec.resetToolChoice — must be a boolean if present + if ( + spec.resetToolChoice !== undefined && + typeof spec.resetToolChoice !== 'boolean' + ) { + errors.push( + `spec.resetToolChoice must be a boolean; got ${typeLabel( + spec.resetToolChoice, + )}`, + ); + } + + // Optional: spec.modelSettings — must be a plain object if present + if (spec.modelSettings !== undefined) { + if ( + typeof spec.modelSettings !== 'object' || + spec.modelSettings === null || + Array.isArray(spec.modelSettings) + ) { + errors.push( + `spec.modelSettings must be an object; got ${typeLabel( + spec.modelSettings, + )}`, + ); + } + } + + // Optional: spec.toolUseBehavior — must be a string or string array + if (spec.toolUseBehavior !== undefined) { + if ( + typeof spec.toolUseBehavior !== 'string' && + !Array.isArray(spec.toolUseBehavior) + ) { + errors.push( + `spec.toolUseBehavior must be a string or an array; got ${typeLabel( + spec.toolUseBehavior, + )}`, + ); + } + } + + // Optional: spec.outputSchema — must be a string or plain object + if (spec.outputSchema !== undefined) { + const isString = typeof spec.outputSchema === 'string'; + const isObject = + typeof spec.outputSchema === 'object' && + spec.outputSchema !== null && + !Array.isArray(spec.outputSchema); + if (!isString && !isObject) { + errors.push( + `spec.outputSchema must be a string or an object; got ${typeLabel( + spec.outputSchema, + )}`, + ); + } + } + + // Optional: spec.handoffDescription — must be a string if present + if ( + spec.handoffDescription !== undefined && + typeof spec.handoffDescription !== 'string' + ) { + errors.push( + `spec.handoffDescription must be a string; got ${typeLabel( + spec.handoffDescription, + )}`, + ); + } + + // Optional: spec.model — must be a string if present + if (spec.model !== undefined && typeof spec.model !== 'string') { + errors.push(`spec.model must be a string; got ${typeLabel(spec.model)}`); + } + + return errors; +} + +/** + * Returns a human-friendly type label for error messages. + * Distinguishes arrays and null from plain "object". + */ +function typeLabel(value: unknown): string { + if (value === null) { + return 'null'; + } + if (Array.isArray(value)) { + return 'array'; + } + return typeof value; +} diff --git a/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/src/index.ts b/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/src/index.ts index 611d02c6ab5..4350efcc89e 100644 --- a/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/src/index.ts +++ b/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/src/index.ts @@ -20,3 +20,4 @@ * @packageDocumentation */ export { catalogModuleAiResourceAgent as default } from './module'; +export { AiResourceAgentProcessor } from './AiResourceAgentProcessor'; diff --git a/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/src/module.ts b/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/src/module.ts index f0cb516bf95..cec8da0afac 100644 --- a/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/src/module.ts +++ b/workspaces/ai-integrations/plugins/catalog-backend-module-ai-resource-agent/src/module.ts @@ -16,8 +16,10 @@ import { createBackendModule } from '@backstage/backend-plugin-api'; import { CatalogModelSources } from '@backstage/catalog-model/alpha'; +import { catalogProcessingExtensionPoint } from '@backstage/plugin-catalog-node'; import { catalogModelExtensionPoint } from '@backstage/plugin-catalog-node/alpha'; import { agentAiResourceEntityModel } from '@red-hat-developer-hub/backstage-plugin-catalog-model-ai-resource-agent'; +import { AiResourceAgentProcessor } from './AiResourceAgentProcessor'; /** * Registers the agent specType for the AiResource kind in the catalog. @@ -35,11 +37,13 @@ export const catalogModuleAiResourceAgent = createBackendModule({ reg.registerInit({ deps: { model: catalogModelExtensionPoint, + catalog: catalogProcessingExtensionPoint, }, - async init({ model }) { + async init({ model, catalog }) { model.addModelSource( CatalogModelSources.static([agentAiResourceEntityModel]), ); + catalog.addProcessor(new AiResourceAgentProcessor()); }, }); }, diff --git a/workspaces/ai-integrations/yarn.lock b/workspaces/ai-integrations/yarn.lock index 0f49e23ea34..22249d8223d 100644 --- a/workspaces/ai-integrations/yarn.lock +++ b/workspaces/ai-integrations/yarn.lock @@ -10974,8 +10974,10 @@ __metadata: resolution: "@red-hat-developer-hub/backstage-plugin-catalog-backend-module-ai-resource-agent@workspace:plugins/catalog-backend-module-ai-resource-agent" dependencies: "@backstage/backend-plugin-api": "npm:^1.9.2" + "@backstage/backend-test-utils": "npm:^1.11.4" "@backstage/catalog-model": "npm:^1.9.0" "@backstage/cli": "npm:^0.36.3" + "@backstage/plugin-catalog-common": "npm:^1.1.10" "@backstage/plugin-catalog-node": "npm:^2.2.2" "@red-hat-developer-hub/backstage-plugin-catalog-model-ai-resource-agent": "workspace:^" languageName: unknown