Skip to content

create: empty mutable-field registry breaks --set for many nouns (all AI-Evals, environment, service, repository) with no -f fallback #28

Description

@nandesh-harness

Summary

For many create nouns, the --set key=value flag rejects every field with unknown or read-only field "<field>", even though --help advertises those exact fields. --list-fields confirms the cause: these nouns return "No mutable fields defined for this command" — the mutable-field registry is empty, so no --set value is ever accepted. There is also no -f/--file fallback, leaving these nouns with no working create path via the CLI at all.

This appears to be the same root cause as #27 (connector) and #23 (PR), but the scope is much broader — it affects all AI-Evals nouns plus several core ones.

Version

harness version 0.1.10 (2026-06-18T04:38:52Z)

Reproduction

$ harness create eval_target --org default --project <proj> \
    --set name=repro --set identifier=repro --set type=prompt
unknown or read-only field "type"; use --list-fields to see mutable fields

$ harness create eval_target --list-fields
No mutable fields defined for this command.

--help for the same command advertises the fields it then rejects:

$ harness create eval_target --help
Create an AI Evals target: harness create eval_target --set name=... identifier=... type=prompt

Scope — --list-fields audit

Working (registry populated):

Noun --list-fields
secret ID / Label / Type ✅
project ID / Label / Type ✅
organization ID / Label / Type ✅

Broken (empty registry → --set unusable, no -f fallback):

Noun --list-fields
eval_target No mutable fields ❌
eval_dataset No mutable fields ❌
eval_metric No mutable fields ❌
eval_model No mutable fields ❌
eval_metric_set No mutable fields ❌
evaluation No mutable fields ❌
pr No mutable fields ❌ (see #23)
connector No mutable fields ❌ (see #27)
environment No mutable fields ❌
service No mutable fields ❌
repository No mutable fields ❌

No workaround

The broken nouns also reject -f:

$ harness create eval_target -f /dev/null
unknown shorthand flag: 'f' in -f

So affected resources cannot be created through the CLI by any means; callers must drop to the REST API.

Expected

Either:

  1. Populate the mutable-field registry for these nouns so the --set fields shown in --help actually work (and support nested keys like git_source.file_path for git-backed eval resources), or
  2. Add a -f/--file <yaml|json> body flag as a general create path for complex/nested resources.

Impact

Standing up an AI-Evals stack (dataset → target → metric set → evaluation) or any git-backed eval is impossible via the CLI today — every component create fails. Same for code pr, connector, environment, service, and repository create.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions