fix(coder-labs/modules/codex): deep-merge config.toml on restart instead of overwriting#896
fix(coder-labs/modules/codex): deep-merge config.toml on restart instead of overwriting#89635C4n0r wants to merge 21 commits into
Conversation
35C4n0r
force-pushed
the
35C4n0r/idempotent-codex-config
branch
from
9be9f74 to
f980245
Compare
35C4n0r
changed the title
After a dasel roundtrip, TOML values use single quotes instead of double quotes. Update the codex-with-ai-gateway and ai-gateway-with-custom-base-config tests to use regex matching that accepts either quote style. Also fix idempotent-run-twice-no-change to read the config file directly from the container instead of piping TOML strings through shell echo (which breaks on single quotes).
The idempotent-run-twice-no-change test was calling dasel in a separate execContainer shell where the PATH export from the install script is not available. Instead, compare the raw config output after runs 2 and 3 (both post-roundtrip, so serialization is stable and byte-comparison is valid).
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
…le dasel conversion Replace TOML string concatenation with jq-native JSON building: - Extract write_minimal_default_config() back as its own function, now returning JSON on stdout via jq. - populate_config_toml() assembles all config sources as JSON, deep-merges with jq, and does a single dasel JSON-to-TOML conversion at the end. - Remove merge_toml_config() and all TOML string building. - Update test assertions to accept either quote style since all output now goes through dasel.
This comment has been minimized.
This comment has been minimized.
1 similar comment
|
/coder-agents-review |
Script fixes: - Rename write_minimal_default_config to build_minimal_default_config (no longer writes to disk, emits JSON to stdout). - Guard corrupted existing config: if dasel cannot parse the existing TOML, error out and exit instead of silently proceeding. - Atomic config write: write to a temp file and mv, preventing data loss if the process is interrupted mid-write. - Add jq availability check before populate_config_toml, consistent with how other registry modules handle hard dependencies. - Normalize blank lines between function definitions. Test fixes: - idempotent-mcp-deep-merge: use sed address range to only replace the github server command, assert filesystem command is still npx. - workdir-trusted-project: tighten regex to require bracket syntax instead of matching any line containing the path. - Rename idempotent-run-twice-no-change to idempotent-stable-after-roundtrip (test runs 3 times, not 2). - Remove unnecessary regex escaping of forward slashes. - Strengthen combination test assertions to check values, not just key presence.
| if [ -n "$${ARG_BASE_CONFIG_TOML}" ]; then | ||
| printf "Using provided base configuration\n" | ||
| echo "$${ARG_BASE_CONFIG_TOML}" > "$${config_path}" | ||
| new_json=$(echo "$${ARG_BASE_CONFIG_TOML}" | dasel -i toml -o json) |
There was a problem hiding this comment.
why are we mixing json and toml?
There was a problem hiding this comment.
dasel does not support deep merge of toml, so I came up with the idea that all the internal processing will be done in json (jq ftw), the final output will be written in toml. dasel only does 2 things 1. convert any toml to json for internal manipulation, 2. write the final json output as toml.
| else | ||
| printf "AI Gateway provider already defined in config, skipping append\n" | ||
| printf "Adding AI Gateway configuration\n" | ||
| local aibridge_json |
There was a problem hiding this comment.
we do not use aibridge anymore. rename to
| local aibridge_json | |
| local ai_gateway_json |
There was a problem hiding this comment.
@matifali, is it okay if we release it in a new patch ? The readme already seems to be using ai_gateway for user facing variables, I'll do internal refactor in a new PR.
There was a problem hiding this comment.
why are we doing toml to json to toml? a few real costs with this:
- comments in
~/.codex/config.tomlget stripped on every restart (dasel can't preserve them). - two new binary deps (dasel + jq), and the dasel download isn't checksum-verified.
two cleaner options:
marker block: module owns a fenced region, user owns everything outside.
# >>> coder-managed: codex module >>>
preferred_auth_method = "apikey"
[mcp_servers.github]
command = "npx"
# <<< coder-managed: codex module <<<sed strips and re-emits the block on each run. no deps, comments preserved, byte-stable. overrides go through terraform variables, which matches the rest of the registry.
yq (mikefarah) native toml: if we keep merging, yq -p toml -o toml eval-all '. as $i ireduce ({}; . *+ $i)' a.toml b.toml is the same thing in one binary instead of two. still loses comments but a strict win over dasel + jq.
prefer marker block. happy to be wrong if there's a reason.
|
yq dosen't seem to work. |
|
as for the block marker, everything written by the module will go inside the block marker (this includes the mcps provided by the user and the base_toml or user_provided_toml). |
…enhance idempotency
…ser bare keys and sections
… in config handling
2 participants
Problem
populate_config_tomlunconditionally overwrote~/.codex/config.tomlon every workspace start, wiping any user edits (custom MCP servers, auth preferences, project trust entries, hand-tuned settings).Solution
Replace the destructive overwrite with a deep-merge strategy. All config is built as JSON using
jq, merged withjq -s '.[0] * .[1]'(recursive object merge), and converted to TOML via a singledasel -i json -o tomlcall at the end.The user's existing on-disk config always takes precedence. Any value a user has set or changed in
~/.codex/config.tomlwill never be overwritten by the module. New keys from the module are added; existing keys are left untouched. This applies at every nesting level (top-level scalars, nested maps likemcp_servers,projects,model_providers).Precedence chain (highest to lowest)
~/.codex/config.toml): always winsbase_config_toml(if provided): wins over module defaultsHow it works
install_dasel()(new): downloads dasel v3.4.0 for the current OS/arch.jqavailability check (new): errors out with a clear message ifjqis not installed, consistent with how other registry modules handle hard dependencies.build_minimal_default_config()(updated): builds the default config as a JSON object usingjqand prints it to stdout. Conditionally includesmodel_provider,model_reasoning_effort, andprojectstrust entries.populate_config_toml()(rewritten): assembles the full config in JSON:base_config_toml(converted viadasel -i toml -o json) or the output ofbuild_minimal_default_config.mv.Changes
scripts/install.sh.tftplinstall_dasel, jq check; rewrote config building to use jq for JSON construction and merging; atomic writes via temp file; graceful error on corrupted config; fixed all$$/$tftpl escapingmain.test.tsREADME.mdNew tests
idempotent-defaults-preserve-user-editsidempotent-mcp-deep-mergeidempotent-base-config-preserves-user-editsidempotent-stable-after-roundtripidempotent-mcp-new-servers-added-existing-keptidempotent-ai-gateway-preserves-user-providermodel_provider; survives AI gateway restartbase-config-plus-mcp-combinedall-config-sources-combinedidempotent-all-sources-user-edits-survive