Skip to content

[Feature]: W3: Provider Logging Unification Across All 15 External Provider Repos, devsy-runtime-sdk & Drivers #1845

Open

Description

@skevetter

Parent Epic: #1841

Overview

Bring all 15 external provider repositories in devsy-org and all built-in drivers onto the unified log/slog JSON contract, level propagation, and structured error classification:

  • 12 Go Providers Importing github.com/devsy-org/devsy/pkg/log: devsy-provider-aws, devsy-provider-gcloud, devsy-provider-azure, devsy-provider-digitalocean, devsy-provider-civo, devsy-provider-terraform, devsy-provider-lima, devsy-provider-orbstack, devsy-provider-kubernetes, devsy-provider-ecs, devsy-provider-dockerless, devsy-provider-ssh
  • 2 Standalone Go Machine Providers (Minimal go.mod, no devsy import): devsy-provider-docker-sandbox, devsy-provider-harvester
  • 2 Runtime SDK & External Plugin Repos: devsy-runtime-sdk, devsy-provider-microsandbox

Subtasks & Engineering Specifications

  • Task 3.1: Host-Side Machine & Custom Driver Streamer (machine_client.go, env.go, custom.go) (size/m)

    • Target Files:
      • pkg/provider/env.go
      • pkg/client/clientimplementation/machine_client.go
      • pkg/driver/custom/custom.go
    • Implementation Details:
      1. In provider.GetBaseEnvironment (pkg/provider/env.go), export DEVSY_LOG_FORMAT=json alongside DEVSY_LOG_LEVEL=log.LevelString().
      2. In machineExecutor.lifecycleCommand (pkg/client/clientimplementation/machine_client.go) and customDriver (pkg/driver/custom/custom.go), replace log.Writer(log.LevelInfo) with log.NewJSONLogStreamer configured with DefaultAttrs: [provider, machine_id, operation] and CaptureLines: 50.
      3. On non-zero exit from a machine binary command, wrap the error in a clierr.CLIError containing the last captured stderr lines and any structured error_code emitted by the provider binary.
      4. Wrap machineClient.Create, Start, Stop, Delete, and Status in status.RunStep(ctx, ...) so machine lifecycle timings and errors flow into pkg/status and pkg/telemetry.
  • Task 3.2: devsy-provider-ssh Migration to log.InitProvider("ssh") & SSH Transport Error Classification (size/m)

    • Target Files:
      • pkg/clierr/errors.go, pkg/clierr/ssh.go (new)
      • pkg/client/clientimplementation/workspace_client.go
      • External repo: devsy-org/devsy-provider-ssh (cmd/root.go, go.mod)
    • Implementation Details:
      1. Add an SSH stderr classifier in pkg/clierr/ssh.go that recognizes common OpenSSH transport failure signatures from JSONLogStreamer.ErrorOutput() and maps them to actionable clierr.Code values (ssh_auth_failed, ssh_host_key_failed, ssh_unreachable, ssh_command_failed) with user-facing remediation hints.
      2. In devsy-org/devsy-provider-ssh, bump github.com/devsy-org/devsy and replace log.Init(log.Config{Verbosity: 1, ...}) in cmd/root.go with log.InitProvider("ssh").
  • Task 3.3: devsy-provider-aws Migration to log.InitProvider("aws") & Structured Lifecycle Logging (size/m)

    • Target Files:
      • External repo: devsy-org/devsy-provider-aws (cmd/root.go, go.mod, and EC2/IAM/SSM lifecycle handlers)
    • Implementation Details:
      1. Bump github.com/devsy-org/devsy in devsy-provider-aws and replace log.Init(log.Config{Verbosity: 1, ...}) in cmd/root.go with log.InitProvider("aws").
      2. Emit structured step timings (aws_step, duration_ms, instance_id) and classified error_code attributes (aws_credentials_expired, aws_quota_exceeded, aws_ssm_unreachable, aws_subnet_misconfigured).
  • Task 3.4: External Runtime SDK (devsy-runtime-sdk), devsy-provider-microsandbox & pkg/driver/external Streamer (size/m)

    • Target Files:
      • External repo: devsy-org/devsy-runtime-sdk/log/slog.go (new)
      • External repo: devsy-org/devsy-provider-microsandbox (cmd/devsy-runtime-microsandbox/main.go, go.mod)
      • pkg/driver/external/session.go, pkg/driver/external/session_logging_test.go
    • Implementation Details:
      1. Add a zero-dependency log/slog helper package in devsy-runtime-sdk/log (sdklog.NewPluginLogger(providerName string) *slog.Logger) reading DEVSY_LOG_LEVEL and emitting Devsy-compatible JSON logs to os.Stderr, and adopt it in devsy-provider-microsandbox.
      2. In pkg/driver/external/session.go, route plugin Stderr through JSONLogStreamer (preserving streamingRedactor guarantees verified by session_logging_test.go) with DefaultAttrs: [component="external-driver", provider].
  • Task 3.5: Built-in Drivers (docker, kubernetes, apple, microsandbox) Structured Attributes (size/m)

    • Target Files:
      • pkg/driver/docker/*
      • pkg/driver/kubernetes/*
      • pkg/driver/apple/*
      • pkg/driver/microsandbox/*
    • Implementation Details:
      1. Enrich driver entrypoints with log.WithContext(ctx, log.AttrComponent("driver"), log.AttrProvider(driverName), log.AttrWorkspaceID(workspaceID)).
      2. Emit container/VM runtime command execution durations and exit codes as structured slog.Attrs at Debug/Info level.
  • Task 3.6: Rollout log.InitProvider Across Remaining 10 pkg/log-Importing Go Provider Repositories (size/m)

    • Target Repositories:
      • Machine providers: devsy-org/devsy-provider-gcloud, devsy-org/devsy-provider-azure, devsy-org/devsy-provider-digitalocean, devsy-org/devsy-provider-civo, devsy-org/devsy-provider-terraform, devsy-org/devsy-provider-lima, devsy-org/devsy-provider-orbstack
      • Custom container driver providers: devsy-org/devsy-provider-kubernetes, devsy-org/devsy-provider-ecs, devsy-org/devsy-provider-dockerless
    • Implementation Details:
      1. Bump github.com/devsy-org/devsy in go.mod across all 10 repositories so they inherit the log/slog engine and drop indirect go.uber.org/zap dependencies.
      2. Replace cfg := log.Config{Verbosity: 1}; if os.Getenv("DEVSY_DEBUG") == "true" { cfg.Debug = true }; log.Init(cfg) in cmd/root.go with log.InitProvider("<provider>").
      3. In devsy-org/devsy-provider-kubernetes, also remove the direct go.uber.org/zap v1.27.1 requirement from go.mod.
  • Task 3.7: Align Standalone Go Machine Providers (devsy-provider-docker-sandbox, devsy-provider-harvester) on log/slog JSON Contract (size/s)

    • Target Repositories:
      • devsy-org/devsy-provider-docker-sandbox (cmd/devsy-provider-docker-sandbox/main.go)
      • devsy-org/devsy-provider-harvester (cmd/harvester-provider/main.go)
    • Implementation Details:
      1. Configure stdlib slog.NewJSONHandler(os.Stderr, ...) in both standalone binaries (without adding a dependency on github.com/devsy-org/devsy), honoring DEVSY_LOG_LEVEL and DEVSY_LOG_FORMAT=json (ts, level, msg, provider, machine_id, error_code) while preserving docker-sandbox and harvester recovery/doctor CLI output contracts.

Verification & Acceptance Criteria

  • go test ./pkg/driver/... ./pkg/client/... ./pkg/provider/... ./pkg/clierr/... passes.
  • All 15 external provider repositories (12 pkg/log-importing Go providers, 2 standalone Go providers, and 1 devsy-runtime-sdk plugin) stream structured slog JSON logs through the host logger with levels, attributes, and secret redaction intact, and zero go.uber.org/zap dependencies remain in any provider go.mod.

Provider and SDK delivery

CI and CodeRabbit apply to these repositories. The owner approved their normal release flows. Publish one repository at a time after its actual dependency versions are on main and its logging PR merges with green CI. Record the release link and verify the published artifact and installation before marking a row complete.

Repository Logging PR Source and verification Release and install
devsy-runtime-sdk PR #35 · PR #36 Adoption squashed as 0f2b4937 after exact CI and CodeRabbit passed. The merged source resolves to v1.6.1-0.20261121014606-0f2b4937cc59. Normal v1.7.0 release PR #37 merged. Release CI pending. v1.7.0 published. The public Go checksum and an isolated consumer verified the module, JSON diagnostics on stderr, and credential masking.
devsy-provider-docker-sandbox PR #40 · PR #41 Adoption squashed as 400a592d after hosted CI, native distribution installs, and exact-head CodeRabbit passed. Automatic v1.1.0 release PR #42 blocked by the native evidence check. Requires reviewed native evidence for the exact candidate. The qualified sbx binary is unavailable on this workstation. The lead is handling this release gate.
devsy-provider-harvester PR #6 · PR #7 · Version PR #9 Adoption squashed as 7176a981. The v1.1.0 manifest and asset URL update passed hosted CI and exact-head CodeRabbit and was squashed at 83808120. v1.1.0 published. Five binaries and manifest checksum-verified. Native Windows install passed.
devsy-provider-microsandbox PR #90 Signed f5bfc97e pins the merged shared logging module and the actual merged SDK. Full local checks passed with workspaces off. Exact-head hosted CI passed. Exact-head CodeRabbit passed with zero open threads. Adoption squashed as 36553629. v1.3.0 published. Release PR #91 auto-merged as 3258eb68. All four public binaries and the regenerated manifest verified. Manifest and alias installation passed.
devsy-provider-aws Draft PR #169 Shared logging pinned on signed 3d539cc5. 64 scoped local race test results passed with a private Devsy Core adapter workspace. The merged Devsy Core adapter pin is pending. Pending
devsy-provider-azure Draft PR #128 Shared logging pinned on signed fca479cf. 19 scoped local race test results passed with a private Devsy Core adapter workspace. The merged Devsy Core adapter pin is pending. Pending
devsy-provider-civo Draft PR #101 Shared logging pinned on signed 61fe0d2c. 20 scoped local race test results passed with a private Devsy Core adapter workspace. The merged Devsy Core adapter pin is pending. Pending
devsy-provider-digitalocean Draft PR #118 Shared logging pinned on signed f18eb5bc. 24 scoped local race test results passed with a private Devsy Core adapter workspace. The merged Devsy Core adapter pin is pending. Pending
devsy-provider-dockerless Draft PR #37 Shared logging pinned on signed c47b5512. 20 scoped local race test results passed with a private Devsy Core adapter workspace. The merged Devsy Core adapter pin is pending. Pending
devsy-provider-ecs Draft PR #45 Shared logging pinned on signed 56596db0. 16 scoped local race test results passed with a private Devsy Core adapter workspace. The merged Devsy Core adapter pin is pending. Pending
devsy-provider-gcloud Draft PR #111 Shared logging pinned on signed 97073cbb. 16 scoped local race test results passed with a private Devsy Core adapter workspace. The merged Devsy Core adapter pin is pending. Pending
devsy-provider-kubernetes Draft PR #53 Shared logging pinned on signed 8f522186. 26 scoped local race test results passed with a private Devsy Core adapter workspace. The merged Devsy Core adapter pin is pending. Pending
devsy-provider-lima Draft PR #33 Shared logging pinned on signed 38207ec8. 56 scoped local race test results passed with a private Devsy Core adapter workspace. The merged Devsy Core adapter pin is pending. Pending
devsy-provider-orbstack Draft PR #50 Shared logging pinned on signed dfdb9878. 22 scoped local race test results passed with a private Devsy Core adapter workspace. The merged Devsy Core adapter pin is pending. Pending
devsy-provider-ssh Draft PR #180 Shared logging pinned on signed ec54d58a. 79 scoped local race test results passed with a private Devsy Core adapter workspace. The merged Devsy Core adapter pin is pending. Pending
devsy-provider-terraform Draft PR #38 Shared logging pinned on signed 13685b5b. 21 scoped local race test results passed with a private Devsy Core adapter workspace. The merged Devsy Core adapter pin is pending. Pending

The host and built-in driver work is in Core PR #1850. Final compatibility and provider installation remain open.

Activity

  1. changed the title [-][Feature]: W3: Provider Logging Unification (devsy-provider-aws, devsy-provider-ssh, devsy-runtime-sdk, Drivers)[/-] [+][Feature]: W3: Provider Logging Unification Across All 15 External Provider Repos, devsy-runtime-sdk & Drivers[/+] on Nov 17, 2026
  2. skevetter commented on Nov 20, 2026

    @skevetter
    ContributorAuthor

    The provider and SDK table now tracks all 16 repositories, including the three open PRs for the SDK, Harvester, and Docker Sandbox. The other source candidates are staged. Publication starts after the actual shared logging version is available from the foundation, and Core-backed providers also need their merged Core version. Normal releases are approved one repository at a time after merge and green CI, followed by artifact and install verification.

  3. skevetter commented on Nov 20, 2026

    @skevetter
    ContributorAuthor

    Private Go workspace checks passed against the updated logging foundation source for the SDK, Harvester, and Docker Sandbox. The table records 16, 22, and 39 passing test results respectively. Published dependency pins, CI, and releases still depend on the foundation merge.

  4. skevetter commented on Nov 20, 2026

    @skevetter
    ContributorAuthor

    Shared logging drafts are open for source review. The reported race tests and repository checks passed in private Go workspaces. Final merged dependency versions, hosted CI, and CodeRabbit are required before any merge or release.

  5. skevetter commented on Nov 20, 2026

    @skevetter
    ContributorAuthor

    Shared logging drafts are open for source review. The reported race tests and repository checks passed in private Go workspaces. Final merged dependency versions, hosted CI, and CodeRabbit are required before any merge or release.

  6. skevetter commented on Nov 20, 2026

    @skevetter
    ContributorAuthor

    Shared logging drafts are open for source review. The reported race tests and repository checks passed in private Go workspaces. Final merged dependency versions, hosted CI, and CodeRabbit are required before any merge or release.

  7. skevetter commented on Nov 20, 2026

    @skevetter
    ContributorAuthor

    Shared logging drafts are open for source review. The reported race tests and repository checks passed in private Go workspaces. Final merged dependency versions, hosted CI, and CodeRabbit are required before any merge or release.

  8. skevetter commented on Nov 20, 2026

    @skevetter
    ContributorAuthor

    Shared logging drafts are open for source review. The reported race tests and repository checks passed in private Go workspaces. Final merged dependency versions, hosted CI, and CodeRabbit are required before any merge or release.

  9. skevetter commented on Nov 20, 2026

    @skevetter
    ContributorAuthor

    Hosted Go checks stopped on the missing shared logging pin for the drafts below. The local race tests used a private Go workspace. Final hosted CI will run after the merged dependencies are pinned.

  10. skevetter commented on Nov 20, 2026

    @skevetter
    ContributorAuthor

    Shared logging drafts are open for source review. The reported race tests and repository checks passed in private Go workspaces. Final merged dependency versions, hosted CI, and CodeRabbit are required before any merge or release.

  11. skevetter commented on Nov 20, 2026

    @skevetter
    ContributorAuthor

    Hosted Go checks stopped on the missing shared logging pin for the drafts below. The local race tests used a private Go workspace. Final hosted CI will run after the merged dependencies are pinned.

  12. skevetter commented on Nov 20, 2026

    @skevetter
    ContributorAuthor

    Hosted Go checks stopped on dependency pins for the drafts below. The logs show either the missing shared module or logging and status APIs absent from the currently pinned Core. The local race tests used a private Go workspace. Final hosted CI will run after the merged dependencies are pinned.

  13. skevetter commented on Nov 20, 2026

    @skevetter
    ContributorAuthor

    Hosted Go checks stopped on dependency pins for the drafts below. The logs show either the missing shared module or logging and status APIs absent from the currently pinned Core. The local race tests used a private Go workspace. Final hosted CI will run after the merged dependencies are pinned.

  14. 6 remaining items

  15. skevetter commented on Nov 21, 2026

    @skevetter
    ContributorAuthor

    Docker Sandbox adoption #41 was squashed at signed 400a592d. The release and published install checks are pending. Harvester remains first in the release queue.

  16. skevetter commented on Nov 21, 2026

    @skevetter
    ContributorAuthor

    Docker Sandbox main CI passed after the adoption squash. The normal Release Please workflow opened v1.1.0 release PR #42. The tag, assets, and published provider install remain pending.

  17. skevetter commented on Nov 21, 2026

    @skevetter
    ContributorAuthor

    The automatic Docker Sandbox v1.1.0 release PR failed its native evidence gate. The documented flow requires maintainer-reviewed native qualification for base 400a592d and the updated SBX_NATIVE_EVIDENCE repository variable. Release, artifacts, and install proof remain pending.

  18. skevetter commented on Nov 21, 2026

    @skevetter
    ContributorAuthor

    SDK PR #36 passed hosted CI at signed 7943089f. CodeRabbit accepted the single re-review request and is reviewing that exact head.

  19. skevetter commented on Nov 21, 2026

    @skevetter
    ContributorAuthor

    SDK PR #36 is ready for squash at signed 7943089f. Hosted CI and exact-head CodeRabbit passed with zero open threads. The actual SDK merge is required before MicroSandbox can pin it.

  20. skevetter commented on Nov 21, 2026

    @skevetter
    ContributorAuthor

    SDK adoption #36 was squashed at signed 0f2b4937. The Go proxy verified v1.6.1-0.20261121014606-0f2b4937cc59. MicroSandbox #90 now uses that real SDK pin on signed f5bfc97e. Full local race tests, build, and Prek passed with workspaces off. Hosted qualification and release checks remain open.

  21. skevetter commented on Nov 21, 2026

    @skevetter
    ContributorAuthor

    Harvester v1.1.0 version PR #9 is ready for squash at signed 31a93cd. The Linux and Windows checks and exact-head CodeRabbit passed, with zero open threads. The signed tag and published install verification follow the merge.

  22. skevetter commented on Nov 21, 2026

    @skevetter
    ContributorAuthor

    Harvester’s version PR #9 was squashed. The signed v1.1.0 tag workflow is active. The SDK’s merged commit passed main CI and its normal automation opened v1.7.0 release PR #37. Both releases still need published verification.

  23. skevetter commented on Nov 21, 2026

    @skevetter
    ContributorAuthor

    MicroSandbox #90 passed hosted CI on signed f5bfc97e, including the package manifest and alias installation. The PR is ready for CodeRabbit review.

  24. skevetter commented on Nov 21, 2026

    @skevetter
    ContributorAuthor

    Harvester v1.1.0 is published. Five binaries and the manifest match the checksums, and native Windows installation passed with managed kubectl. MicroSandbox #90 is ready for squash at f5bfc97e after exact-head CI and CodeRabbit passed with zero open threads.

  25. skevetter commented on Nov 21, 2026

    @skevetter
    ContributorAuthor

    MicroSandbox #90 was squashed at signed 36553629. Main CI starts its normal release flow. The SDK’s automatic v1.7.0 release PR also merged, and its release-building workflow remains pending. Published artifacts and install checks remain open for both.

  26. skevetter commented on Nov 21, 2026

    @skevetter
    ContributorAuthor

    SDK v1.7.0 is public. Its tag resolves to the squash of #37. A client built from the public Go proxy verified JSON diagnostics on stderr and masked both synthetic credentials. MicroSandbox adoption #90 passed main CI and opened automatic release PR #91 for v1.3.0. Publication and install verification are pending.

  27. skevetter commented on Nov 21, 2026

    @skevetter
    ContributorAuthor

    MicroSandbox’s v1.3.0 release PR passed its required checks and auto-merged at 3258eb685e491cdc8423cc96e0320bf2336df53d. The normal release workflow is building assets for that exact commit. Published checksums and install verification remain pending.

  28. skevetter commented on Nov 21, 2026

    @skevetter
    ContributorAuthor

    MicroSandbox v1.3.0 is published from signed release squash 3258eb685e491cdc8423cc96e0320bf2336df53d. Downloaded all four public binaries and verified the published checksums. The release commit regenerated the public manifest and checksums byte-for-byte. The release workflow passed manifest and alias installation.

  29. skevetter commented on Nov 21, 2026

    @skevetter
    ContributorAuthor

    Docker Sandbox release remains with the lead for qualified native evidence. The remaining 12 providers await a published pin for the merged Core adapter. The verified SDK and MicroSandbox releases used their normal automation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions