Skip to content

Move execution-related metadata operations out of CU Master/Worker #5012

@bobbai00

Description

@bobbai00

Task Summary

Sub-issue of #5011.

Move every direct SqlServer call reachable from CU Master / CU Worker
that touches the execution metadata tables (i.e. workflow_executions
and the operator/port URI registry) behind an HTTP service that owns
the credentials. The executor forwards the originating user's JWT.

Code paths in scope (per the Current Usage table in #5011):

  • Execution lifecycle — web/service/WorkflowService.scala (INSERT new
    row, UPDATE on every status / log_location / runtime_stats_uri / result change).
  • State transitions — web/storage/ExecutionStateStore.scala
    (updateWorkflowState from many sites in WorkflowService /
    WorkflowExecutionService).
  • Operator/port URI registry — web/resource/.../WorkflowExecutionsResource.scala
    (insertOperatorPortResultUri, insertOperatorConsoleUri,
    getResultUriByLogicalPortId, getLatestExecutionID, …). Called by
    engine code (e.g. RegionExecutionCoordinator) and by
    SyncExecutionResource.
  • Result export — web/resource/.../WorkflowExecutionsResource.scala
    (exportResultToDataset, exportResultToLocal) →
    web/service/ResultExportService.scala. Endpoints today are hosted on
    CU Master; their DB lookups go through the same registry calls above.
  • Result/log cleanup — web/ComputingUnitMaster.scala (cleanExecutions,
    recurringCheckExpiredResults).
  • Cost-based scheduling — engine/architecture/scheduling/CostEstimator.scala
    (getOperatorExecutionTimeInSeconds reads the latest successful
    workflow_executions.runtime_stats_uri for a wid).

Done when no engine or service code reachable from CU Master / Worker
calls SqlServer for execution-metadata reads or writes, every former
DB call has an HTTP endpoint on web-app with @Auth-checked JWT
authorization, and the existing test suite plus an end-to-end run still
pass.

Task Type

  • Refactor / Cleanup
  • DevOps / Deployment / CI
  • Testing / QA
  • Documentation
  • Performance
  • Other

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No fields configured for Task.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions