Skip to content

Design question: should single-node GPU worker affinity be framework-managed? #538

Open

Description

@WenzheWang

Hi maintainers, before implementing anything, I’d like to clarify whether optional single-node GPU worker affinity belongs in OpenEvolve or should remain entirely evaluator-managed.

From source inspection at 04a37f1, ProcessParallelController derives its worker count from parallel_evaluations and passes the inherited environment to each worker. It does not itself assign different GPUs to different workers.

Custom evaluators can already select devices or submit remote jobs, as discussed in #342. I’m not suggesting that multi-GPU evaluation is unsupported. The question is whether a small built-in option would be useful for local GPU-backed candidate evaluation.

Possible scope, if welcome

  • Opt-in assignment of workers to an explicit set of allocated GPUs.
  • Establish device visibility before evaluator imports or CUDA initialization.
  • Respect scheduler-provided device restrictions rather than discovering or claiming additional host GPUs.
  • Preserve device assignment across normal max_tasks_per_child recycling.
  • Leave existing CPU and unconfigured execution unchanged.

This would not introduce cluster scheduling, automatic GPU-memory admission, preemption, crash recovery/replay, or a security sandbox. It would not duplicate the recovery work in #430.

Validation

I can validate on four A800 GPUs using fixed candidate evaluators: device assignment, concurrent resource use, worker recycling, shutdown and failure cleanup, result correctness, and unchanged default behavior. Throughput and memory measurements would be reported separately from correctness.

Would you prefer:

  1. A minimal optional capability in the process controller;
  2. An executor/helper extension point or maintained evaluator example; or
  3. Keeping this responsibility outside OpenEvolve?

Please flag any existing implementation or planned work. I’d wait for agreement on responsibility and extension points before developing the feature.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions