Add file transfer between client and server

[jgiret]

The following plan has been generated by Claude Opus 4.8. You can notice the presence of gemseo-http plugin, as I have experience in file transfer with GEMSEO, and I wanted to understand the implications with gRPC that I barely know.

As discussed IRL, I'm eager to propose an implementation that would fit Philote, OpenMDAO and GEMSEO. But before that, we should agree on the scope and on some design choices. One is the notion of workspace scope, and as a starter I would go for one-per-process.

Finally, take this proposition as a starter for discussions, for sure it will evolve with our discussions.

Add file transfer between client and server

Summary

Philote currently moves only numeric arrays and small discrete scalars across the
wire. Many real disciplines (CFD, FEA, CAD, pre/post-processing) exchange files
— meshes, restart/solution files, config decks, result archives. This issue proposes
adding first-class file transfer to Philote over the existing gRPC transport, driven
by the way OpenMDAO already represents file paths (discrete string variables), and
taking design cues from gemseo-http, which solves the same
problem for GEMSEO over HTTP/REST.

Motivation

• A remote discipline that reads a mesh or writes a solution file has no way to get
  those bytes to/from the client today.
• OpenMDAO already models "this variable is a file" as a discrete input/output whose
  value is a path string. Philote already transports that string end to end — but the
  string is only meaningful on the machine that produced it. The bytes never move,
  and the path is never rewritten to be valid on the other side.
• We want this to feel native to both Philote (server defines the interface, client
  discovers it) and OpenMDAO (no boilerplate in user components).

Current state (for reviewers)

• Transport: gRPC with DisciplineService, ExplicitService, ImplicitService.
• Payloads: VariableMessage = oneof { Array continuous; DiscreteVariable discrete }.
  • Array = chunked numeric buffer (start/end, chunk size StreamOptions.num_double).
  • DiscreteVariable = google.protobuf.Value (bool/int/float/str/list/dict).
• Compute RPCs are stream_stream and synchronous.
• The server is effectively stateless between calls (no DB, no session, no workspace).
• OpenMDAO binding: add_discrete_input(name, val=None) →
  DiscreteVariable(string_value=<path>) round-trips today via
  _assemble_input_messages / process_inputs / _recover_outputs.

Key consequence: the path already crosses the wire. We only need to add (a) a
byte channel, (b) a way to know which discrete variables are files, (c) path
rewriting on each hop, and (d) somewhere on the server to keep the bytes between calls.

Prior art: how gemseo-http does it

It decouples the file lifecycle from execution into four phases (over HTTP, but the
pattern is what matters):

1. Upload ahead of execution, chunked (POST /v1/file/upload, chunk_number /
   total_chunks); server reassembles, computes sha256sum, stores under a uuid,
   records a File row, returns the file handle.
2. Client-side integrity check — client hashes locally and compares to the
   server's returned sha256sum.
3. Reference by handle in execute — the execute call carries input_files: list[File] (handles, not bytes); file-valued inputs are rewritten to bare
   filenames so the remote discipline reads them from a per-job workspace.
4. Download by job, streamed — GET /v1/job/{id}/files (tree) +
   .../files/download?filename= (streamed bytes).

It also distinguishes variable-bound files (inputs_to_upload/outputs_to_download)
from auxiliary files (file_paths_to_upload/file_paths_to_download), and keeps a
SQL DB + per-job workdir because it is a long-running, multi-user async web service.

We adopt the decoupling, handle-by-reference, chunking, and sha256 integrity
ideas, but implement them natively over gRPC streaming and stay synchronous for a
first cut.

Proposed design

1. A new FileService (bytes travel out-of-band)

protobuf.Value has no bytes kind, so file content must not ride inside
DiscreteVariable. Add a dedicated service mirroring the array-chunking pattern
Philote already uses:

message FileChunk {
  string file_id = 1;   // server-assigned handle (uuid); empty on first upload chunk
  string name    = 2;   // logical/relative name, e.g. "mesh/wing.cgns"
  int64  offset  = 3;   // analogous to Array.start
  bytes  data    = 4;
  string sha256  = 5;   // sent on the final chunk
  bool   last    = 6;
}

message FileRef {
  string file_id = 1;
  string name    = 2;
  string sha256  = 3;
  int64  size    = 4;
}

service FileService {
  rpc UploadFile   (stream FileChunk)       returns (FileRef);         // client-stream, cf. SetVariableShapes
  rpc DownloadFile (FileRef)                returns (stream FileChunk); // server-stream, cf. GetVariableDefinitions
  rpc ListFiles    (google.protobuf.Empty)  returns (stream FileRef);
}

• Add num_bytes to StreamOptions as the byte-chunk analog of num_double; reuse
  utils.get_chunk_indices.
• UploadFile maps onto the existing stream_unary shape (SetVariableShapes);
  DownloadFile onto unary_stream (GetVariableDefinitions).

2. Mark file-valued discrete variables in metadata (server-driven)

Do not introduce a new continuous kFile variable type — reuse the existing
discrete-string variable. Add a flag to VariableMetaData:

// in VariableMetaData
bool is_file = <n>;   // this discrete variable's string value is a file path

The server sets it when declaring the variable; the client learns it during
get_variable_definitions. This keeps Philote's "server owns the interface" model,
rather than gemseo-http's client-side upload/download lists.

3. Server-side workspace (the one real architectural gap)

The server needs a handle→path map that outlives UploadFile so ComputeFunction
can resolve it. Minimum viable version:

• A tempfile.mkdtemp() workspace per server process, file_id → path in memory.
• If concurrent clients must be isolated, key workspaces by a session id passed in
  gRPC metadata. For the common single-discipline-per-process deployment, one
  workspace per process is enough — no DB required (unlike gemseo-http).

4. Path rewriting on each hop

The discrete variable keeps carrying a string; only its meaning changes:

• Client, before compute: value = local absolute path → UploadFile → rewrite the
  discrete value to a bare name/handle (cf. gemseo-http's Path(...).name).
• Server, on receiving an is_file discrete: materialize bytes from the workspace,
  rewrite the value to the server-local absolute path, so the user's
  compute(inputs, outputs, discrete_inputs, ...) gets a real openable file.
• Outputs (reverse): discipline sets discrete_outputs[name] = server path → server
  registers the file and rewrites to a handle → client DownloadFile → rewrite to a
  local path.

Put this logic in the general client/server layer (run_compute /
_assemble_input_messages / process_inputs), so plain gRPC clients benefit and the
OpenMDAO binding stays almost untouched.

5. Integrity + optional dedup (sha256)

• Verify the sha256 returned by UploadFile against a locally computed hash
  (Philote has no integrity story today).
• Optional but valuable for optimization loops: if the server already holds a file with
  the same hash, skip re-upload. OpenMDAO calls compute once per iteration and often
  re-sends an unchanged mesh — this avoids repeatedly shipping identical bytes.

End-to-end flow

Client                                   Server
  |  UploadFile(stream FileChunk) ------>  reassemble -> workspace, sha256 -> FileRef
  |  <----------------------------------  FileRef(file_id, sha256, size)
  | (verify sha256; rewrite discrete value -> handle/name)
  |  ComputeFunction(stream) ----------->  on is_file discrete: resolve handle -> local path
  |                                        discipline.compute(...) reads/writes real files
  |  <----------------------------------  discrete outputs carry handles for produced files
  |  DownloadFile(FileRef) ------------->  stream bytes from workspace
  |  <----------------------------------  FileChunk stream
  | (rewrite discrete output value -> local path)

OpenMDAO integration

Nearly nothing changes in the binding:

• compute() already routes discrete dicts through create_local_discrete_inputs →
  run_compute; string values flow as-is.
• client_setup can optionally use the new is_file flag if we ever want to declare
  file variables specially, but the default add_discrete_input(val=None) already
  works.

Scope / phasing

Phase 1 (MVP, this issue):

• FileService (UploadFile, DownloadFile) + FileChunk/FileRef.
• is_file flag on VariableMetaData; num_bytes on StreamOptions.
• In-memory per-process workspace (file_id → path).
• Path rewriting in the general client/server for explicit disciplines.
• sha256 integrity check on upload.
• Regenerate stubs (utils/compile_proto.py) after the .proto change.

Phase 2 (follow-ups):

• Implicit disciplines (ComputeResiduals) parity.
• sha256 dedup / upload short-circuit for optimization loops.
• Auxiliary (non-variable) files, à la file_paths_to_upload.
• Session-scoped workspaces via gRPC metadata for multi-client servers.
• ListFiles / cleanup / TTL on the workspace.

Explicitly out of scope (for now):

• Async job queue + long polling + SQL DB + auth (gemseo-http has these because it is a
  multi-user web service; Philote compute is synchronous streaming). Revisit only if
  long-running file-heavy disciplines demand it.

Open questions / decisions needed

1. Workspace scope: one-per-process (simplest) vs session-keyed via metadata? This
   determines whether we thread a session id through the RPCs — decide before writing
   the .proto.
2. Handle vs bare name in the rewritten discrete value: opaque file_id, or a bare
   filename the server resolves inside the workspace (gemseo-http uses the latter)?
3. Upload trigger: always re-upload per compute, or hash-gate re-uploads from the
   start (Phase 1 vs Phase 2)?
4. .proto ownership: these messages live in the MDO-Standards/Philote-MDO
   submodule — changes must be proposed there first, then the submodule bumped here.

Testing

• Unit: chunked upload/reassembly, sha256 match/mismatch, path rewriting both
  directions, empty-file edge case.
• Integration: an example explicit discipline that reads an input file and writes an
  output file, exercised through RemoteExplicitComponent end to end.

Docs / changelog

• New tutorial page under docs/docs/ showing a file-based remote discipline.
• CHANGELOG.md [Unreleased] → ### Features: "Add client↔server file transfer for
  file-valued (discrete) variables."

Task checklist

• Propose FileService, FileChunk, FileRef, VariableMetaData.is_file,
  StreamOptions.num_bytes in the Philote-MDO proto repo; bump the submodule.
• Regenerate stubs via utils/compile_proto.py.
• Implement FileServer mixin (UploadFile/DownloadFile) + workspace manager.
• Implement FileClient mixin + sha256 verification.
• Wire path rewriting into run_compute / _assemble_input_messages /
  process_inputs.
• Example discipline + integration test through OpenMDAO.
• Docs page + CHANGELOG.md entry.

[chrislupp]

thanks. let's discuss it sometime soon if you have time. I've skimmed it and will think a bit more about the details