[hyperactor] telemetry: don't record net i/o loop spans in scuba

Pull Request resolved: #1908

These are used primarily to provide context for any logged events, and should not be recorded as separate spans in scuba.

We introduce a general mechanism, exposing the key `hyperactor_telemetry::skip_record` which, if included in a span, causes the downstream scuba subscribe to skip recording the span as its own discrete event. This also prevents us from entering and exiting the spans accordingly.

We do maintain the recorded fields, which provide important context; they continue to propagate to the individually recorded entries.
ghstack-source-id: 324277121
@exported-using-ghexport

Differential Revision: [D87273700](https://our.internmc.facebook.com/intern/diff/D87273700/)

Author: mariusae

hyperactor/src/channel/net/client.rs

@@ -760,6 +760,8 @@ where
         .as_ref()
         .map(|acked_seq| AckedSeqValue(acked_seq.clone()));
 
+    use hyperactor_telemetry::skip_record;
+
     tracing::span!(
         Level::ERROR,
         "net i/o loop",
@@ -769,6 +771,7 @@ where
         largest_acked = largest_acked.as_value(),
         outbox = QueueValue::from(&deliveries.outbox.deque).as_value(),
         unacked = QueueValue::from(&deliveries.unacked.deque).as_value(),
+        skip_record,
     )
 }
 

hyperactor_mesh/examples/dining_philosophers.rs

@@ -232,7 +232,7 @@ impl Waiter {
 
 #[tokio::main]
 async fn main() -> Result<ExitCode> {
-    hyperactor_telemetry::initialize_logging_for_test();
+    hyperactor::initialize_with_current_runtime();
 
     // Option: run as a local process mesh
     // let host_mesh = HostMesh::process(extent!(hosts = 1), BootstrapCommand::current().unwrap())

hyperactor_telemetry/src/lib.rs

@@ -53,6 +53,21 @@ const ENV_VALUE_TEST: &str = "test";
 #[allow(dead_code)]
 const ENV_VALUE_LOCAL_MAST_SIMULATOR: &str = "local_mast_simulator";
 
+/// A marker field used to indicate that a span should not be recorded as
+/// individual start/end span events; rather the span is purely used to
+/// provide context for child events.
+///
+/// Note that the mechanism for skipping span recording uses the precise
+/// name "skip_record", thus it must be used as a naked identifier:
+/// ```ignore
+/// use hyperactor_telemetry::skip_record;
+///
+/// tracing::span!(..., skip_record);
+/// ```
+#[allow(non_upper_case_globals)]
+// pub const skip_record: tracing::field::Empty = tracing::field::Empty;
+pub const skip_record: bool = true;
+
 pub mod in_memory_reader;
 #[cfg(fbcode_build)]
 mod meta;