dev to main release

docs/trials_table_mapping.md

@@ -75,7 +75,7 @@ Columns are grouped by the raw source they map from.
 | Trials column | Source field |
 | --- | --- |
 | `ITI_beta`, `ITI_min`, `ITI_max`, `ITI_duration` | `inter_trial_interval_duration` |
-| `block_beta`, `block_duration`, `block_min`, `block_max` | `block_length` |
+| `block_beta`, `block_duration`, `block_min`, `block_max` | `block_length`. `block_max` is one below the configured maximum, which accounts for the floor applied upstream. |
 | `delay_beta`, `delay_duration`, `delay_min`, `delay_max` | `quiescent_duration_key` (scalar distribution, so no beta/min/max) |
 
 ### From `TrialMetrics.json` (`SoftwareEvents` stream)
@@ -100,7 +100,7 @@ Columns are grouped by the raw source they map from.
 | `auto_waterL` / `auto_waterR` | **Scheduled autowater only**: `1` when `trial.metadata.extra.is_autowater` is `True` **and** `is_auto_reward_right` points to that side. `0` otherwise, including when the trial's free water came from the anti-bias algorithm — that is reported by `anti_bias_left_water` / `anti_bias_right_water`. `is_auto_reward_right` is only the delivery *channel* (free water fired, and to which side); the mechanism comes from the metadata, so the two columns are mutually exclusive. Not gated on `is_rewarded`: the column records what the task did, and free water fires at the go cue regardless of how the animal's own choice resolves. Note this is narrower than the legacy `dynamic-foraging-task` column of the same name, which was the ungated channel ("Autowater given at Left", straight from `B_AutoWaterTrial`) and predates anti-bias water. |
 | `anti_bias_left_water` / `anti_bias_right_water` | Boolean. `True` when the anti-bias algorithm delivered a water intervention to that side — i.e. `trial.metadata.extra.is_bias_water_intervention` is `True` **and** `is_auto_reward_right` points to that side (`False` → left, `True` → right). The anti-bias water uses the same auto-response channel as scheduled autowater, so the `is_bias_water_intervention` flag is what distinguishes it and the two columns are mutually exclusive. `False` otherwise. Like `auto_water*`, **not** gated on `is_rewarded`: these columns record what the algorithm did, and the intervention fires at the go cue regardless of how the animal's own choice resolves. The reward-delivery series *is* reward-keyed, so this column can exceed the series' `auto` count. |
 | `anti_bias_lickspout_movement` | Signed horizontal displacement (mm, positive is rightward) the anti-bias algorithm moved the lickspouts on this trial: `trial.lickspout_offset_delta` when `trial.metadata.extra.is_bias_stage_intervention` is `True`, else `0.0`. |
-| `bait_left` / `bait_right` | Boolean. `bait_right` is `True` if `p_reward_right == 1` and `is_auto_reward_right` is `None` or `False`. `bait_left` is `True` if `p_reward_left == 1` and `is_auto_reward_right` is `None` or `True`. |
+| `bait_left` / `bait_right` | Boolean, read straight from `trial.metadata.extra.is_left_baited` / `is_right_baited` — the bait state the acquisition software reports for each port. `False` when the trial carries no extra metadata. |
 | `response_duration` | `response_deadline_duration`. |
 | `reward_consumption_duration` | `Trial -> reward_consumption_duration`. |
 | `reward_probabilityL` / `reward_probabilityR` | The **block** probability from `Trial -> metadata -> p_reward_left` / `p_reward_right`. The top-level `trial.p_reward_left` / `p_reward_right` is the per-trial probability, not the block probability, so it is not used here. `None` when the trial or its metadata is missing. |
@@ -150,14 +150,14 @@ durations track the configured ones (reward consumption ≈
 | `reward_consumption_start_time` | `RewardConsumptionPeriod` `timestamp`. |
 | `reward_consumption_stop_time` | `ItiPeriod` `timestamp`. |
 | `ITI_start_time` | `ItiPeriod` `timestamp`. |
-| `ITI_stop_time` | The **next** trial's `QuiescentPeriod` `timestamp`; `NaN` on the last trial of the session. |
+| `ITI_stop_time` | The **next** trial's `QuiescentPeriod` `timestamp`. The last trial has no following quiescent period, so it takes the `EndSession` `timestamp`; `NaN` if that stream is unavailable. |
 | `delay_start_time` | `QuiescentPeriod` `timestamp` — the legacy name for `quiescent_start_time` (see the note below). |
 
 There are no `start_time` / `stop_time` trial columns. NWB's `TimeIntervals`
 requires a native `start_time` / `stop_time` per trial, so the pipeline derives
 the trial extent when writing: `start_time` is `quiescent_start_time` and
-`stop_time` is `ITI_stop_time`, falling back to `ITI_start_time` on the last
-trial.
+`stop_time` is `ITI_stop_time` — which on the last trial is the `EndSession`
+timestamp.
 
 > **`delay` means `quiescent`.** The legacy `delay_*` columns describe the
 > acquisition software's *quiescence period* — the lick-free interval preceding
@@ -231,3 +231,5 @@ These were mapped during exploration but are no longer in scope:
 | 2026-08-17 | The reward-delivery series is now reward-keyed: a delivery whose matched trial reports `is_rewarded=False` is dropped rather than annotated (manual water exempt), so the retained count equals the metadata mapper's `sum(is_rewarded)`. Free water fires at the go cue and the trial then continues normally, so `is_rewarded` describes the animal's own choice, not the free water — meaning free water on an unrewarded trial is real water the animal received and is excluded from this series. The trials table's `auto_water*` / `anti_bias_*` columns remain ungated and still record it. |
 | 2026-08-17 | `auto_waterL` / `auto_waterR` now read `trial.metadata.extra.is_autowater` rather than the `is_auto_reward_right` channel, making them **scheduled autowater only** and mutually exclusive with `anti_bias_left_water` / `anti_bias_right_water`. `is_auto_reward_right` says free water fired and on which side but not what kind; the mechanism is in the metadata. Neither column is gated on `is_rewarded`, since both record what the task did. This is narrower than the legacy `dynamic-foraging-task` column of the same name, which was the ungated channel and predates anti-bias water. |
 | 2026-08-17 | The reward-delivery labels stay `earned` / `auto` / `manual`: free water is `auto` whatever mechanism produced it, so the series does not split scheduled autowater from anti-bias water. That split lives in the trials table. Consequence: the series' `auto` count tracks the channel while `auto_waterL` / `auto_waterR` track `is_autowater`, so the two are not expected to be equal. |
+| 2026-08-20 | `block_max` is now one below `block_length`'s configured maximum, which accounts for the floor applied upstream: a block is a whole number of trials, so the configured bound is never itself reachable. `block_min`, `block_beta`, and the `ITI_*` / `delay_*` bounds are unchanged — those durations are continuous and take no such adjustment. |
+| 2026-08-20 | `bait_left` / `bait_right` now read `trial.metadata.extra.is_left_baited` / `is_right_baited` from the acquisition software instead of being re-derived from `p_reward_left` / `p_reward_right` and the `is_auto_reward_right` channel. The software is the authority on bait state, so the two can disagree — notably a port with `p_reward == 1` is no longer assumed baited. `False` when the trial carries no extra metadata. |

pyproject.toml

@@ -17,7 +17,7 @@ readme = "README.md"
 version = "0.3.2"
 
 dependencies = [
-    "aind-behavior-dynamic-foraging[data] @ git+https://github.com/AllenNeuralDynamics/Aind.Behavior.DynamicForaging.git@v0.0.2rc36",
+    "aind-behavior-dynamic-foraging[data] @ git+https://github.com/AllenNeuralDynamics/Aind.Behavior.DynamicForaging.git@10cec2d64f0770405d5fcee40ec01a88d288a6b0",
     "ipykernel",
 ]
 

src/dynamic_foraging_processing/pipeline/_pipeline.py

@@ -59,10 +59,12 @@
 #: ``TimeIntervals`` requires both, so they are derived here.
 _NWB_START_COLUMN = "quiescent_start_time"
 
-#: Columns NWB's required native ``stop_time`` is taken from, in order of
-#: preference: the end of the ITI, falling back to its start on the last trial
-#: of the session (where the ITI end is unknown).
-_NWB_STOP_COLUMNS = ("ITI_stop_time", "ITI_start_time")
+#: Trials-table column NWB's required native ``stop_time`` is taken from: the
+#: end of the ITI, which on the last trial of the session is the ``EndSession``
+#: timestamp. It is ``NaN`` only when that stream is unavailable, and the ``NaN``
+#: is propagated rather than substituted, so an unknown trial end reads as
+#: unknown instead of as a shortened trial.
+_NWB_STOP_COLUMN = "ITI_stop_time"
 
 #: Source repository recorded in the ``processing.json`` data process.
 _CODE_URL = "https://github.com/AllenNeuralDynamics/dynamic-foraging-processing"
@@ -236,8 +238,11 @@ def _trial_extent(row: pd.Series) -> t.Tuple[float, float]:
 
         The trials table has no trial start/stop columns of its own, so the
         trial's extent is taken from its period bounds: it starts with the
-        quiescent period and ends with the ITI, falling back to the ITI start on
-        the last trial of the session (whose ITI end is unknown).
+        quiescent period and ends with the ITI. The last trial of the session has
+        no following quiescent period, so its ITI — and therefore its stop time —
+        ends at the ``EndSession`` timestamp. Should that be unavailable the stop
+        time is ``NaN``: an unknown end is reported as unknown rather than
+        substituted with an earlier landmark.
 
         Parameters
         ----------
@@ -247,11 +252,10 @@ def _trial_extent(row: pd.Series) -> t.Tuple[float, float]:
         Returns
         -------
         tuple of (float, float)
-            The trial start and stop time (seconds).
+            The trial start and stop time (seconds); the stop time is ``NaN``
+            where the ITI end is unknown.
         """
-        stops = [row[column] for column in _NWB_STOP_COLUMNS if pd.notnull(row[column])]
-        stop = stops[0] if stops else np.nan
-        return float(row[_NWB_START_COLUMN]), float(stop)
+        return float(row[_NWB_START_COLUMN]), float(row[_NWB_STOP_COLUMN])
 
     @classmethod
     def _add_trials(cls, nwb_file: pynwb.NWBFile, trials: pd.DataFrame) -> None:
@@ -265,7 +269,7 @@ def _add_trials(cls, nwb_file: pynwb.NWBFile, trials: pd.DataFrame) -> None:
         (named ``id``) is replicated as each trial's NWB ``id``. An empty table
         (or one missing the period columns the extent is derived from) is skipped.
         """
-        required = (_NWB_START_COLUMN, *_NWB_STOP_COLUMNS)
+        required = (_NWB_START_COLUMN, _NWB_STOP_COLUMN)
         if trials.empty or any(col not in trials.columns for col in required):
             return
         descriptions = TrialConfig.column_descriptions()