Float32 utility overflow with allow_zero_probs in Sharrow trip destination

[waddell]

Describe the bug

Sharrow returns float32 utilities. In trip destination, interaction_sample_simulate calls logit.utils_to_probs with allow_zero_probs=True, overflow_protection=False. Finite float32 utilities above approximately 88 overflow during exponentiation, producing infinite exponentiated utilities. The corresponding float64 values complete.

Confirmed at ActivitySim commit 61078c7198b46c4d4ff336caa7678bd814b6fa82 (1.5.1), with Sharrow 2.16.2 and Numba 0.66.0 on Linux. The minimal probability-conversion reproducer does not need Sharrow, regional data, or a simulation.

To reproduce

Run the following in the canonical ActivitySim environment. The temporary replacement of report_bad_choices captures the error without requiring a simulation State or tracing setup; it does not change probability conversion and is restored afterward.

"""Minimal #262 reproducer; run with canonical ActivitySim on PYTHONPATH.

No regional data or simulation is needed. Capturing report_bad_choices only
replaces error reporting; the actual canonical probability conversion runs.
"""

from __future__ import annotations

import json

import numpy as np
import pandas as pd


def replay():
    from activitysim.core import logit

    class ProbabilityFailure(Exception):
        pass

    def report(state, bad_row_map, frame, **kwargs):
        raise ProbabilityFailure(kwargs["msg"])

    original = logit.report_bad_choices
    logit.report_bad_choices = report
    records = {}
    try:
        for dtype in (np.float32, np.float64):
            utilities = pd.DataFrame([[200.0, 199.0, -999.0]], dtype=dtype)
            try:
                with np.errstate(over="ignore"):
                    probabilities = logit.utils_to_probs(
                        None, utilities, allow_zero_probs=True, overflow_protection=False
                    )
                records[np.dtype(dtype).name] = {
                    "status": "completed",
                    "probabilities": probabilities.to_numpy().tolist(),
                }
            except ProbabilityFailure as error:
                records[np.dtype(dtype).name] = {"status": "failed", "error": str(error)}
        empty = logit.utils_to_probs(
            None,
            pd.DataFrame([[-999.0, -999.0]], dtype=np.float32),
            allow_zero_probs=True,
            overflow_protection=False,
        )
        records["all_unavailable"] = empty.to_numpy().tolist()
    finally:
        logit.report_bad_choices = original
    return records


if __name__ == "__main__":
    print(json.dumps(replay(), indent=2))

Actual: float32 reports infinite exponentiated utilities; float64 returns [0.7310585786300049, 0.26894142136999516, 0.0]. The all-unavailable control returns [0.0, 0.0].

Expected behavior

Finite inputs [200, 199, -999] should yield the normalized probabilities above, while all-unavailable rows must retain zero probabilities. Simply enabling row-max shifting must account for that all-unavailable contract.

Additional context and candidate fix

A saved trip-destination failure matrix contained 81 rows with entirely finite utilities (maximum 215.122). Replaying the same already-rounded values in float64 resolves all 81 failures. This identifies exponentiation overflow, not an established Sharrow expression-semantic error.

A narrow benchmark patch immediately after utils_arr = utils.values was:

if allow_zero_probs and not exponentiated and utils_arr.dtype == np.float32:
    utils_arr = utils_arr.astype(np.float64)

With this patch, cold and warm full-population Sharrow runs completed and their four exported tables matched byte-for-byte. This is a candidate for review, not a general solution for arbitrary extreme utilities: float64 can also overflow. A more robust normalization that preserves all-unavailable semantics may be preferable.

The benchmark did not clip utilities or change expressions, coefficients, or sentinels. Reproducer, patch provenance, and results: https://github.com/urbansim/activitysim-torch/pull/271 . Local investigation: https://github.com/urbansim/activitysim-torch/issues/262 .

[jpn--]

Thanks for reporting this scenario. This is a good quality issue report, with a minimal reproducer that demonstrates the concern.

There is relevant consortium history here. In #821, @i-am-sijia expressed concern that automatic protection hides input and specification errors, and the meeting follow-up favored skipping failed households instead. The later product decision in #1052 explicitly confirmed that overflow protection should be disabled when failed-household skipping is enabled.

That leaves two questions to resolve:

1. Are these large utilities valid for the affected model? @waddell, in your comment above you provide additional context via links to an "activitysim-torch" repository that appears to be private. Can you publish more context here in a way we all can see? Please provide a trace or expression-level breakdown for representative failing rows, showing what produces utilities at or above about 38.5, which is where exp(U) will overflow. Finite utilities and successful float64 evaluation establish the numerical failure mechanism, but do not establish that the underlying inputs and expressions are correct and reasonable.

2. Should this path exponentiate float32 utilities in float64 for consistency with legacy evaluation? @ActivitySim/engineering should assess this separately from automatic utility shifting. The proposed promotion preserves the original utility levels and still permits overflow beyond float64’s range, so it could be a reasonable compatibility fix without restoring automatic normalization. On the other hand, upcasting to float64 eats memory, and if the "compatibility" we are enabling is usually a symptom of a bug or poorly specified model rather than desired operation, then we get almost no benefit at some cost.

[stefancoe]

@jpn-- PSRC is experiencing this issue in the park-and-ride lot choice model. We get 0 failed HHs when running without sharrow, but enough to exceed the default limit when running with sharrow, due to large negative utilities that exceed the representable range of float32. We tested changing this line to include .astype(np.float64), which fixed the issue. I’m not suggesting this as a permanent fix, but it does confirm that the issue is related to the float32 representation. @dhensle and I have talked about some options for eliminating the large negative utilities from this model. This might be a good agenda item for an upcoming engineering meeting. Unfortunately, I won’t be able to make tomorrow’s meeting.