Select reversibility heuristics per thermodynamic data source (eQuilibrator gets its own cutoff) - #285
Open
Cooper-Taylor wants to merge 8 commits into
Open
Select reversibility heuristics per thermodynamic data source (eQuilibrator gets its own cutoff)#285Cooper-Taylor wants to merge 8 commits into
Cooper-Taylor wants to merge 8 commits into
Conversation
The direction cascade had exactly one rule set, DEFAULT_HEURISTICS, derived
from the Group-Contribution era and applied to every source. Split it into a
registry keyed by source, with GC as the default for anything without a set of
its own.
GC unchanged Jankowski 2008 cascade -- the default
EQ eQuilibrator: Noor 2012 reversibility index, gated by Beber 2022
uncertainty (new)
EQ2 eQuilibrator 2.0: the same index as a bare point estimate
Three defects in the eQuilibrator path motivated this:
1. `Estimate_Reaction_Reversibility.py EQ` never read eQuilibrator energies.
It pulled the canonical `deltag`, merely gated on eQuilibrator eligibility.
Since the additive-thermodynamics refactor nothing overwrites `deltag`, so
only 1,797 of 25,028 reactions with an eQuilibrator record actually had
`deltag == thermodynamics['eQuilibrator'][0]`; the other 23,140 were scored
on the Group-Contribution number and labelled eQuilibrator. EQ runs now read
the eQuilibrator sublist's own dG and sigma.
2. eQuilibrator's ~1e5 kJ/mol "cannot decompose this reaction" marker was being
consumed as an error bar. 4,933 records carry it; the GC bounds rule cannot
fire that wide, so they fell through to a permissive "=". They now return
"?". Observed real sigma tops out at 65.35 kcal/mol against a marker of
23,900.57, so the cut at 1e4 kJ/mol sits in an empty gap.
3. `Retrieve_eQuilibrator_Reactions_Energies.py` keys its MetaNetX formula on
compound id and so discards compartment, collapsing any species present on
both sides; 1,102 transport reactions carry a dG for a different reaction.
Beber 2022 separately notes the transformed framework needs a
-N_H*RT*ln(10^dpH) - Q*F*dPhi term across a membrane that we never apply.
Transport is now decided structurally (ATPS/ABCT) or returns "?".
The reversibility index eQuilibrator has been computing for us since the table
was first generated -- column 4 of MetaNetX_Reaction_Energies.tbl -- was parsed
and discarded on every run, and `make_ln_reversibility_index_heuristic` was
never invoked. It now decides 8,944 reactions, and the bare no-evidence
`default` fallback drops from 7,120 to zero.
Add_Reaction_Thermodynamics_Operators, _thermo_helpers and Promote_* now pass
the source label through, so each method's stored operator is computed with its
own rule set instead of GC's.
Scripts/Tests/test_eq_heuristics.py verifies the ln(Gamma) implementation
reproduces eQuilibrator's own published ln_reversibility_index on 17,771
reactions; every residual is a MetaNetX-collapsed reaction.
test_reaction_direction.py now distinguishes invariant sources (GC,
dGPredictor -- must match exactly) from intentionally re-scored ones
(eQuilibrator), with --strict to require equality everywhere.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Output of Scripts/Thermodynamics/Rerun_Thermodynamics.sh with the per-source
rule sets. Verified idempotent: a second run reproduces every file byte-for-byte
("Entries refreshed/added: 0").
Per-method operators, vs the previous dev state:
Group contribution 56,002 records 0 changed
eQuilibrator 25,028 records 7,103 changed
dGPredictor 27,715 records 0 changed
dGPredictor-ModelSEED 31,924 records 0 changed
Only eQuilibrator moves. The three columns scored with the untouched GC rule
set are byte-identical, which is what isolates the change to the new cutoff
rather than the refactor.
Canonical reversibility: > 12,670 -> 9,584 (-3,086)
< 2,323 -> 1,603 (-720)
= 15,059 -> 14,002 (-1,057)
? 25,960 -> 30,823 (+4,863)
The database gets less confident, and that is the point: the +4,863 unknowns
track the 4,933 undecomposable records almost exactly. Those had been receiving
a permissive "=" or an inherited direction on the strength of a dG carrying
+/-23,900 kcal/mol -- an unread sentinel, not evidence.
Deciding rule for the EQ run:
Incomplete (no eQuilibrator energy) 24,828
EQ:lnGamma 8,944
EQ:reversible 8,905
Incomplete (GCC, keeps GC answer) 6,146
EQ:undecomposable 4,933
ABCT 1,151
EQ:ambiguous 871
EQ:transport-uncorrected 209
ATPS 15
Empty 10
Canonical deltag is untouched and mostly remains the Group-Contribution value;
reconciling it is Promote_Reaction_Thermodynamics_to_Canonical.py, which is
deliberately not part of Rerun_Thermodynamics.sh and was not run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Write-up of the work behind PR ModelSEED#285, following the convention of the other Papers/NAR_Update_2026/data/ investigation notes. Records what dev was actually doing and how that was established (the eQuilibrator reversibility index was computed, stored in column 4 of MetaNetX_Reaction_Energies.tbl, and discarded unread on every run), the three defects in the eQuilibrator path, and the results of the rerun. Section 3 is the part worth keeping for a methods section: eQuilibrator ships no directionality classifier, Beber 2022 defines no reversibility index and defers directionality to downstream tools, and the table there separates the parts of EQ_HEURISTICS that are faithful reimplementation (the ln Gamma formula, verified against 17,771 published values) from the parts that are ours (the +/-1 sigma margin, the undecomposable and transport gates). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both corrections came from supervisor review of PR ModelSEED#285. The code is unchanged; the characterisation of why it was needed was wrong. 1. "The EQ run never read eQuilibrator energies" was wrong. The design was correct: from c263e23 (2019) through 17c9739 (2023-09-11), Update_Reaction_eQuilibrator_Energies.py overwrote canonical deltag/deltagerr with the eQuilibrator values and stamped the EQU note, so `Estimate_Reaction_Reversibility.py EQ` reading deltag gated on EQU was reading eQuilibrator energies exactly as intended. 3e50646 (2023-09-13) removed the write half -- moving the updater to thermodynamics['eQuilibrator'] only, dropping both the deltag overwrite and the EQU stamp -- and left the read half in place. Later GC rebuilds and the PR ModelSEED#265 promotion then rewrote deltag from other sources. Only 1,586 of the 17,094 reactions still carrying EQU have deltag == the eQuilibrator value. EQU is a fossil of the pre-2023 pipeline, and is distinct from EQC (which the retrieval script writes to mean "all reagents had structures"). 2. "The marker was read as an error bar" understated it. For the residual case GibbsEnergyPredictor.standard_dg returns Q_(0, "kJ/mol") outright -- the computed mean is discarded, not projected. standard_dg_prime then adds the Legendre/pH transform, so the nonzero dG'o stored for those reactions is the transform alone. Of 4,607 sigma-flagged reactions, 2,895 are this zeroed case (61 still exactly 0.000000, having no net proton change) and 1,712 have a real mean with an undetermined covariance direction. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Standalone companion to the heuristics write-up, aimed at a reader who wants to follow the defect end to end rather than the design of the fix. Covers: the pipeline step by step against origin/dev's code with file:line references; what eQuilibrator returns when it cannot decompose a reaction (standard_dg returns Q_(0, "kJ/mol") and standard_dg_prime adds the Legendre transform on top, so the stored number is the transform alone); the 61 reactions still sitting at exactly 0.000000 because they have no net proton change; the three separate breakages introduced by 3e50646 on 2023-09-13 and why each later version preserved them; and why the 2026-07-05 eQuilibrator rerun turned a handful of such records into 4,934. Impact table contrasts dev (0 of 4,934 marked "?", 2,940 given a hard direction) against this branch (4,933 marked "?"), and attributes dev's answers to the specific GC rules that produced them -- 627 directional calls from low_energy_heuristic, which reads the sign of what is only a pH transform. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Third companion document, focused on the defect where the EQ reversibility step
scores canonical deltag rather than the eQuilibrator estimate. Leads with
verification rather than assertion -- four independently reproducible checks:
1. No script in Rerun_Thermodynamics.sh writes canonical deltag. The only
writer is Promote_Reaction_Thermodynamics_to_Canonical.py, which is not in
the rerun. deltag is static data, not pipeline output.
2. Estimate_Reaction_Reversibility.py:74 nonetheless reads it as the energy
source; reversibility_heuristics.py:94 looks up the eQuilibrator pair,
discards it, and returns deltag tagged with eQuilibrator's label.
3. deltag matches no currently-stored source for 49.4% of reactions, and for
EQ-eligible reactions matches the eQuilibrator value only 7.2% of the time.
Canonical deltagerr is never the eQuilibrator sentinel (0 of 4,934).
4. Replaying both energies through the cascade gives different directions for
1,575 reactions where eQuilibrator is confident (sigma <= 5 kcal/mol), e.g.
rxn00003 flips '=' to '<' on a 1.13 kcal/mol input difference.
Traces the cause to 3e50646 (2023-09-13) deleting the writer's half of a
two-sided contract, explains why it failed silently (EQU is persisted data,
deltag stayed populated, the cascade cannot see provenance), and records that no
later commit repaired it.
Also states plainly what is NOT fixed: the GC path reads the wrong energy 79.3%
of the time for the identical structural reason, left alone deliberately so the
byte-identical GC column remains a control for PR ModelSEED#285. And that deltag is
currently neither pipeline output nor a curated field, which needs a decision.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Promotes the GC path from a "not fixed" footnote to a full section, with the impact actually measured rather than inferred from the energy drift. Simulating the full pipeline both ways (GC step then EQ step, in Rerun_Thermodynamics.sh order) and comparing final canonical reversibility: 1,156 reactions differ. But the failure mode is not eQuilibrator's -- only 5 are hard directional flips and 105 turn a confident call into "?", because the stale values are mostly older GC numbers sitting a median of 2.07 kcal/mol from the current ones, close enough that the cascade usually lands the same way. The dominant effect is lost coverage: 843 reactions move from "?" to a confident call. 892 reactions hold a stored Group-contribution energy that is unreachable because canonical deltag is the 10000000 sentinel, so _energy_for returns None and _incomplete_decision emits "?" despite the energy being present. Records why the GC fix cannot be the same one-line swap the EQ fix was: of those 892, three carry |dG| > 1000 kcal/mol and five carry error > 100, and they are currently invisible precisely because Promote_Reaction_Thermodynamics_to_Canonical.py applies MAX_ABS_DG/MAX_ERR before writing deltag. Reading deltag is wrong but protective -- it is accidentally acting as a quality filter. The EQ rule set carries its own gate (eq_undecomposable_heuristic) and so needed no equivalent; the GC cascade has none, since stored_bounds_heuristic abstains on a wide error bar rather than rejecting it. A follow-up PR should swap the GC source, port the plausibility guards into the GC rule set, and report the ~884 recovered reactions as an intended coverage gain. Kept out of PR ModelSEED#285 so the byte-identical GC column remains a control. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Restructured from 343 lines to 197, led by the mental model rather than the evidence. The core is now: every reaction stores energies in two places (a worked rxn01442 record where canonical deltag is visibly dGPredictor's number, not eQuilibrator's), the two-step handoff that used to keep them in sync, and what 3e50646 broke -- with a sticky-note/whiteboard analogy for a reader who does not know the schema. Verification is kept but compressed to the four numbers that carry it: no pipeline script writes deltag, deltag matches no current source for 49.4% of reactions, it is eQuilibrator's value only 7.2% of the time in the EQ path, and 1,575 directions differ. The long code excerpts and per-rule cascade tables are dropped -- they live in the zero-energy companion note. The GC section is retained in full since it carries the finding that changes the follow-up plan: deltag is accidentally acting as a quality filter, so that fix needs a plausibility gate ported from Promote_*, not just a source swap. Also keeps the correction on record: the EQ step read eQuilibrator energies correctly for four years; "never" was wrong. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The reaction-direction cascade had exactly one rule set —
DEFAULT_HEURISTICS, derived from the Group-Contribution era — applied to every thermodynamic source. This splits it into a registry keyed by source:GC(default)=EQ=EQ2GCis the default for any source without a set of its own, so both dGPredictor variants are unchanged. The GC cascade itself is untouched.Why
Three defects in the eQuilibrator path. (Points 1 and 2 were re-characterised after review — see the "Corrected after review" note at the bottom. The code did not change; the explanation of why it was needed was wrong.)
1. The
EQrun stopped reading eQuilibrator energies in 2023. The design was correct. Fromc263e233(2019) through17c9739b(2023-09-11),Update_Reaction_eQuilibrator_Energies.pyoverwrote the canonical energy and stamped the note that gates the reversibility step:So reading
deltaggated onEQUwas reading eQuilibrator energies, exactly as intended.3e50646b(2023-09-13) removed the write half and left the read half in place — the updater moved tothermodynamics['eQuilibrator']only, dropping both thedeltagoverwrite and theEQUstamp. Later GC rebuilds (Convention A,ad34d6ab) and the PR #265 promotion then rewrotedeltagfrom other sources. Today only 1,797 of 25,028 reactions with an eQuilibrator record havedeltag == thermodynamics['eQuilibrator'][0]; restricted to the 17,094 still carryingEQU, only 1,586 do.EQUis now a fossil of the pre-2023 pipeline. Pointing the EQ run at the eQuilibrator sublist restores the original intent rather than inventing a new one.2. For most σ-flagged reactions eQuilibrator returns no energy at all — we stored a transform term as if it were one.
GibbsEnergyPredictor.standard_dgshort-circuits when a component falls outside both the reactant- and group-contribution spans:Literally zero — the computed mean is discarded, not projected.
standard_dg_prime, which our retrieval script calls, then adds the Legendre/pH transform on top, so the nonzero ΔG′° we store is only the transform.rxn00017's −20.46 kcal/mol is the pH-7 transform of an energy eQuilibrator declined to estimate.Of the 4,607 σ-flagged reactions: 2,895 are this zeroed case (σ exactly
RMSE_inf; 61 still exactly0.000000, having no net proton change for the transform to act on — e.g.rxn02677,rxn04059), and 1,712 carryRMSE_inf·‖σ_inf‖and do have a real mean but with an undetermined covariance direction. Neither supports a directional call; the GC bounds rule cannot fire at that width, so all of them fell through to a permissive=.3. Transport energies are compartment-collapsed.
Retrieve_eQuilibrator_Reactions_Energies.pykeys its MetaNetX formula on compound id and discards compartment, so any species on both sides nets out — 1,102 transport reactions carry a ΔG′° for a different reaction (rxn12518at +524 kcal/mol). A further 76 non-transport reactions collapse via stereo-neutral InChIKey matching (rxn00816: D-glucose and galactose merge). Beber 2022 separately notes the transformed framework needs a−N_H·RT·ln(10^ΔpH) − Q·FΔΦterm we never apply.Separately: the reversibility index eQuilibrator has been computing for us — column 4 of
MetaNetX_Reaction_Energies.tbl— was parsed and discarded on every run, andmake_ln_reversibility_index_heuristicwas never invoked. It now decides 8,944 reactions, and the bare no-evidencedefaultfallback drops from 7,120 to zero.Results
Per-method operators vs current
dev:Only eQuilibrator moves. The three columns scored with the untouched GC rule set are byte-identical, which isolates the change to the new cutoff rather than the refactor.
Canonical reversibility:
>12,670→9,584 (−3,086) ·<2,323→1,603 (−720) ·=15,059→14,002 (−1,057) ·?25,960→30,823 (+4,863)The database gets less confident, and that is the point. The +4,863 unknowns track the σ-flagged records almost exactly — reactions that had been getting a permissive
=on the strength of a number that, for 2,895 of them, is eQuilibrator's literal zero plus a pH transform.Verification
Scripts/Tests/test_eq_heuristics.py— the ln Γ implementation reproduces eQuilibrator's own publishedln_reversibility_indexon 17,771 reactions; every residual is a MetaNetX-collapsed reaction, asserted rather than assumed.Scripts/Tests/test_reaction_direction.pynow separates invariant sources (GC, dGPredictor — must match exactly) from intentionally re-scored ones (eQuilibrator), with--strictto require equality everywhere. Passes.Rerun_Thermodynamics.shis idempotent: a second run reproduces every file byte-for-byte.rxn00001→>(ln Γ −9.18) ·rxn00017→?·rxn12518→?·rxn08173ATP synthase →=.Full write-up:
Papers/NAR_Update_2026/data/equilibrator_reversibility_heuristics_2026-08-18.md.Scope notes for review
deltagis untouched and mostly remains the Group-Contribution value. Reconciling it isPromote_Reaction_Thermodynamics_to_Canonical.py, deliberately not part ofRerun_Thermodynamics.shand not run here.>calls are the main review surface: some are the index genuinely disagreeing with the GC bounds rule, others are reactions whose only evidence was the zeroed value. Separable from the generated report on request.is_using_group_contribution()would replace the σ-threshold with eQuilibrator's own verdict;standard_dg_prime_multi()returns the covariance the current per-reaction loop cannot; and the MetaNetX collapse in the retrieval script is fixable at source.Corrected after review
Commit
a86f6adfrevises the framing of defects 1 and 2 following review. The original text claimed the EQ run "never" read eQuilibrator energies (it did, correctly, until 2023) and that the σ marker was "read as an error bar" (understated — the energy itself is a discarded mean replaced by zero). No code changed.🤖 Generated with Claude Code