Fix Gamma/ChiSquared inverse-CDF collapse in the small-shape lower tail - #1157
Open
gaoflow wants to merge 1 commit into
Open
Fix Gamma/ChiSquared inverse-CDF collapse in the small-shape lower tail#1157gaoflow wants to merge 1 commit into
gaoflow wants to merge 1 commit into
Conversation
GammaLowerRegularizedInv (the Cephes igami port behind Gamma.InvCDF and ChiSquared.InvCDF) resolves the quantile by linear-space bisection over [0, big]. For small shape the true lower-tail quantile is far below the resolver's ~1e-15 floor (P^-1(0.1, 1e-6) is ~6e-61), and the Wilson-Hilferty seed underflows negative, so the routine returns a spurious ~1.11e-15 for every small p. The result also contradicts the library's own CDF: CDF(InvCDF(0.1, 1, 1e-6)) came back as 0.0336 instead of 1e-6. Two coupled defects, same root cause (a small value rounded to zero): - GammaLowerRegularized rounded any x <= ~2.22e-15 up to zero via x.AlmostEqual(0.0) and returned P = 0, even though P(0.1, 6e-61) = 1e-6 is a perfectly representable double. Only exact zero should short-circuit; genuine underflow is already caught by the ax guard. - GammaLowerRegularizedInv now solves P(a,x) = p directly for small quantiles, seeding from the leading series inversion x ~ (p*Gamma(a+1))^(1/a) and refining with Newton. GammaLowerRegularized keeps full relative precision as x -> 0, so the residual avoids the 1 - P cancellation the old path relied on. Small-shape quantiles now match mpmath (dps 40) to ~1e-14 and round-trip CDF(InvCDF(p)) = p to ~1e-15; moderate and large shape are unchanged. Adds forward, inverse, distribution-level and oracle-free self-consistency tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Gamma.InvCDFandChiSquared.InvCDFcollapse in the small-shape lower tail, returning a spurious~1.11e-15floor and disagreeing with the library's own CDF:The true quantile of
Gamma(0.1)atp = 1e-6is~6.07e-61(mpmath, dps 40) — the returned value is wrong by ~46 orders of magnitude, and it fails the round-trip identityCDF(InvCDF(p)) == pby ~4 orders, which is enough to call it a bug without any external reference. It degrades with shape:a = 0.5is fine,a = 0.3is off ~36x,a = 0.1is off ~34000x atp = 1e-6.ChiSquaredinherits it for fractional degrees of freedom (dof < ~1.2).Cause
Both entries delegate to
SpecialFunctions.GammaLowerRegularizedInv, a Cephesigamiport that resolves the root by linear-space bisection over[0, big]. For small shape the true quantile is far below the resolver's~1e-15floor, and the Wilson–Hilferty initial guess underflows negative, so the routine floors. There are two coupled defects sharing one root cause — a small value rounded to zero:GammaLowerRegularizedrounded anyx <= ~2.22e-15up to zero (x.AlmostEqual(0.0)) and returnedP = 0, even thoughP(0.1, 6e-61) = 1e-6is a perfectly representable double. Only exact zero should short-circuit; genuine underflow is already caught by the existingaxguard.GammaLowerRegularizedInvnow solvesP(a,x) = pdirectly for small quantiles, seeding from the leading series inversionx ~ (p·Γ(a+1))^(1/a)and refining with Newton.GammaLowerRegularizedkeeps full relative precision asx -> 0, so the residual avoids the1 - Pcancellation the old path relied on. Moderate/large shape still take the original path unchanged.Verification
~1e-14across a grid of shape{0.1 … 20}x probability{1e-9 … 0.999999}.CDF(InvCDF(p)) == pholds to~1e-15, and the inverse is monotone inp.SpecialFunctions+Distributionssuites pass (4719 tests).New tests cover the forward function (
GammaLowerRegularizedSmallArgument), the inverse (GammaLowerRegularizedInvSmallShapeLowerTail, including trigger-boundary cases where the Newton refinement is load-bearing), self-consistency/monotonicity, and both distributions.