Add an API reference to the documentation

[vahid-ahmadi]

The documentation has no API page. For a JOSS submission that is the largest documentation gap — a reviewer is asked directly whether the functionality is documented.

docs/api.md lists every weighted estimator and weight-handling method on MicroSeries and MicroDataFrame, with signature and a one-line description taken from the docstrings, grouped as:

• MicroSeries — inequality and distribution (gini, the top/bottom share family, t10_b50), ranking (decile_rank and friends), variance from replicate weights, and weight handling
• MicroDataFrame — poverty (poverty_rate, poverty_gap, the deep and squared variants), and weight handling
• Module-level — replicate_variance, replicate_standard_error

26 methods in total. I checked every documented name against the installed package, so the page cannot drift from the API without the check failing:

documented methods: 26
missing from API: []

The page closes with the estimator conventions a user needs in order to interpret the numbers — inverse-CDF quantiles checkable against survey::svyquantile, frequency-weight variance, and proportional splitting at top-share cutoffs. These are the decisions the paper argues are the package's substance, and they were documented nowhere a user would find them.

Registered in both docs/myst.yml and docs/_toc.yml. The two configs duplicate each other and should be reconciled, but that is a separate change; until then, leaving one out would silently drop the page from whichever builder is used.

I could not build the docs locally — no Node — so the documentation job on this PR is the check that matters.

[juaristi22]

Two things before this merges, one of them a versioning issue.

1. The fragment releases 1.6.0. api-reference.added.md maps to a minor bump in bump_version.py. I ran it on this branch and it produces 1.5.2 -> 1.6.0. A docs page should be a patch, so rename the fragment to api-reference.changed.md.

2. The page omits the core weighted estimators. I introspected the installed package against docs/api.md. It documents 24 methods but leaves out 23 public names on MicroSeries and 13 on MicroDataFrame, including the ones the README's "Key Features" promises and the closing "estimator conventions" note explains:

• MicroSeries: sum, count, mean, median, quantile, var, std, cov, corr, rank, cumsum, groupby, astype, clip, round, repeat, sqrt, values, to_numpy, copy, equals
• MicroDataFrame: sum, cov, corr, groupby, merge, reset_index, drop, astype, copy, equals

(scalar_function, vector_function, override_df_functions, catch_series_relapse and get_args_as_micro_series are also public names but read as internals; fine to leave out.) The intro says the page covers weighted estimators and weight-preserving operations, and those two lists are the core of both. The "missing from API" check only runs one way, which is why it passed.

Smaller:

• quintile_rank, quartile_rank and percentile_rank have empty description cells because they have no docstrings. Better to add the docstrings in the source than to hand-write the cells.
• set_weights renders weights: <built-in function array> because the parameter is annotated np.array in microseries.py. np.ndarray fixes both the page and the annotation.

The conventions note is accurate. I checked the frequency-weight variance and the inverse-CDF median numerically against the docstrings, and the example on the page runs. Agree the _toc.yml / _config.yml reconciliation is separate; both are jupyter-book v1 leftovers, and _config.yml still points at PSLmodels/microdf.

[vahid-ahmadi]

All four taken. Thank you — the coverage gap was the important one, and you are right about why it slipped through.

Fragment. Renamed to api-reference.changed.md. Confirmed with bump_version.py on this branch: Version: 1.5.2 -> 1.5.3 (patch).

Missing estimators. Added two sections to each class — Weighted aggregation and Weight-preserving operations — covering everything on your lists. The page goes from 26 documented methods to 59.

The one-way check. Made it a test rather than a script I run once: microdf/tests/test_docs_api_coverage.py asserts in both directions, so a new public method that is not documented now fails CI. It scopes to names each class defines or overrides itself, since anything inherited unchanged from pandas is pandas' to document. scalar_function, vector_function, override_df_functions, catch_series_relapse and get_args_as_micro_series are excluded as internals, as you suggested.

Docstrings and the annotation. Added docstrings to quintile_rank, quartile_rank and percentile_rank in microseries.py rather than hand-writing the cells, and changed set_weights from np.array — a function — to np.ndarray. The signature now renders properly.

Where a pandas docstring described the unweighted behaviour and said nothing about weights, I wrote the description cell rather than inheriting a misleading first line; cumsum, merge, reset_index and drop are the notable ones.

Full suite passes, 837 tests. Noting your _config.yml point — it still names PSLmodels/microdf — for the separate docs-config change.

[vahid-ahmadi]

@juaristi22 all four points are addressed in 9c0a9cd — fragment renamed to .changed (1.5.2 -> 1.5.3 (patch) confirmed), the weighted aggregation and weight-preserving methods added (26 → 59 documented), the coverage check turned into a two-way test, and the three docstrings plus the np.ndarray annotation fixed in source.

Re-review when you have a moment. The two-way check is the part worth a second look — I scoped it to names each class defines or overrides itself, on the reasoning that anything inherited unchanged from pandas is pandas' to document, and excluded the five internals you listed. If you would rather it covered inherited names too, that is a different and much longer page, and I would rather agree the scope than guess it.

All 8 checks green, including the documentation build.

[juaristi22]

Second pass on 4eb1082. The fragment, the coverage test and the page structure check out; three of the new descriptions do not.

What holds up. bump_version.py on this branch gives 1.5.2 -> 1.5.3 (patch). The two-way test is real: adding an undocumented method to MicroSeries and adding a bogus row to api.md each fail the right assertion. Full suite passes locally (837). I built the page with mystmd: all eleven tables render, and the escaped pipe in the DataFrame cov row comes out as one cell with a literal pipe. On scope: own and overridden names only, with the five internals excluded, is the right scope. Inherited pandas names are pandas' to document.

Blocking.

1. MicroDataFrame.cov and corr are not weighted, but the rows say they are. The overrides call pandas' cov / corr on the plain frame, and the source comment says as much ("Column summaries have no observation weights"). Numerically, with x=[1,2,3,4], y=[1,4,2,8], w=[1,1,1,5]: MicroDataFrame.cov() gives 3.1667, the same as unweighted pandas, while the frequency-replicated sample gives 3.1786, which is what m.x.cov(m.y) returns. corr() is 0.792 against 0.896 replicated. The rows sit under "Weighted aggregation" and read "Pairwise frequency-weighted covariance/correlation of the columns." Either say plainly that these return unweighted pandas results, or move them out of that section. Making them weighted is a code change, filed as #327.

2. equals compares weights on both classes. Both implementations return equal_values and equal_weights. The rows say "weights are not part of the comparison", which is the opposite.

3. The set_weights row still reads <built-in function array>. The annotation is fixed in source but the page row was not regenerated, so the rendered page still shows the artifact. The new quantile row has the same artifact from q: np.array at microseries.py:596, which is unchanged.

Smaller.

• values is a property, so the () signature cell is misleading. Mark it as an attribute or leave the cell blank.
• The test ships in both the sdist and the wheel (include = ["microdf*"] picks up microdf.tests), but docs/ ships in neither, and with the file absent the test fails rather than skips (two failures). A module-level pytest.skip when DOCS is missing covers anyone running the packaged tests.
• inspect is imported but unused, and public_methods has an unused base parameter. Lint only runs the formatters, so nothing catches these.
• The comments cite 9c0a9cd, which is not on the remote; the head is 4eb1082.

[vahid-ahmadi]

All three blocking points were real errors of mine, and I reproduced each before fixing. Head is now 7f3f9a0 — and you were right that my earlier comment cited a hash that never reached the remote.

1. cov and corr on the frame. Confirmed with your figures: w=[1,1,1,5] gives 3.1667 from MicroDataFrame.cov(), identical to unweighted pandas, against 3.1786 from both the replicated sample and MicroSeries.cov(); corr is 0.7924 against 0.8963. I wrote those two description cells by hand on the assumption they matched the MicroSeries versions, and did not check. They are now out of the weighted table, marked unweighted, under a warning box pointing at #327 and directing readers to the MicroSeries methods for weighted values.

2. equals. Both implementations return equal_values and equal_weights, and a.equals(b) is False for identical values with different weights. The rows said the opposite; now "True when both the values and the weights are equal."

3. The rendering artifacts. You are right that I fixed the annotation and never regenerated the page. quantile had the same problem from q: np.array at microseries.py:596, which I had missed entirely. Both are np.ndarray in source now — also the two :type weights: lines and the MicroSeries.__init__ and microdataframe.py annotations — and the rows are regenerated from the live signatures.

Smaller. values now renders as attribute with no signature cell. The coverage test carries a module-level skipif on DOCS.exists(), so the packaged tests skip rather than fail. Unused inspect import and the base parameter removed; ruff check --select F is clean on the file.

Thank you for the second pass — three hand-written description cells asserting the opposite of what the code does is exactly the failure mode a generated page invites, and I would not have caught the cov one by reading.

[vahid-ahmadi]

Audited every remaining description on the page that I wrote by hand rather than took from a docstring, on the grounds that all three errors you found were in exactly those cells. e6df79a.

One more was wrong: cumsum. It returns a plain pandas.Series and warns that the weights have already been applied and cannot be reused — so it is the one method in the Weight-preserving operations section that does not preserve them, and my cell described it as a weighted cumulative sum without saying it drops them. The row now says so explicitly.

The rest hold, checked by running them: clip, round, sqrt, copy and astype carry weights through; repeat repeats the weights alongside the values ([3,4] repeated twice gives [3,3,4,4]); merge, reset_index and drop keep them aligned, and drop(index=[0]) drops the matching weight; groupby returns MicroSeriesGroupBy and MicroDataFrameGroupBy.

And they are now a test rather than something I ran once. test_documented_weight_behaviour_holds pins the claims no docstring enforces: the frame's cov/corr matching unweighted pandas, MicroSeries.cov matching the frequency-replicated sample, equals comparing weights, cumsum dropping them, and repeat carrying them. If any of those behaviours changes — #327 being the obvious candidate — the test fails and the page gets updated with the code.

838 tests pass, formatter clean.

[juaristi22]

Third pass on e6df79a. Every round-2 item is fixed and holds up; one item from round 1 is still open.

Verified. The DataFrame cov and corr rows sit outside the weighted table under a warning box that says unweighted and links #327. The equals rows now say weights are compared. The set_weights and quantile rows are regenerated and show numpy.ndarray, and no np.array annotation remains in the source. values renders as an attribute. The coverage test skips when docs/api.md is absent (3 skipped with the file moved aside), and ruff check --select F is clean on it. The cumsum correction is right. 838 pass locally, the page builds with mystmd, and the bump is a patch.

Still open: the three rank rows are empty. quintile_rank, quartile_rank and percentile_rank got docstrings in round 1, but their description cells on the page were never regenerated and are still blank:

| `quintile_rank` | `() -> 'MicroSeries'` |  |
| `quartile_rank` | `() -> 'MicroSeries'` |  |
| `percentile_rank` | `() -> 'MicroSeries'` |  |

Same miss as the set_weights row in round 2: the source changed, the page did not. One regeneration fixes all three.

Coordination with #330. test_documented_weight_behaviour_holds pins the frame's cov and corr to unweighted pandas. I trial-merged this branch with #330: the merge is textually clean, but that assertion fails on the merged tree (3.1786 against 3.1667), as you predicted. Whichever lands second has to flip the assertion and rewrite the two rows and the warning box. The clean order is this PR first, since it is accurate about main today; I will then rebase #330 and update the page and the test there.

[vahid-ahmadi]

Fixed, and I have stopped the class of error rather than just this instance. 552dc4c.

The three rank rows. Regenerated from the docstrings — quintile, quartile and percentile now read "Calculate weighted quintile/quartile/percentile ranks". You are right that it is the same miss as set_weights: three rounds running I changed the source and hand-patched the page instead of regenerating it, which is precisely how the cell and the code drift apart.

So the page is now held to the code by two tests.

• test_no_row_is_missing_its_description fails on any blank description cell.
• test_signatures_match_the_live_ones compares every rendered signature against inspect.signature, attributing each row to the class whose ## heading it falls under, since sum, astype, copy, equals and set_weights all exist on both.

The second test immediately caught a real error I had introduced and you had not seen yet: the MicroDataFrame.set_weights row was carrying the MicroSeries signature, because my round-2 regex fix matched both rows and overwrote the wrong one. Regenerated. That is the fourth instance of the same root cause, and the first one found by CI rather than by you.

840 pass, formatter clean.

On merge order — agreed, and your way round rather than mine. You are right that this branch is accurate about main as it stands, so it should land first; I had it backwards in my #330 comment. Once this merges, over to you to rebase #330 and flip the two rows, the warning box and the cov/corr assertion in test_documented_weight_behaviour_holds. Ping me if you would rather I did that part, since the page and the test are mine.

[juaristi22]

Fourth pass on 552dc4c. The rank rows and the MicroDataFrame.set_weights row are right, and the blank-description test is sound. On Python 3.9 with pandas 2.3.3 the full suite passes at 840, which is the environment behind "840 pass". CI is red, though, and the new signature test cannot go green across the matrix as written.

Blocking: test_signatures_match_the_live_ones compares annotation reprs as text, and they differ between pandas 2 and pandas 3. The page was rendered under pandas 2, where a Series annotation renders as pandas.core.series.Series; pandas 3 renders the same annotation as pandas.Series. Pandas 3 requires Python 3.11 or newer, so the 3.9 and 3.10 jobs resolve pandas 2 and pass, while 3.11, 3.12 and 3.13 resolve pandas 3 and fail. The 3.13 job failed on exactly seven rows, all this one cause: quantile, cov, corr, rank, cumsum and weight on MicroSeries, and sum on MicroDataFrame. The other four jobs were cancelled after it. Regenerating the page under pandas 3 would move the failure to 3.9 and 3.10 instead.

Compare structure rather than text: parameter names, kinds and defaults from inspect.signature are stable across versions. Leave the annotation text unchecked, or normalise it on both sides before comparing. Forward note: on Python 3.14, not yet in the matrix, Optional[X] also renders as X | None, which takes the mismatch count to 23, so text comparison only gets worse.

Root cause of the four regeneration misses. The generator that produces the rows lives outside the repo and is run by hand. Committing it, for example as docs/build_api.py, makes regeneration one command, and lets the test call the same renderer, which is where any normalisation belongs.

Merge order as agreed: this PR first once green, then I rebase #330 and update the two rows, the warning box and the cov/corr assertion there.

[juaristi22]

Merge order flipped after Vahid's review on #330: #330 lands first. This PR then drops the warning box, relabels cov/corr as frequency-weighted, and flips the assertion in test_documented_weight_behaviour_holds to the replicated-sample value before it merges. My round-4 line saying the reverse is superseded.

[vahid-ahmadi]

@juaristi22 green across the whole matrix now, a37aab0. Both of your points are addressed, and the handover from #330 is done.

The version-fragile comparison is gone, and the generator is committed. docs/build_api.py regenerates the page, and microdf/_docs.py holds one markdown_signature renderer that the generator and the test both import — so normalisation lives in a single place, which is where you said it belonged. It builds the text from parameter names, kinds and defaults, strips module qualifiers so pandas.core.series.Series and pandas.Series both render Series, and collapses Optional[X], X | None and string annotations to one spelling.

Verified the way you framed the problem — same committed page, both pandas majors:

pandas<3  (2.3.1): 5 passed
pandas>=3 (3.0.6): 5 passed

Also run on 3.9, 3.10 and 3.14, the last being where your 23-row X | None breakage would have appeared. And the generator produces a byte-identical page under either pandas major, so a contributor regenerating on any version gets the same file rather than flipping the matrix.

Your root-cause diagnosis was the valuable part. The generator living outside the repo and being run by hand is exactly why the page fell behind the source four separate times, and none of the individual fixes would have stopped a fifth.

#330 handover done. The warning box is gone, cov and corr are back in the weighted aggregation table as frequency-weighted, and the assertion in test_documented_weight_behaviour_holds now checks the replicated sample. I added one more while I was there: each frame cell equals the corresponding MicroSeries value, which is the property your change actually introduced and is stronger than matching a hand-computed number.

On the red build: Test Python 3.12 was #330 arriving in the merge commit, not a defect — the assertion correctly failed and the signature test correctly called the page stale once ddof and method changed type. Lint was docformatter, not ruff: make lint runs both, and the new _docs.py needed rewrapping. My local check had only run ruff, which is why I missed it.

852 pass. Over to you for a fifth look — sorry it took five.

[juaristi22]

Fifth pass on a37aab0. Everything from the earlier rounds is closed, by construction rather than by hand.

Version-stable comparison. One renderer in microdf/_docs.py builds each signature from parameter names, kinds and defaults, strips module qualifiers so pandas 2 and pandas 3 render alike, and collapses the union spellings, including the Python 3.14 form. The generator and the test both import it, so the page and the test cannot disagree. CI is green across the matrix; locally 852 pass on Python 3.9 with pandas 2.3.3 and on Python 3.14 with pandas 3.0.6.

Generator committed. I ran docs/build_api.py under both pandas majors and got a byte-identical page each time. Descriptions come from docstrings, the curated cells sit in one overrides table, and the script refuses to write a row with no description. With the blank-cell test, the miss that recurred four times cannot recur.

#330 handover. Warning box gone, cov and corr back in the weighted table as frequency-weighted, and test_documented_weight_behaviour_holds checks the replicated sample and that each frame cell equals its MicroSeries value. The branch contains main, so no rebase is needed.

Lint is clean on the new files. One observation, not a request: the renderer lives inside the package and ships in the wheel; that is right given the tests ship and import it, and nothing imports it at runtime.

Approving. Thanks for staying with it for five rounds; the page is now held to the code in both directions.