Repository navigation
Conversation
Collaborator
Author
|
Closing this PR because I used the wrong branch name. I'll open a new PR from the correct branch. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #147.
Works towards #160.
weighted_choice/zipf_choice_directwere rebuilding their weight tables on every single generated row (#147). Also, there is a separate, pre-existing bottleneck inChoiceProposerFactory.get_proposers— the SQL step that builds proposers in the first place — dominates total time for high-cardinality columns and was masking the improvement. This PR bundles all three fixes together.Changes
1.
weighted_choice/zipf_choice_directper-row cachingdatafaker/providers.pyweighted_choice(a)previously rebuilt its(values, counts)lists from scratch on every call, andrandom.choices(..., weights=...)redid its own O(n) cumulative-weight setup every time too, even thoughais the same list object across an entiregenerate_data(count)loop.id(a)-keyed cache (DistributionProvider._weighted_choice_cache) so the derived(values, cumulative_counts)are only rebuilt whenais actually a new list, and switchedrandom.choicestocum_weights=(precomputed) instead ofweights=(recomputed every call).zipf_choice_direct:zipf_cumulative_weights(size)isfunctools.cached bysize.Measured (
generate_data(4000),concept_idcolumn, isolated from any SQL changes):zipf_choice [sampled]weighted_choice [sampled]choice [sampled]2. Stop sorting
stmt_countresults that get thrown awaydatafaker/proposers/choice.pyChoiceProposerFactory.get_proposers's exactstmt_countquery hadORDER BY count DESC, sorting the entire table in the database — even though, for any column overMAXIMUM_CHOICES(500) distinct values (the common case for a real column), the result is discarded unseen once the row-count cap check fails.ORDER BY; when the cap check does pass (≤500 rows), the small result is sorted in Python instead, preservingZipfChoiceProposer's most-frequent-first ordering requirement exactly.Measured (
get_proposers,concept_id, a near-unique PK column): ~1.0s → 0.397s.3. Skip the exact query entirely for high-cardinality columns
datafaker/proposers/choice.pyconcept_name),stmt_count's full-tableGROUP BYwas taking 12.6s, confirmed viaEXPLAIN (ANALYZE, BUFFERS). Its result is discarded unseen anyway, same as above.get_proposersnow runs the cheap, boundedSAMPLE_COUNT-row sample query first. If that sample comes back ≥90% distinct (EXACT_SKIP_SAMPLE_DISTINCT_RATIO), the exact query is skipped outright, both unaffordable and pointless in that case.MAXIMUM_CHOICES).Measured (
get_proposers):concept_name(high cardinality, unindexed)concept_id(near-unique PK)Testing
tests/test_proposer.py:ChoiceProposerFactoryOrderingTests— values still come back most-frequent-first after removing the SQL-side sort (verified to fail without the fix).ChoiceProposerFactoryExactSkipTests— the exact query is skipped for a high-cardinality column and still runs for a low-cardinality one (captured via SQLAlchemy'sbefore_cursor_execute; verified to fail without the fix).