Skip to content

Contrib data handling - #2117

Draft
bfoley12 wants to merge 258 commits into
devfrom
contrib-data-handling
Draft

Contrib data handling#2117
bfoley12 wants to merge 258 commits into
devfrom
contrib-data-handling

Conversation

@bfoley12

@bfoley12 bfoley12 commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

Reworks how Contribution.data is parsed, validated, and stored, and implements throughout CRUD paths. Also introduces pivoting a single contribution into multiple based on the presence of conditions in the data field, SI unit canonicalization, bulk-updates of Contributions (currently only used to efficiently mark many contributions as public), and string-normalization types.

Major changes:

Domain changes

  • domains/contributions/data.py: Contribution.data parses annotated keys of the form: "col_name (unit, cond1=a, cond2=b)" into properly formatted data entries.
    • "properly formatted" means we store:
      • original unit, value, error,
      • the SI converted unit, value, and error
      • precision (sig figs)
  • domains/contributions/pivot.py: expands one contribution whose data keys carry conditions into multiple contributions, keyed by condition_key
  • domains/contributions/stats.py: used to derive Project.stats and Project.columns from contributions
  • domains/_shared/units.py: SI unit normalization (Pint) and error support (uncertainties)

Model changes

  • Contribution: added condition_key as an identifier, removed needs_build, server generates _id
  • Structure.Lattice.matrix: switched from PolarsFrame to LatticeMatrix (3x3 matrix) validation to skip polars-frame encoding
  • Project: stats and columns are now derived by the server from Contributions rather than supplied, is_approved is now admin-only
  • MPContribSettings: defining controls on model relations (max contribution.data depth, max_columns, max_components, float_precision)

bfoley12 added 30 commits June 17, 2026 13:38
…rom query. Users can now request partial documents with _fields
…xistence against the Document, unless field is a dict
…w sets last_modified on any modification result
…ponent attachments. Added project-level claude.md. Added just file for development ease.
bfoley12 added 30 commits July 23, 2026 15:27
…orms. Increased enforcement of Project management business logic
… parse scalar numbers to leaves, while leaving legacy Contributions untouched
…entity and sparse_always, and _all (all fields)
…hich right now only allows setting of is_public
…d inlined logic since it was called in one place
…nto body

this is the proper way of handling defaults defined by class methods
…fiy vs Contribution(Patch)Data

Also consolidataed leaf-parsing logic to within the class
…ionFilter and removed ContributionBulkUpdate

Bulk updates to contributions now permit all fields to be set rather than just 'is_public'
…oes not default to a dict overwrite

updating Contribution.data appends/updates specific fields rather than overwriting the entire dict. Full replacements can still be performed by passing 'replace_data=true'
…tructive modification of Contribution.data in bulk and single patch cases
Primarily focused on additive Contribution.data logic
…data

validator logic now lives in data.py:ContributionStoredData rather than directly in models
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant