Add hamilton.packaging for reusable, validatable dataflow subgraphs#1665
Open
btwitsshivanshu wants to merge 1 commit into
Open
Add hamilton.packaging for reusable, validatable dataflow subgraphs#1665btwitsshivanshu wants to merge 1 commit into
btwitsshivanshu wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a new
hamilton.packagingmodule that lets a set of nodes from an existingdataflow be bundled into a portable, versioned unit, exported as a JSON manifest,
and re-validated against a different host graph. Today there's no first-class way
to share a subgraph across projects: users copy modules and discover name, type,
and config mismatches only at runtime. This surfaces them as structured,
categorized errors before the driver is built.
Changes
NodePackagedeclaresprovidesandrequiresas name-to-type mappings, alongwith inter-node
dependenciesandconfig_keys.NodePackage.from_graph(...)derives all of that from a live
Driver.graph, typing provides by node output andclassifying each required input as a config key, an internal edge, or an external
requirement;
include_upstream=Truepulls in transitive non-input dependencies.dependency_order()returns provided names dependencies-first, breaking tiesalphabetically.
PackageManifestis the serialization layer over the sixMANIFEST_FIELDS, withto_dict/to_json/from_dict/from_jsonround-tripping, sorted output, andtypes rendered through
hamilton.htypes.get_type_as_string.export_packageandimport_manifesthandle file I/O.validate_package(package, graph, minimum_version=1)returns aValidationReportof
ValidationProblems spanningVALIDATION_CATEGORIES:version_too_low,node_collision,dependency_type_mismatch,missing_dependency, andconfig_requirement_missing. Type checks reuse Hamilton's existing subclass rule,so widening passes (
boolfeeding anintrequirement) but narrowing does not.find_conflicts(first, second)performs order-independent pairwise detection overCONFLICT_CATEGORIES(duplicate_node,requirement_mismatch,requirement_disagreement), andcompose_packages(packages, name)checks all pairsbefore merging provides, dependencies, and config keys, internalizing any requirement
satisfied within the set. Conflicts raise
PackageConflictErrorcarrying all of them.diff_manifests(old, new)reports added, removed, and changed nodes and requirementsplus config-key deltas, for reviewing what a version bump actually changed.
Errors hang off a
PackagingErrorbase:PackageValidationErrorcarries.report,PackageConflictErrorcarries.conflicts, and both stringify every underlyingmessage rather than just the first.
On the driver side,
Builder.with_packages(*packages)registers packages, bothbuild()andasync_driver'sbuild_without_init()validate them against the builtgraph,
Driver.list_packages()andDriver.validate_packages()expose them, andBuilder.copy()carries them forward.How I tested this
New unit tests live in
tests/test_pkg.py. Constructor coverage exercises emptyand non-string names, non-integer and boolean versions, string values where types are
expected, a name appearing in both
providesandrequires, bare-stringconfig_keys,and single-pass iterables for both
config_keysanddependencies.from_graphistested against real driver graphs: it rejects
HamiltonGraphand plain dicts, rejectsunknown, input-node, empty, and non-string selections, accepts generators, correctly
classifies config keys versus internal edges versus external requirements, ignores
optional parameters, and handles
include_upstreamacross shared ancestors and deepchains without recursing.
Manifest tests cover sort stability,
get_type_as_stringrendering, dict and JSON roundtrips, unknown and missing fields, malformed JSON, semantically invalid dependencies on
import, hand-written manifests, and cycle detection on both packages and manifests.
Validation tests hit every category in
VALIDATION_CATEGORIESand pin the direction ofthe subclass rule, along with generic aliases,
typing.Anyon either side,config-satisfied requirements, version boundary equality, and aggregate multi-problem
reports.
Conflict and composition tests assert every
CONFLICT_CATEGORIESentry in both argumentorders, merge semantics, internal satisfaction honoring subtypes, version defaulting to
the highest member, all-pairs checking even when a requirement is internally satisfied,
and identical serialized output across every input permutation. Error tests check the
hierarchy, the
.reportand.conflictspayloads, and that every underlying messageappears in both
str()andrepr(). Driver integration tests coverwith_packagesvarargs and accumulation, rejection of non-packages, validation on both build paths,
copy()isolation, and requirements satisfied across multiple modules.Notes
Fully additive — existing driver behavior is unchanged when no packages are registered.
Type compatibility deliberately delegates to Hamilton's existing subclass rule rather
than introducing a second notion of type equality. Manifests are pinned by
MANIFEST_FIELDS, so adding a field later will need a compat path infrom_dict.Checklist