Repository navigation
fix(data): pass non-string column labels to PreTab by position - #512
Merged
Merged
Conversation
PreTab detects column types by label but hands the labels to ColumnTransformer, which treats integer specifiers as positions. Integer labels that are not 0..n-1 in order (e.g. a headerless frame after drop(columns=0)) therefore crashed fit() or matched columns with the wrong preprocessing. DeepTab already treats non-string labels positionally (feature_names_in_ is only recorded for string names), so relabel such frames 0..n-1 in column order before they reach the preprocessor, for fitting and for every transform. Fixes #507
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #507
Problem
PreTab detects column types by label but passes the labels to
ColumnTransformer, which treats integer specifiers as positions. A DataFrame with integer labels other than0..n-1in order therefore crashedfit(), or matched columns with the wrong preprocessing. The common case is a headerless CSV afterdrop(columns=0).Fix
deeptab/data/datamodule.pyadds_positional_columns(). DataFrames whose labels aren't all strings, and aren't already0..n-1in order, are relabelled0..n-1in column order before PreTab sees them, both for fitting and for every transform. This matches how DeepTab already treats such frames:feature_names_in_is only recorded for string names, and prediction input is checked by column count. String-named frames and NumPy input are unchanged.Tests
tests/data/test_datamodule.py::test_positional_columns_relabels_only_non_positional_integer_labelstests/integration/test_estimator_input_validation.py::TestIntegerColumnLabels: a frame afterdrop(columns=0), and reordered integer labels with a string categorical column. Both give the same categories and predictions as the same data with string column names.All of these fail on
mainand pass with this change.Docs
docs/core_concepts/sklearn_api.md("Accepted Inputs") now says that a DataFrame whose column labels aren't all strings is treated like an array: its columns are used by position, andfeature_names_in_is not set.Checks
just test: 3243 passed, 46 skipped, 28 xfailed; coverage 91.30% (re-run after the docs commit)just docs: build succeeded (warnings treated as errors)just check: all hooks passed (ruff, prettier, pyright)