Feature Summary
The Sklearn and Sklearn Training operators vectorize one text column. Text Attribute names a single column, and the generated pipeline hands that column to a CountVectorizer, so a dataset whose text is spread over more than one column cannot be used as it stands.
A review that carries a title and a body, or a record with a description beside a comment, is ordinary. Today the only way through is a Concat upstream, which merges the columns into one string and gives up which column each word came from. That distinction is usually worth keeping: a word in a title and the same word in a body are not the same signal.
Naming several columns is not a workaround either. CountVectorizer takes a flat sequence of documents, so a two-column frame reaches it as two documents rather than as the rows, and it learns a vocabulary of the column names:
>>> CountVectorizer().fit(df[["title", "body"]]).vocabulary_
{'body': 0, 'title': 1}
The estimator then receives two samples against however many labels, and the run ends with Number of labels=15 does not match number of samples=2.
Proposed Solution or Design
Text Attribute names any number of columns, and the generated pipeline gives each its own CountVectorizer, with the outputs concatenated. ColumnTransformer does this and keeps the columns distinguishable, prefixing each feature with the column it came from:
ColumnTransformer([
("short_text", CountVectorizer(), "short_text"),
("long_text", CountVectorizer(), "long_text"),
])The operator's output is unchanged: one row of model_name and model, with the pipeline pickled into the binary column as before. Only the model's own input contract widens, from one column to the named several.
The field becomes a list, and old workflows keep loading if it accepts a single value as one:
@JsonFormat(with = Array(JsonFormat.Feature.ACCEPT_SINGLE_VALUE_AS_ARRAY))
@AutofillAttributeNameList
var text: List[EncodableString]
Tfidf Transformer stays one switch over the whole matrix, which keeps the shape of the generated pipeline as it is:
make_pipeline(CountVectorizer(), TfidfTransformer(), estimator()) # today
make_pipeline(ColumnTransformer([...]), TfidfTransformer(), estimator()) # proposed
Turning Count Vectorizer on still discards the numeric columns, as it does today. Carrying them along needs a way to say which columns are features, which the Advanced Sklearn operators have as Selected Features and these two families do not, so it is a separate change and not proposed here.
A smaller alternative is to concatenate the columns into one string before a single CountVectorizer, which is what a Concat operator does today. It merges the vocabularies, so a word keeps no record of which column it came from, and the model loses a distinction it could have used.
The field is declared on SklearnModelOpDesc, so this reaches all fifty-one operators of the two groups.
Feature Summary
The Sklearn and Sklearn Training operators vectorize one text column.
Text Attributenames a single column, and the generated pipeline hands that column to aCountVectorizer, so a dataset whose text is spread over more than one column cannot be used as it stands.A review that carries a title and a body, or a record with a description beside a comment, is ordinary. Today the only way through is a Concat upstream, which merges the columns into one string and gives up which column each word came from. That distinction is usually worth keeping: a word in a title and the same word in a body are not the same signal.
Naming several columns is not a workaround either.
CountVectorizertakes a flat sequence of documents, so a two-column frame reaches it as two documents rather than as the rows, and it learns a vocabulary of the column names:The estimator then receives two samples against however many labels, and the run ends with
Number of labels=15 does not match number of samples=2.Proposed Solution or Design
Text Attributenames any number of columns, and the generated pipeline gives each its ownCountVectorizer, with the outputs concatenated.ColumnTransformerdoes this and keeps the columns distinguishable, prefixing each feature with the column it came from:The operator's output is unchanged: one row of
model_nameandmodel, with the pipeline pickled into the binary column as before. Only the model's own input contract widens, from one column to the named several.The field becomes a list, and old workflows keep loading if it accepts a single value as one:
Tfidf Transformerstays one switch over the whole matrix, which keeps the shape of the generated pipeline as it is:Turning
Count Vectorizeron still discards the numeric columns, as it does today. Carrying them along needs a way to say which columns are features, which the Advanced Sklearn operators have asSelected Featuresand these two families do not, so it is a separate change and not proposed here.A smaller alternative is to concatenate the columns into one string before a single
CountVectorizer, which is what a Concat operator does today. It merges the vocabularies, so a word keeps no record of which column it came from, and the model loses a distinction it could have used.The field is declared on
SklearnModelOpDesc, so this reaches all fifty-one operators of the two groups.