[data/preprocessors] concatenator should preserve order of concatenated #47997
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
martinbomio
force-pushed
the
martinbomio/fix-concatenator-order
branch
4 times, most recently
from
531a467
to
b8cf5d6
Compare
martinbomio
requested review from
hongpeng-guo,
justinvyu,
matthewdeng,
raulchen,
woshiyyya,
scottjlee,
bveeramani,
stephanie-wang and
omatthew98
as code owners
anyscalesam
added
triage
richardliaw
changed the title
martinbomio
force-pushed
the
martinbomio/fix-concatenator-order
branch
2 times, most recently
from
2f701df
to
071ad2b
Compare
richardliaw
reviewed
Oct 17, 2024
Comment on lines
62
to
63
| columns: A list of columns to concatenate. The order will define the order | ||
| of the concatenated columns. |
There was a problem hiding this comment.
Suggested change
| columns: A list of columns to concatenate. The order will define the order | |
| of the concatenated columns. | |
| columns: A list of columns to concatenate. The provided order of the columns | |
| will be retained during concatenation. |
There was a problem hiding this comment.
Also - should we just keep it as include?
There was a problem hiding this comment.
ok with keeping it, but just want to point out that it would be still a breaking change, and that other preporcessors use columns to define the columns that the preprocessor would be apply to
There was a problem hiding this comment.
What would be breaking in particular?
There was a problem hiding this comment.
it will be a required argument right now, if non is provided it will fail
There was a problem hiding this comment.
discussed offline - this makes sense and columns is more consistent with other preprocessors as well.
martinbomio
force-pushed
the
martinbomio/fix-concatenator-order
branch
from
071ad2b
to
191af73
Compare
martinbomio
force-pushed
the
martinbomio/fix-concatenator-order
branch
2 times, most recently
from
88acac8
to
a5ec10e
Compare
martinbomio
force-pushed
the
martinbomio/fix-concatenator-order
branch
2 times, most recently
from
351b643
to
a88e22a
Compare
scottjlee
reviewed
Oct 17, 2024
| @@ -20,6 +20,8 @@ class Concatenator(Preprocessor): | |||
| :class:`~ray.air.util.tensor_extensions.pandas.TensorArrayElement` objects of | |||
| shape :math:`(m,)`, where :math:`m` is the number of columns concatenated. | |||
| The :math:`m` concatenated columns are dropped after concatenation. | |||
| The preprocessor preserves the order of the columns provided in the ``colummns`` | |||
| argument and will use that order when on transform, and transform_batch. | |||
There was a problem hiding this comment.
Suggested change
| argument and will use that order when on transform, and transform_batch. | |
| argument and will use that order when calling ``transform()`` and ``transform_batch()``. |
| @@ -20,6 +20,8 @@ class Concatenator(Preprocessor): | |||
| :class:`~ray.air.util.tensor_extensions.pandas.TensorArrayElement` objects of | |||
| shape :math:`(m,)`, where :math:`m` is the number of columns concatenated. | |||
| The :math:`m` concatenated columns are dropped after concatenation. | |||
| The preprocessor preserves the order of the columns provided in the ``colummns`` | |||
There was a problem hiding this comment.
i cannot add a comment to the docstring above this line because it wasn't modified in this PR. but can we also modify this line, to indicate that only columns specified in the columns arg will be included?
Combine numeric columns into a column of type
:class:`~ray.air.util.tensor_extensions.pandas.TensorDtype`.
Signed-off-by: Martin Bomio <[email protected]>
martinbomio
force-pushed
the
martinbomio/fix-concatenator-order
branch
from
a88e22a
to
ddb92f4
Compare
|
@richardliaw @scottjlee adressed all your comments! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Make concatenator preserve order.
This is a breaking change, since the
Concatenatorsince now the expectation is that the list of columns to include is specify always, and that list will define the order of the columns in the output column.Why are these changes needed?
See issue #47996
Related issue number
Closes #47996
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.