-
Notifications
You must be signed in to change notification settings - Fork 106
feat(io): add bulk delete API to FileIO. #659
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -154,6 +154,24 @@ class ICEBERG_EXPORT FileIO { | |||||
| virtual Status DeleteFile(const std::string& file_location) { | ||||||
| return NotImplemented("DeleteFile not implemented"); | ||||||
| } | ||||||
|
|
||||||
| /// \brief Delete files at the given locations. | ||||||
| /// | ||||||
| /// Implementations that can delete multiple files efficiently should override this | ||||||
| /// method. The default implementation deletes files sequentially using DeleteFile | ||||||
| /// and returns the first error encountered. | ||||||
| /// | ||||||
| /// \param file_locations The locations of the files to delete. | ||||||
| /// \return void if all deletes succeeded, an error code if any delete failed. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| virtual Status DeleteFiles(std::span<const std::string> file_locations) { | ||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It would be better if you could make some changes that utilize this new API. I saw you mentioned
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks, that makes sense. I originally planned to split this into two PRs: first add the I agree that this PR would be more useful if it also adopted the new API. I’ll take a look at updating
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. To be complete, let's override this function in A relevant question is whether to use |
||||||
| for (const auto& file_location : file_locations) { | ||||||
| auto status = DeleteFile(file_location); | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit: put the implementation to file_io.cc and use |
||||||
| if (!status.has_value()) { | ||||||
| return status; | ||||||
| } | ||||||
| } | ||||||
| return {}; | ||||||
| } | ||||||
| }; | ||||||
|
|
||||||
| } // namespace iceberg | ||||||
| Original file line number | Diff line number | Diff line change | ||
|---|---|---|---|---|
| @@ -0,0 +1,72 @@ | ||||
| /* | ||||
| * Licensed to the Apache Software Foundation (ASF) under one | ||||
| * or more contributor license agreements. See the NOTICE file | ||||
| * distributed with this work for additional information | ||||
| * regarding copyright ownership. The ASF licenses this file | ||||
| * to you under the Apache License, Version 2.0 (the | ||||
| * "License"); you may not use this file except in compliance | ||||
| * with the License. You may obtain a copy of the License at | ||||
| * | ||||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||||
| * | ||||
| * Unless required by applicable law or agreed to in writing, | ||||
| * software distributed under the License is distributed on an | ||||
| * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||||
| * KIND, either express or implied. See the License for the | ||||
| * specific language governing permissions and limitations | ||||
| * under the License. | ||||
| */ | ||||
|
|
||||
| #include "iceberg/file_io.h" | ||||
|
|
||||
| #include <string> | ||||
| #include <utility> | ||||
| #include <vector> | ||||
|
|
||||
| #include <gtest/gtest.h> | ||||
|
|
||||
| #include "iceberg/test/matchers.h" | ||||
|
|
||||
| namespace { | ||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We don't need this or its counterpart. If you want to wrap
Suggested change
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Makes sense. I’ll shrink the anonymous namespace scope to cover only the helper
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why not putting all cases in the |
||||
|
|
||||
| class RecordingFileIO : public iceberg::FileIO { | ||||
| public: | ||||
| explicit RecordingFileIO(std::string failure_path = "") | ||||
| : failure_path_(std::move(failure_path)) {} | ||||
|
|
||||
| iceberg::Status DeleteFile(const std::string& file_location) override { | ||||
| deleted_paths.push_back(file_location); | ||||
| if (file_location == failure_path_) { | ||||
| return iceberg::IOError("failed to delete {}", file_location); | ||||
| } | ||||
| return {}; | ||||
| } | ||||
|
|
||||
| std::vector<std::string> deleted_paths; | ||||
|
|
||||
| private: | ||||
| std::string failure_path_; | ||||
| }; | ||||
|
|
||||
| } // namespace | ||||
|
|
||||
| TEST(FileIOTest, DeleteFilesFallsBackToDeleteFileForEachPath) { | ||||
| RecordingFileIO file_io; | ||||
| std::vector<std::string> paths = {"file-a.avro", "file-b.avro"}; | ||||
|
|
||||
| EXPECT_THAT(file_io.DeleteFiles(paths), iceberg::IsOk()); | ||||
| EXPECT_THAT(file_io.deleted_paths, | ||||
| ::testing::ElementsAre("file-a.avro", "file-b.avro")); | ||||
| } | ||||
|
|
||||
| TEST(FileIOTest, DeleteFilesReturnsFirstDeleteFileError) { | ||||
| RecordingFileIO file_io("file-b.avro"); | ||||
| std::vector<std::string> paths = {"file-a.avro", "file-b.avro", "file-c.avro"}; | ||||
|
|
||||
| auto status = file_io.DeleteFiles(paths); | ||||
|
|
||||
| EXPECT_THAT(status, iceberg::IsError(iceberg::ErrorKind::kIOError)); | ||||
| EXPECT_THAT(status, iceberg::HasErrorMessage("failed to delete file-b.avro")); | ||||
| EXPECT_THAT(file_io.deleted_paths, | ||||
| ::testing::ElementsAre("file-a.avro", "file-b.avro")); | ||||
| } | ||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why change this line?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I changed this only to avoid the unused parameter warning in the default implementation, since
file_locationis not used when returningNotImplemented.I thought it was a small cleanup while touching this area, but I agree it is not required for this PR. I can revert it if you prefer.