Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions src/iceberg/file_io.h
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,24 @@ class ICEBERG_EXPORT FileIO {
virtual Status DeleteFile(const std::string& file_location) {
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why change this line?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed this only to avoid the unused parameter warning in the default implementation, since file_location is not used when returning NotImplemented.

I thought it was a small cleanup while touching this area, but I agree it is not required for this PR. I can revert it if you prefer.

return NotImplemented("DeleteFile not implemented");
}

/// \brief Delete files at the given locations.
///
/// Implementations that can delete multiple files efficiently should override this
/// method. The default implementation deletes files sequentially using DeleteFile
/// and returns the first error encountered.
///
/// \param file_locations The locations of the files to delete.
/// \return void if all deletes succeeded, an error code if any delete failed.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// \return void if all deletes succeeded, an error code if any delete failed.
/// \return void if all deletes succeed, or an error code if any delete fails.

virtual Status DeleteFiles(std::span<const std::string> file_locations) {
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be better if you could make some changes that utilize this new API. I saw you mentioned ExpireSnapshots.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, that makes sense. I originally planned to split this into two PRs: first add the FileIO::DeleteFiles API, then update ExpireSnapshots to use it.

I agree that this PR would be more useful if it also adopted the new API. I’ll take a look at updating ExpireSnapshots to use DeleteFiles for grouped file cleanup.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be complete, let's override this function in arrow_io_internal.h by adapting to https://github.com/apache/arrow/blob/eab3d11b484df5bb6ddd75630f9bfa17c0a3f491/cpp/src/arrow/filesystem/filesystem.h#L274

A relevant question is whether to use const std::vector<std::string>& as the parameter so no conversion is needed to call the Arrow equivalent.

for (const auto& file_location : file_locations) {
auto status = DeleteFile(file_location);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: put the implementation to file_io.cc and use ICEBERG_RETURN_UNEXPECTED to save some lines.

if (!status.has_value()) {
return status;
}
}
return {};
}
};

} // namespace iceberg
1 change: 1 addition & 0 deletions src/iceberg/test/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ add_iceberg_test(util_test
data_file_set_test.cc
decimal_test.cc
endian_test.cc
file_io_test.cc
formatter_test.cc
lazy_test.cc
location_util_test.cc
Expand Down
72 changes: 72 additions & 0 deletions src/iceberg/test/file_io_test.cc
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

#include "iceberg/file_io.h"

#include <string>
#include <utility>
#include <vector>

#include <gtest/gtest.h>

#include "iceberg/test/matchers.h"

namespace {
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need this or its counterpart. If you want to wrap RecordingFileIO in an anonymous namespace, you should shrink the scope instead.

Suggested change
namespace {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. I’ll shrink the anonymous namespace scope to cover only the helper RecordingFileIO and keep the tests outside of the iceberg namespace.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not putting all cases in the iceberg namespace so we don't need to use iceberg:: prefix everywhere?


class RecordingFileIO : public iceberg::FileIO {
public:
explicit RecordingFileIO(std::string failure_path = "")
: failure_path_(std::move(failure_path)) {}

iceberg::Status DeleteFile(const std::string& file_location) override {
deleted_paths.push_back(file_location);
if (file_location == failure_path_) {
return iceberg::IOError("failed to delete {}", file_location);
}
return {};
}

std::vector<std::string> deleted_paths;

private:
std::string failure_path_;
};

} // namespace

TEST(FileIOTest, DeleteFilesFallsBackToDeleteFileForEachPath) {
RecordingFileIO file_io;
std::vector<std::string> paths = {"file-a.avro", "file-b.avro"};

EXPECT_THAT(file_io.DeleteFiles(paths), iceberg::IsOk());
EXPECT_THAT(file_io.deleted_paths,
::testing::ElementsAre("file-a.avro", "file-b.avro"));
}

TEST(FileIOTest, DeleteFilesReturnsFirstDeleteFileError) {
RecordingFileIO file_io("file-b.avro");
std::vector<std::string> paths = {"file-a.avro", "file-b.avro", "file-c.avro"};

auto status = file_io.DeleteFiles(paths);

EXPECT_THAT(status, iceberg::IsError(iceberg::ErrorKind::kIOError));
EXPECT_THAT(status, iceberg::HasErrorMessage("failed to delete file-b.avro"));
EXPECT_THAT(file_io.deleted_paths,
::testing::ElementsAre("file-a.avro", "file-b.avro"));
}
1 change: 1 addition & 0 deletions src/iceberg/test/meson.build
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ iceberg_tests = {
'data_file_set_test.cc',
'decimal_test.cc',
'endian_test.cc',
'file_io_test.cc',
'formatter_test.cc',
'lazy_test.cc',
'location_util_test.cc',
Expand Down
18 changes: 7 additions & 11 deletions src/iceberg/update/expire_snapshots.cc
Original file line number Diff line number Diff line change
Expand Up @@ -75,26 +75,22 @@ class FileCleanupStrategy {
CleanupLevel level) = 0;

protected:
/// \brief Delete a single file
void DeleteFile(const std::string& path) {
/// \brief Delete files at the given locations.
void DeleteFiles(const std::unordered_set<std::string>& paths) {
try {
if (delete_func_) {
delete_func_(path);
for (const auto& path : paths) {
delete_func_(path);
}
} else {
std::ignore = file_io_->DeleteFile(path);
std::vector<std::string> path_list(paths.begin(), paths.end());
std::ignore = file_io_->DeleteFiles(path_list);
}
} catch (...) {
/// TODO(shangxinli): add retry
}
}

/// TODO(shangxinli): Add bulk deletion
void DeleteFiles(const std::unordered_set<std::string>& paths) {
for (const auto& path : paths) {
DeleteFile(path);
}
}

bool HasAnyStatisticsFiles(const TableMetadata& metadata) const {
return !metadata.statistics.empty() || !metadata.partition_statistics.empty();
}
Expand Down
Loading