Retry dataset extraction after a failed fetch - #131
Open
vishesh9131 wants to merge 1 commit into
Open
vishesh9131 wants to merge 1 commit into
vishesh9131 wants to merge 1 commit into
Conversation
fetch_imagenette and fetch_flickr8k ignored tar extraction errors and only check whether the dataset directory exists, so an interrupted extraction (e.g. disk full) left a partial dataset that was never fetched again. Extract into the download directory, fail on extraction errors, and move the result into place only on success.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of the change
cmake/fetch_imagenette.cmakeandcmake/fetch_flickr8k.cmakeruncmake -E tarwithout checking its result, and decide whether to fetch only by checking whether the dataset directory exists. If extraction is interrupted (I hit this with a full disk:cmake -E tar: error: No space left on device), the script still deletes the archive and prints "done", and every later build skips the fetch because the partial directory exists. I ended up with 12,234 of 13,394 Imagenette files, and the CNN tutorial would silently train on that.MNIST and CIFAR-10 are not affected, since they verify the extracted files' MD5s via
check_files.Fix
COMMAND_ERROR_IS_FATAL ANY, available since CMake 3.19; the project requires 3.28.6).file(RENAME)only after extraction succeeded, then remove the download directory.The resulting layout of
data/is unchanged.Testing
CI builds with
DOWNLOAD_DATASETS=OFF, so I tested the scripts in script mode (cmake -P) against small local archives served viafile://URLs (URL and MD5 substituted), including truncated archives:I also ran a full build with
DOWNLOAD_DATASETS=ON(real downloads, CMake 4.4, Ubuntu 22.04): Imagenette and Flickr8k were fetched and extracted into the same layout as before, the download directories were removed, and all 17 tutorials built.Type Of Change
Related Issues
Development & Code Review
cmake -P cpplint.cmake)