Repository navigation
Rewrite FileSecure in Kotlin with new pipeline architecture - #343
Merged
Merged
Conversation
FileSecure 13 processes files through pipelines of ordered stages: name and date filters, folder/rename transformations from FileSecure, and a Convert stage bringing MediaConverter's ffmpeg/ImageMagick conversions, probe filters and H2 scan cache. Files are then copied or moved, with a single conflict policy (rename or skip), and folder actions run afterwards. Also brings MediaConverter's parallel conversions, progress bars, p/r/q console commands and Windows CPU affinity. The configuration format is new, the README documents it and how to migrate from both tools. Fixes carried over bugs: transformers ran in an unspecified order, extensionless name collisions aborted a mapping, folders beyond maxDepth were moved as files, DeleteIfEmpty could delete the input folder, a converted input could be stranded in the temp folder, and the affinity mask was applied to a pid instead of a process handle. The Docker image now uses the ffmpeg base image, like MediaConverter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ozsCgvJdvHoNqTQzgMJMU
MigrateConfigCommand is a standalone entry point, runnable from the jar or with ./gradlew migrateConfig. It merges any number of old configuration files into one new configuration, validates the result with the new parser and warns about everything that can't be converted exactly. Old transformers are ordered by their intended priority, as the old version stored them in an unordered set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ozsCgvJdvHoNqTQzgMJMU
Jackson's default Path deserializer reads "D:/folder" as a URI with the scheme "D" when not running on Windows, so Windows configurations could not be loaded or migrated elsewhere. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ozsCgvJdvHoNqTQzgMJMU
Many similar mappings, such as one per "<name> - <number>" folder moved to "Whatsapp/<number>", can be replaced by a single mapping and a regex rule. Files matching no rule are skipped by default, and rewritten destinations can't leave the output folder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ozsCgvJdvHoNqTQzgMJMU
Configuration files are now YAML, regexes no longer need their backslashes escaped and comments are native. Files ending with .json are still read as JSON: the YAML parser rejects tab-indented JSON, so it can't be relied on to read JSON files. The converter now writes YAML, and validates the YAML it writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ozsCgvJdvHoNqTQzgMJMU
NoRename restores the original file name, so it is a no-op when no RenameWithDate or RenameWithDatePattern stage runs before it. The converter now drops it in that case and adds a warning; it is kept when a rename stage is present. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ozsCgvJdvHoNqTQzgMJMU
A top-level defaults section is merged into every pipeline not opting out with useDefaults: false: its stages and folder actions come first, scan is merged key by key and other keys apply when the pipeline doesn't set them. YAML is now read through SnakeYAML's composer so anchors, aliases and << merge keys work, anchors can be defined under the ignored templates key. Binding errors keep the line of the faulty node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ozsCgvJdvHoNqTQzgMJMU
A pipeline's mappingTable (input and output base folders, ordered entries of relative paths, optional regex rewrites of keys and values) is expanded on the configuration tree into regular mappings appended after the pipeline's own, once defaults are applied. It can't be set in defaults. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ozsCgvJdvHoNqTQzgMJMU
Moves first try an atomic rename; across file systems, and for COPY and MOVE_WITH_COPY, files are copied to a .<name>.filesecure-partial file in the destination folder, size-checked, then renamed, the source being deleted only afterwards. Converted files are committed the same way. The scan ignores partial files left by an interrupted run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ozsCgvJdvHoNqTQzgMJMU
Each pipeline counts the files found, moved, copied, converted (or that would be, during dry runs and with operation NONE), skipped by stages, skipped because the destination exists, already in place, and errors. A summary is logged at the end of the run, also when stopped with q or by a signal (Log4j's own shutdown hook is disabled so it can still be logged from the application one). Folder actions now report failures to the runner, which logs and counts them and carries on with the other actions and folders. A pipeline failing as a whole counts as one error. Exit codes: 0 success, 1 invalid configuration, 2 errors during the run, 3 stopped with the q command. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ozsCgvJdvHoNqTQzgMJMU
Every ERROR logged during a run is now a counted error and every counted error is logged once at ERROR. Expected situations are logged at DEBUG, problems the run works around at WARN. - Metadata of files that aren't supported images or videos is DEBUG, unreadable metadata of supported formats is WARN, an unset MP4 creation time (1904) is DEBUG - Failing to set the creation date is counted, so it is now ERROR - Geonames failures, probe-check failures and undecodable name dates are WARN, with the path and reason - ffmpeg/ffprobe error lines are no longer logged by Jaffree at ERROR, they are part of the probe warning or conversion error message - Per-file creation date, affinity and cache save lines are DEBUG Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ozsCgvJdvHoNqTQzgMJMU
config.schema.json (draft-07) describes the configuration as written, before defaults and mapping tables are applied, giving autocompletion, hover documentation and validation in IntelliJ and VS Code. Stages, folder actions and probe filters are discriminated by their type with a oneOf of definitions whose type is a const. ConfigSchemaTest checks the schema against the configuration classes (subtypes, constructor parameters, mandatory parameters, enum values). ConfigSchemaValidationTest validates the README examples, the configurations of the tests and invalid configurations with json-schema-validator, a test-only dependency. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ozsCgvJdvHoNqTQzgMJMU
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ozsCgvJdvHoNqTQzgMJMU
Replace the geonames.org web service, reached over plain HTTP with a rate limited account, by the time zone boundaries bundled in timeshape. The data is loaded on the first geotagged file only; if it can't be loaded, or no zone matches, the system time zone is used as before. Remove the --geonames-username option. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ozsCgvJdvHoNqTQzgMJMU
zstd-jni loads a native library: native access is enabled in the jar manifest, the tests and the Docker image. protobuf is raised to 4.36, the first version no longer calling deprecated sun.misc.Unsafe memory methods. timeshape's loading messages are logged at WARN level only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ozsCgvJdvHoNqTQzgMJMU
The new pipeline key verify (SIZE by default, or CHECKSUM) chooses how a file copied to its destination is checked before its final rename. With CHECKSUM, the source is hashed while it is copied, the partial file is forced to storage then read back and both SHA-256 must match; the source dates are copied to the partial file. Atomic renames are not affected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ozsCgvJdvHoNqTQzgMJMU
Keeps the dependency bumps from main (jackson 2.22.4, metadata-extractor 2.22.0, Gradle 9.8.2); picocli and jspecify are no longer used by the Kotlin version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ozsCgvJdvHoNqTQzgMJMU
A pipeline can list more {input, output} mappings in a YAML or JSON file,
added after its own mappings and before its mappingTable. Another tool can
then rewrite that file without touching the main configuration, which keeps
its comments and anchors.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
FileSecure 13 is a rewrite in Kotlin that merges FileSecure and MediaConverter into one tool. Files found in the input folders of a pipeline go through ordered stages that can filter them, change their destination or plan a conversion, then they are converted (if needed) and copied or moved. Pipelines run one after another, so the output of one can feed the next. The goal of this version is feature parity with both tools; the configuration format is new and a converter is provided.
Configuration
.jsonfiles), validated strictly: unknown keys are rejected. A JSON schema (config.schema.json) gives completion and validation in editors.mappings(or amappingTablefor many similar input/output pairs),scanoptions (maxDepth,skipHidden,excludedFolders),stages,operation(COPY,MOVE,MOVE_WITH_COPY,NONE),conflict(RENAMEorSKIP, an existing destination is never overwritten),verify(SIZEby default, orCHECKSUM),setFileCreationDate,tempandfolderActions.defaultsare merged into every pipeline (opt-out withuseDefaults: false), and atemplatessection holds YAML anchors reused with<<merges.MigrateConfigCommand(also./gradlew migrateConfig) converts FileSecure 12 and MediaConverter configuration files to the new format. Several files can be given; they are merged into one configuration, keeping their order.Stages and folder actions
NameFilter,IfOlderThan.FolderCanonical,FolderPerDay,FolderPerWeek,FolderPerMonth,FolderPerYear,FolderUniquePerDay, andRegexPath(regex rewrite of the relative path, replacing long lists of near-identical mappings).Lowercase,NoRename,RenameWithDate,RenameWithDatePattern. Dates come from the file name, then from the file metadata read with metadata-extractor (EXIF, XMP, QuickTime/MP4...). The time zone of geotagged media is resolved offline with timeshape, no network access is needed.Convert: ffmpeg and ImageMagick processors (VIDEO_TO_HEVC_MP4,VIDEO_TO_HEVC_MKV,VIDEO_TO_AV1_MKV,MP3_TO_AAC,AUDIO_TO_OPUS,TIFF_TO_JPG,PHOTO_TO_AVIF,PHOTO_TO_JXL), probe filters (LengthMoreThan,LengthLessThan), and the MediaConverter H2 cache of files not worth converting (existing databases can be reused).DeleteIfEmpty,DeleteIfOlderThan,DeleteChildren.Safe transfers
A
MOVEtries an atomic rename first. Otherwise the file is copied to a hidden.filesecure-partialfile next to the destination, checked, then renamed, and the source is only deleted afterwards. An interrupted run never leaves an incomplete file under its final name, and the source is untouched on failure. The check compares sizes by default; withverify: CHECKSUMthe SHA-256 of the source, computed during the copy, is also compared with the copy read back (opt-in, it rereads every copied file).Running
--dry,--threads(parallel conversions),--workers(parallel scan/probe/copy),--ffmpeg-threads,--ffmpeg,--ffprobe,--magick,--affinity(Windows CPU affinity of conversions),--progress/--no-progress.p(pause),r(resume),q(quit, running conversions are stopped and their temporary files removed); Ctrl+C andSIGTERMbehave likeq.q.Behaviour changes
Listed in the README ("Behaviour changes"): only regular files are processed, stages run in the written order (transformer priorities were never applied), an error on one file no longer stops the others, pipelines run sequentially, the Windows affinity is now actually applied, and a few edge cases of
DeleteIfEmptyand MediaConverter's temporary files are fixed.Build
Kotlin 2.4.21 with coroutines, Clikt instead of picocli, Jackson YAML, kotlin-logging; Lombok and Unirest are removed. The Docker image is built with Jib on an ffmpeg base image.
Testing
167 tests (JUnit), covering configuration parsing and schema consistency, migration of old configurations, stages, safe transfers including cross-file-system moves and checksum failures, conversions with real ffmpeg runs, metadata dates and offline time zones, the run summary and exit codes.
🤖 Generated with Claude Code
https://claude.ai/code/session_019ozsCgvJdvHoNqTQzgMJMU