Unified Nim library for creating/extracting archives and compress/decompress files
Zero-copy I/O core, walk virtual file system, SIMD-accelerated brotli compression
Supporting .tar, .zip, .rar (5.0/7.0), .gz, .zz, .lz4, .bz2, .br

clue install archive

API reference

Note

This is an LLM-accelerated Nim package ported from mholt/archives (Go). It has diverged substantially from the original: own DEFLATE/gzip/zip/tar codecs in pure Nim with no C bindings, split CompressionFormat/ArchiveFormat enums, no global registry, and a zero-copy I/O core. As elsewhere in nimbase, it is written with heavy LLM assistance — APIs may still change, and contributions (bug reports, test vectors, benchmarks) are welcome.

Features

• Archive and compress across tar, zip, rar (5.0/7.0 read + write), gzip, zlib (.zz), lz4, bzip2 and brotli with one uniform API
• Split CompressionFormat/ArchiveFormat string enums plus a MediaType enum: misuse fails at compile time, no runtime registry
• High-level one-shots: compress/decompress (strings, files, buffers), archiveDir/archiveData, auto-identifying extractFile/extractAll/listEntries
• Own DEFLATE/gzip/zip/tar/crc32, no C bindings, brotli via pkg/nbrotli (SIMD-accelerated internally)
• Encrypted archives via pkg/nimcypher: XChaCha20-Poly1305 envelope with Argon2id passwords or raw keys, streaming with fixed reused buffers
• Zero-copy I/O core powered by std/memfiles: memory-mapped inputs, caller-reused output buffers, borrowed spans — memory-efficient with bounded working sets, no streams anywhere
• Format detection by filename and magic bytes without consuming input, plus tar.gz-style two-layer composition
• Virtual filesystem with uniform reads over dirs, files, archives and compressed archives
• Per-entry progress callbacks plus a cancellable context, threaded through every operation
• Per-block CRCs and stream checksums verified on every decode; corrupt inputs raise descriptive errors
• Interop-tested both directions against system gzip/unzip, bzip2, lz4 and CPython zlib/bz2

Examples

Runnable versions live in examples/ (nim.cfg points at ../src, so no install needed — just clue build). Snippets below: High-level API first (import archive covers types and dispatch too); zero-copy Input/Output plumbing is still there when you need it:

import std/[options, strutils]
import archive

# one-shot compression with per-call level (0 = codec default)
let enc = compress(cfBz2, "hello ".repeat(100), level = 9)
assert decompress(cfBz2, enc) == "hello ".repeat(100)

# directory -> backup.tar.gz -> read back (files memory-mapped lazily,
# nothing is read up front; context is internal unless you pass one)
archiveDir(afTar, "my-docs", "backup.tar.gz", "docs",
  compression = some(cfGz))

# extractAll is an iterator of (archive path, content) pairs
for e in extractAll("backup.tar.gz"):
  if e.name.endsWith(".txt"):
    echo e.name, ": ", e.data.len, " bytes"

Archive scattered files by path (stored as given), or rename on the way in with (disk path, archive path) pairs:

let blob = archivePaths(afTar, @["notes.txt", "src/main.nim"])
let renamed = archivePaths(afTar, @[("run.sh", "bin/run")],
  compression = some(cfGz))

Archive in-memory files straight to disk, as zip or compressed tar:

import archive/files   # filesFromEntries, FsEntry: describe files in code

# entries pair an on-disk path with its path inside the archive;
# here both live in memory via the data field
var ctx: ArchiveContext
let entries = filesFromEntries(ctx, FromFSOptions(), [
  FsEntry(path: "a.txt", meta: FileMeta(name: "a.txt", size: 5,
    isRegular: true), data: "hello"),
  FsEntry(path: "sub/b.txt", meta: FileMeta(name: "b.txt", size: 5,
    isRegular: true), data: "world"),
], ".", "")

# plain zip, then tar compressed with gzip (some(cfGz))
archiveFile(afZip, entries, "out.zip", ctx)
archiveFile(afTar, entries, "out.tar.gz", ctx,
  compression = some(cfGz))

# the one-shots above also work without ctx:
archiveFile(afZip, entries, "out.zip")

Inspect an archive, then extract only what you need:

import archive/io   # Input/Output plumbing for entry bodies

# peek at entry paths without decompressing file contents
for name in listEntries("out.tar.gz"):
  echo name

# auto-identifies the format; the callback runs once per entry,
# so filter inside it and read bodies lazily via info.open()
extractFile("out.tar.gz", proc(c: ArchiveContext; info: FileInfo) =
  if info.isRegular() and info.nameInArchive.endsWith(".txt"):
    echo(info.nameInArchive & ": " & $readAll(info.open()).len & " bytes"))

Or extract the whole tree to disk in one call (creates directories, recreates symlinks, rejects entries escaping the destination):

extractFile("out.tar.gz", "restored")

Encrypt a file with a password (XChaCha20-Poly1305 chunked AEAD via pkg/nimcypher; raw Key32 keys work anywhere a password does):

import archive/crypt
encryptFile("backup.tar.gz", "backup.tar.gz.enc", "correct horse")
decryptFile("backup.tar.gz.enc", "backup.tar.gz", "correct horse")

Same work with reusable buffers instead of fresh strings:

import archive/io   # Input/Output: zero-copy cursors, reusable buffers

# caller-owned buffer reused across calls: no per-call allocation
let dst = newOutput()

# zero-copy in (shares the string), compressed bytes appended to dst
compressInto(cfGz, openInputString("hello ".repeat(100)), dst)
let back = newOutput()

# toInput borrows dst's bytes; decompressed output lands in back
decompressInto(cfGz, toInput(dst), back)

# take moves the bytes out as an exact-size string
assert take(back) == "hello ".repeat(100)

Browse an archive through the virtual filesystem:

import archive/vfs  

for path, e in walkFs("out.tar.gz"):
  echo (if e.isDir: "dir  " else: "file ") & path

Roadmap

• tar (ustar/pax, symlinks, hardlinks, insert/append)
• zip (deflate/store, insert/replace)
• gzip (.gz), zlib (.zz), lz4 (.lz4), bzip2 (.bz2)
• brotli (.br) via nbrotli (decode full, store-only encode, SIMD-accelerated internally)
• Identify by name and magic bytes, CompressedArchive composition (tar.gz, tar.bz2, ...)
• Virtual filesystem (DirFS, ArchiveFS, DeepFS, path helpers)
• Encrypted archives via pkg/nimcypher (XChaCha20-Poly1305 chunked AEAD, envelope around the whole blob): EncryptedArchive composition (archive → compress → encrypt on write, reverse on read), password keys via Argon2id plus raw-key API, streaming file paths with fixed reused buffers, own framing magic + salt + nonce header
• xz (.xz, LZMA2 framing + range coder)
• zstd (.zst)
• snappy/s2 (.sz/.s2, own port)
• lzip (.lz), minlz (.mz)
• rar 5.0/7.0 read + write (Unpack50/70 LZ incl. solid groups and DELTA/E8/E8E9/ARM decode; own m1..m5 LZ encoder with STORE fallback and opt-in E8E9/DELTA/ARM filters; solid-group write; AES-256-CBC per-file + header encryption via PBKDF2-HMAC-SHA256; multivolume read/write incl. solid and header-encrypted sets; archive comments (CMT); Unpack70 80-symbol distance path; recovery records skipped)
• 7z read + write (container + STORE/LZMA/LZMA2 + Delta/BCJ-x86 + 7zAES read; STORE + LZMA2-solid fast-level write incl. AES-encrypted solid, header-encrypted (-mhe) write, Delta auto-filter (useFilters, BCJ-forward follow-up) with kdfLg2 control; plain + packed + header-encrypted headers; solid folder cache with lazy decode; per-substream CRC verify; SkipDir/SkipAll/ continueOnError walk protocol; registry/vfs/high-level integration; PPMd/BCJ2 refuse cleanly)

Credits

Thanks to Gareth Davidson (bitplane) for the excellent rar-research documentation and the rars Rust implementation. Both were invaluable references for the RAR support in this library.

❤ Contributions & Support

• 🐛 Found a bug? Create a new Issue
• 👋 Wanna help? Fork it!

🎩 License

MIT license | Nim Community.