'Immutable' Haskell library depends on a C library that has changed - How to fix?

[ulidtko]

Related materials:

• How to reinstall a package? #1476

---

Background

Once a dependency gets installed into Stack's compiled package database ~/.stack/snapshots, it gets aggressively cached for as long as the snapshot hash keeps matching. This is as designed and as desired:

$ stack install lzma
[...]
$ stack exec -- ghc-pkg unregister lzma
$ stack install lzma
lzma> using precompiled package
$

This using precompiled package log (and the fast build) — is exactly as expected.

The issue

It's hard to surgically yank one package/subtree from this precompiled_cache. I met reasons to want to do this many times; will give one below.

Despite stack install offering the --[no-]force-dirty option, it's not effective for this purpose, and behaves the same:

$ stack exec -- ghc-pkg unregister lzma
$
$ stack install --force-dirty lzma
lzma> using precompiled package
$

Expected: fresh rebuild of the lzma package. Actual: using precompiled package again.

Thus, the most precise option to force a rebuild of one dependency is still "nuclear":

• identify the active snapshot hash, e.g. stack exec -- ghc-pkg list | grep /pkgdb,
• drop the entire snapshot, rm -rf ~/.stack/snapshots/{{arch}}/{{hash}}.

More precision is doable with pointwise file removals and DELETE FROM precompiled_cache WHERE library LIKE '%/lzma-%'; — but this feels like violating API boundaries & messing with internal guts. I think anyone would prefer if --force-dirty would instead do that.

Sample rationale

Unlike other linkers, MacOS ld warns about nonexistent -L paths. A real-world scenario I sorted out today proceeded like this:

1. xz library 5.8.1 installed via Homebrew at /opt/homebrew/Cellar/xz/5.8.1;
2. A Haskell project having lzma (a libxz FFI wrapper) in transitive dependency closure was built;
3. xz library upgraded in Homebrew to 5.8.3, /opt/homebrew/Cellar/xz/5.8.3.

Ever since, rebuilds of the Haskell project had been repeatedly emitting:

ld: warning: search path '/opt/homebrew/Cellar/xz/5.8.1/lib' not found

because ~/.stack/snapshots/aarch64-osx/4426a...775284/9.6.7/pkgdb/lzma-0.0.1.1-obp6...LW.conf contains include-dirs: /opt/homebrew/Cellar/xz/5.8.1/include which no longer exists (the 5.8.3 one does).

Wiping the project's .stack-work was not effective to clear the warning, neither git clean -fxd. Such a minor issue, yet necessarily invokes nuclear wipes to enact an eviction of a cached build which went stale, for external reasons.

Tested on:

$ stack --version
Version 3.11.1, Git revision 2352d78a8ac5b42d021c8064b8f64ac1c8b8b3d5 aarch64 hpack-0.39.6

from GHCup.

[ulidtko]

Attaching an ad-hoc throwaway script (AI-generated) that does perform the surgical removal as expected: lzma-warn-cleanup.sh

— as a sketch of what stack install --force-dirty could do with targets in precompiled_cache, prior to rebuilding them.

[mpilgrem]

@ulidtko, I see the problem. In theory, only the build artefacts of (deemed) immutable Haskell libraries are cached. In practice, your experience is that some Haskell libraries are not, in fact, immutable (as deemed), because they have dependencies on mutable C libraries.

The 'traditional' approach is to delete the cache (the snapshots directory in the Stack root). The 'cost' of deleting the cache is, of course, that you lose the benefits of past caching.

You are correct that the --force-dirty flag will do nothing - that flag relates only to mutable packages, where there can, in principle, be changes.

You are also correct that deleting the Stack work directory will do nothing. That directory does not relate to (deemed) immutable packages.

[mpilgrem]

@ulidtko, I have added a link (in your post) to a 2015 version of your issue. There, it is suggested that:

stack exec -- ghc-pkg unregister --force PKG

may be a solution to the 'immutable' package that needs to be rebuilt. I've not tested it myself.

EDIT: I tried a 'toy' single-package Stack project with a dependency on the 'joke' package acme-missiles (so, also added acme-missiles-0.3 as an extra-dep):

> stack build # Also builds and registers acme-missiles-0.3
> stack exec -- ghc-pkg list # Shows acme-missiles-0.3 in the snapshots database
> stack build # No op, of course
> stack exec -- ghc-pkg unregister --force acme-missiles # Unregister acme-missiles from the database
> stack exec -- ghc-pkg list # Shows acme-missiles-0.3 no longer in the snapshots database
> stack build # Also builds and registers acme-missiles-0.3