Fix meta-modules preferred compiler bug

[eap]

The motivation for this PR comes from the the function custom_sort_key which was broken, leading to some bugs when generating meta-modules with compilers that were dependent on the preferred compiler.

When sorting, Spec.__contains__, checks the dependency tree of a spec, since llvm was built with gcc it got the same sort order as gcc in the list.

Other fixes

The for compiler in compilers loop had a whole bunch of meta-module logic that only applied to the preferred compiler, this was gated with a "continue" clause. I just dropped that clause and moved everything into the function context. That saved some variable scope bookkeeping and (along with the sort fixes) guarantees that it runs every time.

[climbfuji]

Thanks for this cleanup. I'll need to give this a thorough test on a few systems.

[climbfuji]

@eap I am finally testing this PR - stay tuned

[eap]

review notes

[climbfuji]

I think this PR is good the way it's now. I tested it on 2/3 platforms, just finishing the third.

[climbfuji]

Works on the three sites I tested (HPCs / devbox, lmod / tcl, intel / gcc / llvm)

[climbfuji]

@AlexanderHrabski-NOAA @AlexanderRichert-NOAA I'd like to merge this soon, if possible. Any desire to review and/or test? Thanks!

[rickgrubin-noaa]

Successfully tested on two EPIC hosts.

[climbfuji]

Successfully tested on two EPIC hosts.

Thanks, Rick for testing! Appreciate it.

[AlexanderHrabski-NOAA]

@climbfuji I'm not too familiar with the meta-module piece of spack-stack (though I am learning), and I'll need to shift my attention elsewhere for a couple days. I can try to find time on Friday to review this if it's still open!

[climbfuji]

@climbfuji I'm not too familiar with the meta-module piece of spack-stack (though I am learning), and I'll need to shift my attention elsewhere for a couple days. I can try to find time on Friday to review this if it's still open!

We can wait until Friday, no problem. We have plenty of other PRs in the pipeline :-)