Issues with test runner configuration

[Bodigrim]

hackage-server/exes/BuildClient.hs

Lines 654 to 666 in 1217eb0

	testPackage :: Verbosity -> BuildOpts -> DocInfo -> IO (Maybe String, Maybe FilePath, Maybe FilePath)
	testPackage verbosity opts docInfo = do
	let pkgid = docInfoPackage docInfo
	testLogFile = (installDirectory opts) </> display pkgid <.> "test"
	testReportFile = (installDirectory opts) </> "reports" </> display pkgid <.> "test"
	testResultFile = (resultsDirectory opts) </> display pkgid <.> "test"
	pkg_flags =
	["all",
	"--enable-coverage",
	"--test-log=" ++ testReportFile,
	"--test-show-details=never",
	"--disable-optimization"]
	notice verbosity ("Testing " ++ display pkgid)

IMO there are a few issues with this:

• Hiding --test-show-details is very unhelpful. Presumably users are supposed to access --test-log somehow, but I cannot find any link to it in UI.
• Why --disable-optimizations? This breaks inspection testing (fusion rules and such), here is an example: Inspection tests fail if -O0 is forced text#634
• --enable-coverage prevents test execution for any package depending on vector (so pretty much any non-trivial package) because of

Resolving dependencies...
Error:
    Internal libraries only supported with per-component builds.
    Per-component builds were disabled because program coverage is enabled
    In the package 'vector-0.13.2.0'

[gbaz]

The first issue is a good call! I believe the plan was to add a ui to give access to the test build reports, but I can't find an issue for that. Imho that's the right solution.

On point two, I believe optimizations are disabled because they interact poorly with coverage.

Finally, on point three imho the right long term solution is to fix coverage with internal libraries. There is a workaround, but it appears to be somewhat tricky to implement as part of docbuilder, and really we "shouldn't" need it: haskell/cabal#6440

[ulysses4ever]

Fun fact: cabal-install adopted --test-show-details=direct as the default since version 3.12 because people kept asking for it.

[Bodigrim]

The first issue is a good call! I believe the plan was to add a ui to give access to the test build reports, but I can't find an issue for that. Imho that's the right solution.

While we wait for the long-term solution to arrive, could we enable test logs being reported directly for time being? It's very challenging to figure out what failed and why when there are no logs available.

On point two, I believe optimizations are disabled because they interact poorly with coverage.

Hmm, I don't recall any particular issues with regards to this. After all, the coverage metric we care about is when a package is used "normally", which means "with optimizations".

[ysangkok]

the plan was to add a ui to give access to the test build reports

The test logs are already available, as far as I can see. For example, for lens-5.3.5, click "build log" in the side bar. Then at the very bottom, there is a section called "Test log". The section is only shown if logs are available, as you can see in report.html.st.

[Bodigrim]

That's build logs, but not test logs. If you look at the bottom of https://hackage.haskell.org/package/text-2.1.2/reports/2, it says "Error: cabal: Tests failed for text-2.1.2", but the failure is nowhere to be found.

Running 1 test suites...
Test suite tests: RUNNING...
Test suite tests: FAIL
Test suite logged to:
/var/lib/hackage-doc-builder/build-cache/tmp-install/reports/text-2.1.2.test

There is no way to access /var/lib/hackage-doc-builder/build-cache/tmp-install/reports/text-2.1.2.test.

[Bodigrim]

Thanks for working on test logs! However, I think #1417 does not cover all the points raised, the issues about --disable-optimisations and --enable-coverage still stand.

[ysangkok]

Ok, for now I suppose we can just remove those two flags? Because having tests working for "non-trivial packages" seems more important than having coverage.

[Bodigrim]

Yes, I think removing both flags is a reasonable way forward.

[gbaz]

I believe that more recent cabals finally have coverage working with per-component builds? Can we put a pin in changing the flags until we conform the status of cabal and per-component builds

[Bodigrim]

@gbaz I think the problem is that the test runner uses v1- infrastructure, which can just barely cope with internal sublibraries, and enabling coverage pushes it over the limit. I don't recall any improvements in recent Cabals about it. So short of people willing to migrate the test runner to v2-, I'd say that getting tests without coverage is much better than not getting any tests at all (in the majority of cases).

[ysangkok]

The current BuildClient code seems to use v2-test. But the problem does appear with cabal-install-3.10:

reflex-dom-core-0.8.1.4 $ cabal-3.10.3.0 v2-test all --enable-coverage --dry-run
Error:
    Internal libraries only supported with per-component builds.
    Per-component builds were disabled because program coverage is enabled
    In the package 'vector-0.13.2.0'

I get another message if I pass --enable-tests, so I didn't, since it seems BuildClient doesn't pass it either. With cabal-install-3.18, there is another issue with that package, with and without --enable-tests, but they seem unrelated:

reflex-dom-core-0.8.1.4 $ cabal-3.18.1.0 v2-test all --enable-coverage --dry-run --enable-tests
Resolving dependencies...
Error: [Cabal-7107]
Could not resolve dependencies:
[__0] trying: reflex-dom-core-0.8.1.4 (user goal)
[__1] trying: reflex-dom-core:*test
[__2] trying: reflex-dom-core:+gc-tests
[__3] unknown package: chrome-test-utils (dependency of reflex-dom-core +gc-tests *test)
[__3] fail (backjumping, conflict set: chrome-test-utils, reflex-dom-core, reflex-dom-core:gc-tests, reflex-dom-core:test)
After searching the rest of the dependency tree exhaustively, these were the goals I've had most trouble fulfilling: reflex-dom-core, reflex-dom-core:gc-tests, reflex-dom-core:test, chrome-test-utils

So this suggests to me that the issue would be resolved when cabal is updated on the Hackage builder.

[philderbeast]

@ysangkok the unknown package: chrome-test-utils problem testing this package stems from a test suite dependency not being on hackage, doesn't it? The chrome-test-utils package is in the same reflex-dom repository.