ci: free speedups

[Firestar99]

Free speedups I found while working on #526 that don't require rust-cache. I'm also gonna use this PR as a reference point for that PR.

• remove unnecessary env vars
• move "test difftest" step from difftest to test
• disable vulkan sdk cache on mac, cache restore is broken, see Cache restore on MacOS broken jakoch/install-vulkan-sdk-action#558

Results

Speedup main vs branch:

job	windows	mac	linux
difftest main	26m 30s	15m 0s	14m 30s
difftest branch	22m 16s	20m 0s	11m 25s
difftest ~change	-4m 15s	+5m	-3m
test main	18m 10s	15m 41s	11m 28s
test branch	22m 47s	13m 53s	12m 30s
test ~change	+4m 30s	-2m	+1m

• mac difftest: that runner seems to have been slow overall, likely fluke
• mac test: surprising decrease, seems like mac runners have high variance in run times, likely fluke
• windows test: surprisingly high but real increase compared to other OSs, likely caused by cargo on windows not reusing artifacts between test and dev profiles. Still an overall speedup in end to end, since difftest was the bottleneck

End to end: 26m 41s -> 23m 29s: -3min 10s
#526 has an end to end of 15m 41s with a cache per job, so there's still 8min to gain

[LegNeato]