test(global): a test run owns a temporary folder of its own, which its teardown checks and removes

CONTRIBUTING.md

@@ -43,6 +43,7 @@ The preview opens under `/docs/`. Read [Documentation contributions](docs/contri
 - **TypeScript must be erasable**, because Node strips the types at runtime. That means no `enum`, `namespace`, parameter properties or decorators. Relative imports include the `.ts` extension.
 - **Tests sit next to the code** as `*.test.ts` and test behaviour through the public interface, with injected fakes rather than mocks.
 - **A test waits for the event that shows the state it needs, never for the clock,** and the one-minute ceiling in `vitest.shared.ts` only guards against a hang.
+- **A test removes what it makes.** Each run of a package's tests works in a temporary folder of its own, which `tmpdir()` answers in every worker, and its teardown fails the run, naming what a test left there, before it removes the folder.
 
 ## Commits and pull requests
 

packages/identity/src/keys/api-keys-setting.test.ts

@@ -1,9 +1,9 @@
 import { spawnSync } from 'node:child_process';
-import { mkdtempSync, writeFileSync } from 'node:fs';
+import { mkdtempSync, rmSync, writeFileSync } from 'node:fs';
 import { tmpdir } from 'node:os';
 import { join } from 'node:path';
 
-import { describe, expect, it } from 'vitest';
+import { describe, expect, it, onTestFinished } from 'vitest';
 
 import { InvalidApiKeysError, createApiKey, readApiKeys, type ApiKey } from '../index.ts';
 
@@ -59,6 +59,9 @@ describe('readApiKeys', () => {
 
   it('reads an unquoted compact JSON line from an env file unchanged', () => {
     const directory = mkdtempSync(join(tmpdir(), 'api-keys-'));
+    onTestFinished(() => {
+      rmSync(directory, { recursive: true, force: true });
+    });
     const line = JSON.stringify([entry, everything]);
     writeFileSync(join(directory, 'keys.env'), `API_KEYS=${line}\n`);
 

packages/operations/README.md

@@ -215,3 +215,5 @@ The digest is Node's, behind the one module of the repository for it: the digest
 `recordedContentBehaviour(entry)` is the behaviour suite of the content port, in `src/content-testing`: a content of no bytes to 64 MiB kept and streamed back, a text kept once with its first media type in its playbook alone, a content put again whole or with its digest, the refusals of a wrong digest, a wrong size and a stream that fails, wrong bytes put with the digest of a content held, a wrong put racing a right one, and staging, read through a second client, ended, refused and raced. An adapter runs it with a `ContentEntry` whose `aStore()` answers `{ content, elsewhere, unnamedLeft }`: the port, the port of a second client of the same store, and how many pieces of puts no kept content names the store holds, which is 0 after every test. This package runs it on `memoryRecordedContent`, the ledger on SQLite and PostgreSQL, and `@on.auto/object-storage` on its fake bucket and on a real S3-compatible server. The samples it is made of are exported with it: `generated(bytes, seed)` and `generatedStream`, bytes made the same for a seed, `answerOfAbout(bytes)`, a JSON answer of about that size, `readBack` and `readThrough`, which digest what a read streams, `deliveredBeforeFailing(stream)`, how many bytes a failing stream handed over, `brokenAfter(bytes)` and `unread()`, streams that fail or are never read, and `resultOf`, `refusalOf` and `present`, which run an effect for its value, its failure or its value present.
 
 `spawnForTheTest(command, args, options)` (`src/process-testing/test-processes.ts`) is how a test of the repository starts a process it runs itself. It starts it as `spawn` of `node:child_process` does and registers its end with `onTestFinished` of Vitest, the hook that runs once the test finished, whether it passed, failed or timed out: a process still there is sent SIGTERM, on which the server stops, and SIGKILL if it is still there `stoppedWithinMs`, 10 s, later, a bound the server's test of its longest stop holds that stop under, and well within the 60 s a hook may take. `endWithTheTest(pid)` ends in the same way, by its pid, a process the test did not start itself, such as a server started in the background, and signals no pid that `kill(pid, 0)` finds gone or another user's; it cannot tell a pid the system gave to another process of the same user since. `endWithTheTest(reader)` takes a `PidReader` in place of the pid, registered before the process is started: once the test finished, it reads the pid with `read()` again and again while `stillToCome()` holds, for at most `readWithinMs`, and ends the pid it read, if any. A worker of the test runner that dies runs no hook, so it ends nothing.
+
+`aFolderOfTheTestRun(runningIn)` (`src/process-testing/test-run-folder.ts`) is the global setup of the tests of every package, which `vitest.shared.ts` names with its default, `aFolderOfTheTestRun()`. Each run makes a folder of its own in the temporary folder, `auto-runtime-test-run-<project>-<random>`, and sets `TMPDIR` to it before any worker starts, so `tmpdir()` of `node:os` answers it in every worker and in every process a test starts with the environment it inherits, and two runs at once have two folders. Its teardown, which a worker that dies, a hook that times out or a child that is killed does not keep from running, lists what the tests left there, ends each `RunningProcess` that `runningIn(entry)` finds still running in an entry, with SIGTERM and with SIGKILL if `isStillRunning()` still holds `stoppedWithinMs` later, sending a signal only while it holds, fails the run naming each entry and each process it ended, and removes the folder whatever happened. A package whose own processes may outlive their test gives it a `runningIn` of its own, as the server does for the runtimes its tests start in the background, through `@on.auto/operations/test-run-folder`. That is an entry of its own because a global setup runs in Vitest's own process and not in a worker, so neither it nor anything it imports takes Vitest, with whose hooks `@on.auto/operations/testing` registers what it ends; `endedWhileItRuns(running)` of `@on.auto/operations/testing` is the same end for the hook of a test, over the same `RunningProcess`.

packages/operations/package.json

@@ -6,6 +6,7 @@
   "type": "module",
   "exports": {
     ".": "./src/index.ts",
+    "./test-run-folder": "./src/process-testing/test-run-folder.ts",
     "./testing": "./src/testing/index.ts"
   },
   "scripts": {

packages/operations/src/process-testing/process-endings.ts

@@ -0,0 +1,48 @@
+import { setTimeout } from 'node:timers/promises';
+
+export interface Ending {
+  readonly isGone: () => boolean;
+  readonly signal: (name: NodeJS.Signals) => unknown;
+}
+
+export interface RunningProcess {
+  readonly name: string;
+  readonly pid: number;
+  readonly isStillRunning: () => boolean;
+}
+
+export const stoppedWithinMs = 10_000;
+
+export const lookEveryMs = 50;
+
+export function signalled(pid: number, signal: NodeJS.Signals | 0): boolean {
+  try {
+    process.kill(pid, signal);
+    return true;
+  } catch {
+    return false;
+  }
+}
+
+async function goneBy(ending: Ending, deadline: number): Promise<boolean> {
+  if (ending.isGone() || performance.now() >= deadline) {
+    return ending.isGone();
+  }
+  await setTimeout(lookEveryMs);
+  return goneBy(ending, deadline);
+}
+
+export async function endedWithin(ending: Ending, withinMs: number): Promise<void> {
+  if (ending.isGone()) {
+    return;
+  }
+  ending.signal('SIGTERM');
+  if (!(await goneBy(ending, performance.now() + withinMs))) {
+    ending.signal('SIGKILL');
+    await goneBy(ending, Number.POSITIVE_INFINITY);
+  }
+}
+
+export function endedWhileItRuns({ pid, isStillRunning }: RunningProcess): Promise<void> {
+  return endedWithin({ isGone: () => !isStillRunning(), signal: (name) => signalled(pid, name) }, stoppedWithinMs);
+}

packages/operations/src/process-testing/test-processes.test.ts

@@ -5,7 +5,8 @@ import { setTimeout } from 'node:timers/promises';
 
 import { describe, expect, it, onTestFinished } from 'vitest';
 
-import { endWithTheTest, spawnForTheTest, stoppedWithinMs, type PidReader, type TestFinish } from './test-processes.ts';
+import { stoppedWithinMs } from './process-endings.ts';
+import { endWithTheTest, spawnForTheTest, type PidReader, type TestFinish } from './test-processes.ts';
 
 const stopsWhenTold = [
   "process.on('SIGTERM', () => { process.stdout.write('stopped'); process.exit(0); });",

packages/operations/src/process-testing/test-processes.ts

@@ -3,6 +3,8 @@ import { setTimeout } from 'node:timers/promises';
 
 import { onTestFinished } from 'vitest';
 
+import { endedWithin, lookEveryMs, signalled, stoppedWithinMs, type Ending } from './process-endings.ts';
+
 export interface TestProcessOptions {
   readonly cwd?: string;
   readonly env?: Readonly<Record<string, string | undefined>>;
@@ -23,50 +25,13 @@ export interface PidReader {
   readonly readWithinMs: number;
 }
 
-interface Ending {
-  readonly isGone: () => boolean;
-  readonly signal: (name: NodeJS.Signals) => unknown;
-}
-
-export const stoppedWithinMs = 10_000;
-
-const lookEveryMs = 50;
-
 const theTestsFinish: TestFinish = {
   registered: (end) => {
     onTestFinished(end);
   },
   stoppedWithinMs,
 };
 
-function signalled(pid: number, signal: NodeJS.Signals | 0): boolean {
-  try {
-    process.kill(pid, signal);
-    return true;
-  } catch {
-    return false;
-  }
-}
-
-async function goneBy(ending: Ending, deadline: number): Promise<boolean> {
-  if (ending.isGone() || performance.now() >= deadline) {
-    return ending.isGone();
-  }
-  await setTimeout(lookEveryMs);
-  return goneBy(ending, deadline);
-}
-
-async function ended(ending: Ending, withinMs: number): Promise<void> {
-  if (ending.isGone()) {
-    return;
-  }
-  ending.signal('SIGTERM');
-  if (!(await goneBy(ending, performance.now() + withinMs))) {
-    ending.signal('SIGKILL');
-    await goneBy(ending, Number.POSITIVE_INFINITY);
-  }
-}
-
 function endingOfPid(pid: number): Ending {
   return { isGone: () => !signalled(pid, 0), signal: (name) => signalled(pid, name) };
 }
@@ -90,7 +55,7 @@ export function endWithTheTest(pidOrReader: number | PidReader, finish: TestFini
   finish.registered(async () => {
     const pid = await pidOf(pidOrReader);
     if (pid !== undefined) {
-      await ended(endingOfPid(pid), finish.stoppedWithinMs);
+      await endedWithin(endingOfPid(pid), finish.stoppedWithinMs);
     }
   });
 }
@@ -114,6 +79,6 @@ export function spawnForTheTest(
     isGone: () => child.pid === undefined || child.exitCode !== null || child.signalCode !== null,
     signal: (name) => child.kill(name),
   };
-  finish.registered(() => ended(ending, finish.stoppedWithinMs));
+  finish.registered(() => endedWithin(ending, finish.stoppedWithinMs));
   return child;
 }

packages/operations/src/process-testing/test-run-folder.test.ts

@@ -0,0 +1,130 @@
+import { spawnSync } from 'node:child_process';
+import { once } from 'node:events';
+import { existsSync, mkdirSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { basename, dirname, join } from 'node:path';
+
+import { describe, expect, it } from 'vitest';
+
+import type { RunningProcess } from './process-endings.ts';
+import { spawnForTheTest } from './test-processes.ts';
+import theFolderOfTheTestRun, { aFolderOfTheTestRun, type RunningIn } from './test-run-folder.ts';
+
+const run = { name: 'probe' };
+
+function tmpdirOfAProcessItStarts(): string {
+  return spawnSync(process.execPath, ['--eval', "process.stdout.write(require('node:os').tmpdir())"], {
+    encoding: 'utf8',
+  }).stdout;
+}
+
+function leftIn(folder: string, ...names: readonly string[]): void {
+  for (const name of names) {
+    mkdirSync(join(folder, name));
+  }
+}
+
+describe('the temporary folder of a test run', () => {
+  it('is made in the temporary folder it is set up in, named for the run, and is the temporary folder of the run and of what it starts until its teardown removes it', async () => {
+    const outside = tmpdir();
+    const teardown = theFolderOfTheTestRun(run);
+    const folder = tmpdir();
+    const ofAProcessItStarts = tmpdirOfAProcessItStarts();
+
+    await teardown();
+
+    expect([dirname(folder), basename(folder).startsWith('auto-runtime-test-run-probe-')]).toEqual([outside, true]);
+    expect([ofAProcessItStarts, existsSync(folder), tmpdir()]).toEqual([folder, false, outside]);
+  });
+
+  it('fails the run, naming in order what its tests left there, and is removed all the same', async () => {
+    const teardown = theFolderOfTheTestRun(run);
+    const folder = tmpdir();
+    leftIn(folder, 'left-second');
+    writeFileSync(join(folder, 'left-first'), '');
+
+    await expect(teardown()).rejects.toThrow(
+      new Error('The tests of the run did not remove what they left in its temporary folder: left-first, left-second.'),
+    );
+    expect(existsSync(folder)).toBe(false);
+  });
+
+  it('is its own in each of two runs set up at once, so the teardown of one names nothing the other left', async () => {
+    const outside = tmpdir();
+    const first = theFolderOfTheTestRun(run);
+    const ofTheFirst = tmpdir();
+    process.env['TMPDIR'] = outside;
+    const second = theFolderOfTheTestRun(run);
+    leftIn(ofTheFirst, 'left-by-the-first');
+
+    await second();
+
+    await expect(first()).rejects.toThrow(/: left-by-the-first\.$/u);
+  });
+});
+
+const stopsWhenTold = [
+  "process.on('SIGTERM', () => { process.stdout.write('stopped'); process.exit(0); });",
+  'setInterval(() => {}, 1000);',
+  "process.stdout.write('ready ');",
+].join('\n');
+
+interface Probe {
+  readonly ready: Promise<void>;
+  readonly printed: () => string;
+  readonly closed: Promise<readonly unknown[]>;
+  readonly runningIn: RunningIn;
+  readonly lookedIn: () => readonly (readonly [string, boolean])[];
+}
+
+function aProbeRunningIn(home: string): Probe {
+  const child = spawnForTheTest(process.execPath, ['--eval', stopsWhenTold], {
+    stdio: ['ignore', 'pipe', 'inherit'],
+  });
+  const printed = { text: '' };
+  const ready = Promise.withResolvers<void>();
+  child.stdout?.setEncoding('utf8').on('data', (chunk: string) => {
+    printed.text += chunk;
+    ready.resolve();
+  });
+  const running: RunningProcess = {
+    name: 'a probe',
+    pid: Number(child.pid),
+    isStillRunning: () => child.exitCode === null && child.signalCode === null,
+  };
+  const lookedIn: (readonly [string, boolean])[] = [];
+  return {
+    ready: ready.promise,
+    printed: () => printed.text,
+    closed: once(child, 'close'),
+    runningIn: (entry) => {
+      lookedIn.push([basename(entry), existsSync(entry)]);
+      return basename(entry) === home ? [running] : [];
+    },
+    lookedIn: () => lookedIn,
+  };
+}
+
+describe('a process still running in what the tests of a run left in its folder', () => {
+  it('is ended as its teardown looks, before the folder is removed, and named with what it ran in', async () => {
+    const probe = aProbeRunningIn('a-home');
+    const teardown = aFolderOfTheTestRun(probe.runningIn)(run);
+    leftIn(tmpdir(), 'a-home', 'a-folder');
+    await probe.ready;
+
+    await expect(teardown()).rejects.toThrow(
+      new Error(
+        'The tests of the run did not remove what they left in its temporary folder: a-folder, a-home. The teardown ended a probe, which still ran in a-home.',
+      ),
+    );
+
+    expect(await probe.closed).toEqual([0, null]);
+    expect([probe.printed(), probe.lookedIn()]).toEqual([
+      'ready stopped',
+      [
+        ['a-folder', true],
+        ['a-home', true],
+      ],
+    ]);
+  });
+});

packages/operations/src/process-testing/test-run-folder.ts

@@ -0,0 +1,56 @@
+import { mkdtempSync, readdirSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+
+import { endedWhileItRuns, type RunningProcess } from './process-endings.ts';
+
+export type RunningIn = (entry: string) => readonly RunningProcess[];
+
+interface TestRun {
+  readonly name: string;
+}
+
+type TestRunTeardown = () => Promise<void>;
+
+interface RunningInAnEntry {
+  readonly entry: string;
+  readonly running: RunningProcess;
+}
+
+const nothingRuns: RunningIn = () => [];
+
+function leftWords(left: readonly string[], ended: readonly RunningInAnEntry[]): string {
+  return [
+    `The tests of the run did not remove what they left in its temporary folder: ${left.join(', ')}.`,
+    ...ended.map(({ entry, running }) => `The teardown ended ${running.name}, which still ran in ${entry}.`),
+  ].join(' ');
+}
+
+async function checkedAndEnded(folder: string, runningIn: RunningIn): Promise<void> {
+  const left = readdirSync(folder).toSorted();
+  const ended = left.flatMap((entry) =>
+    runningIn(join(folder, entry)).map((running): RunningInAnEntry => ({ entry, running })),
+  );
+  await Promise.all(ended.map(({ running }) => endedWhileItRuns(running)));
+  if (left.length > 0) {
+    throw new Error(leftWords(left, ended));
+  }
+}
+
+export function aFolderOfTheTestRun(runningIn: RunningIn = nothingRuns): (run: TestRun) => TestRunTeardown {
+  return ({ name }) => {
+    const madeIn = tmpdir();
+    const folder = mkdtempSync(join(madeIn, `auto-runtime-test-run-${name}-`));
+    process.env['TMPDIR'] = folder;
+    return async () => {
+      try {
+        await checkedAndEnded(folder, runningIn);
+      } finally {
+        rmSync(folder, { recursive: true, force: true });
+        process.env['TMPDIR'] = madeIn;
+      }
+    };
+  };
+}
+
+export default aFolderOfTheTestRun();

packages/operations/src/testing/index.ts

@@ -33,4 +33,5 @@ export {
   type Unread,
 } from '../content-testing/content-samples.ts';
 export { internalTerms, internalTermsIn } from '../plain-language/internal-terms.ts';
-export { endWithTheTest, spawnForTheTest, stoppedWithinMs, type PidReader } from '../process-testing/test-processes.ts';
+export { endedWhileItRuns, stoppedWithinMs, type RunningProcess } from '../process-testing/process-endings.ts';
+export { endWithTheTest, spawnForTheTest, type PidReader } from '../process-testing/test-processes.ts';