refactor(project): Support cache-exempt build stages

.agents/skills/incremental-build/architecture.md

@@ -367,7 +367,15 @@ Two DSL rules make the per-step recording honest, and both are visible here. The
 
 **Returns.** A step may return resources (stored in the CAS by integrity, as before) or a JSON-serializable value (persisted inline with the invocation's invocation data). Either is injected into a consumer via `needs.<name>`, and the return's signature (resource integrities, or the serialized value) folds into the consumer's per-invocation selection so a changed producer return re-runs the consumer. A map producer's return signature is order-independent: it hashes a `keyId -> signature` map sorted by `keyId`, not a positional list in `keys()` order, so a key reordering that changes nothing semantically (an adapter change, filesystem ordering, a reader-collection reshuffle) does not move the signature and re-run every consumer for nothing. A step returning nothing has a `null` return descriptor. A returned value that is neither resources nor JSON-serializable throws.
 
-**The task descriptor (step-based vs legacy).** From Specification Version 5.0 (and for every built-in task), a task module's default export is a factory `(options) => descriptor`. The descriptor carries the whole task contract in one object: exactly one of `steps` (the step-based path) or `task` (a legacy single-stage body), plus the optional `determineRequiredDependencies` / `determineBuildSignature` callbacks and an optional declared `specVersion`. A task is step-based when its descriptor has a `steps` array. This replaced the former scattered opt-in (a static `stepBased` export on a custom task module, or a `stepBased: true` entry in `definitions/*.js`) and the former top-level named-export callbacks. The descriptor is validated by `validateDescriptor` (exported from `StepRunner.js`, next to `validateSteps`): exactly one of `steps`/`task`, callbacks must be functions, a declared `specVersion` must equal the task extension's configured Specification Version. For a **custom** task extension below 5.0 the gate is a hard version boundary with no shape-sniffing: the default export is the legacy task body invoked directly (any value it returns is ignored), so there is no descriptor and none of its callbacks apply. **This `specVersion` is the custom task extension's own (from its `ui5.yaml`), unrelated to the Specification Version of the project being built; built-in tasks carry the descriptor shape unconditionally and their project-specVersion gating stays in `definitions/*.js`.** Note the cache-internal `stepBased` flag (the persisted `stages` array `[stageId, stepBased?1:0]`, `BuildStageCache`), which means "this stage ran the step runner", is a different thing and is **retained** — only the task-API opt-in was removed.
+**The task descriptor (step-based vs legacy).** From Specification Version 5.0 (and for every built-in task), a task module's default export is a factory `(options) => descriptor`. The descriptor carries the whole task contract in one object: exactly one of `steps` (the step-based path) or `task` (a legacy single-stage body), plus the optional `determineRequiredDependencies` / `determineBuildSignature` callbacks and an optional declared `specVersion`. A task is step-based when its descriptor has a `steps` array. This replaced the former scattered opt-in (a static `stepBased` export on a custom task module, or a `stepBased: true` entry in `definitions/*.js`) and the former top-level named-export callbacks. The descriptor is validated by `validateDescriptor` (exported from `StepRunner.js`, next to `validateSteps`): exactly one of `steps`/`task`, callbacks must be functions, a declared `specVersion` must equal the task extension's configured Specification Version, and an optional `cache` flag must be a boolean (see "Cache-exempt custom tasks"). For a **custom** task extension below 5.0 the gate is a hard version boundary with no shape-sniffing: the default export is the legacy task body invoked directly (any value it returns is ignored), so there is no descriptor and none of its callbacks apply. **This `specVersion` is the custom task extension's own (from its `ui5.yaml`), unrelated to the Specification Version of the project being built; built-in tasks carry the descriptor shape unconditionally and their project-specVersion gating stays in `definitions/*.js`.** Note the cache-internal `stepBased` flag (the persisted `stages` array `[stageId, stepBased?1:0]`, `BuildStageCache`), which means "this stage ran the step runner", is a different thing and is **retained** — only the task-API opt-in was removed.
+
+**Cache-exempt custom tasks.** A custom task can be exempt from the build cache, so it executes on every build. Many custom tasks read and write files outside the resource model (direct `fs` access), which the cache cannot track, so a cached result could not be invalidated when such an untracked input changes, and skipping the task would drop its untracked side effects. A custom task is exempt when it declares a Specification Version below 5.0 (no descriptor, legacy body, assumed to do untracked I/O) or when a 5.0 task sets `cache: false` on its descriptor. The single rule lives in `getCustomTaskCacheExemption(isSpecV5, descriptor)` (exported from `TaskDefinitions.js`), returning `{reason: "specVersion" | "optOut"}` or `undefined`. Built-in and other standard tasks are always cacheable (a built-in descriptor could set `cache: false`, honored uniformly, but none does). The exemption only ever disables caching, and is scoped to the task's own project, so an untracked write that reaches *another* project's inputs is still outside what any invalidation tracks.
+
+Exemption keeps the project's *other* tasks cached. Two things change for an exempt task, and only for it:
+- Its stages are always executed and never stored. `prepareStageExecutionAndValidateCache` returns `false` (no cache lookup) for an exempt stage, and `recordStageResult` withholds the stage output and its per-key map from the cache (it skips `#stageCache.addSignature`). The signature tuple, request indices (`recordRequests`) and written-path accounting still run, so the result-signature machinery and the persisted `stages`/`stage_request_metadata` stay consistent. The exempt stage's request metadata is restorable, so only its `stage_metadata` and the project result row are absent, and neither is ever read for such a project.
+- The project's result cache is disabled. `ProjectBuildContext.create` detects an exempt task via `TaskDefinitions.getCustomTaskCacheExemptions()` (sharing the memoized descriptors with the build-signature phase) and calls `ProjectBuildCache.disableResultCache()` **before** `validateCache` runs. The result-cache fast path can skip the whole task loop before any stage runs, so it must be off up front. `#findResultCache` then force-misses and `#prepareResultCache` writes nothing, so `validateCache` always reports the project not-fresh and the task loop runs, reaching the exempt task while the cacheable stages are still served from their individual stage caches inside that loop. `possiblyRequiresBuild` is unaffected: an exempt project only reaches `isFresh()` right after a real build this session, where skipping a redundant rebuild is correct (the cached reader serves until the next source change).
+
+The per-stage exemption flows from `TaskRunner` into `ProjectBuildCache.setTasks` via a per-task `cacheable` flag (absence means cacheable, keeping the payload minimal). `setTasks` collects the stage ids of `cacheable: false` tasks into `#cacheExemptStageIds` and, as a belt-and-suspenders for callers that bypass `ProjectBuildContext` (e.g. tests), also sets the result-cache-disabled flag when any exempt stage is present. The `TaskRunner` reports the exemption to the user when such a task runs. A below-5.0 task warns **once per process** (`warnedAboutUncacheableCustomTasks`, module-scoped, since a long-running `ui5 serve` must not repeat it per rebuild), while a deliberate 5.0 `cache: false` opt-out is logged at verbose level.
 
 **One factory call, memoized on `TaskDefinitions`.** The descriptor is resolved once per build and shared by the two phases that need it. `TaskDefinitions.getBuildSignatures()` (which needs `determineBuildSignature`) runs inside `ProjectBuildContext.create()`, before the `ProjectBuildCache` and `TaskRunner` exist; `TaskRunner` (which needs `steps`/`task`/`determineRequiredDependencies`) runs later off the **same** `TaskDefinitions` instance. For a built-in, `TaskDefinitions.#resolveStandardDescriptor` calls the factory from `taskRepository.getTask(name)` and caches it on the standard-task map entry (`getStandardDescriptor` exposes it to `TaskRunner`); a skipped (`skip:true`) or `taskFunction`-overridden task has no descriptor and falls back to hashing its options. For a custom task, `Task#getDescriptor(taskName, options)` memoizes per project and runtime `taskName`: two projects that use the same shared extension instance receive descriptors created with their own options, while a `--N`-suffixed second configuration within one project also gets its own descriptor. `TaskDefinitions` completes `projectName`/`projectNamespace` into each standard task's options **before** the first factory call, so a factory that branches on `projectNamespace` at factory time (`generateThemeDesignerResources` emits its `libraryTheming` step only for a namespace) produces the same descriptor at signature time and at execution.
 

packages/project/lib/build/TaskDefinitions.js

@@ -1,6 +1,39 @@
 import crypto from "node:crypto";
 import {getLogger} from "@ui5/logger";
 import {validateDescriptor} from "./helpers/StepRunner.js";
+import composeTaskList from "./helpers/composeTaskList.js";
+
+/**
+ * Decides whether a custom task is exempt from the build cache, the single rule shared by the
+ * build-signature phase (which disables the project's result cache, see
+ * [ProjectBuildContext]{@link @ui5/project/build/helpers/ProjectBuildContext}) and the TaskRunner
+ * (which marks the task's stages non-cacheable and emits the user-facing warning).
+ *
+ * A custom task below Specification Version 5.0 is always exempt: it has no descriptor, and its legacy
+ * body may perform file reads and writes outside the resource model that the cache cannot track, so a
+ * cached result could not be invalidated when those untracked inputs change. From 5.0 a task opts out
+ * explicitly by setting <code>cache: false</code> on its descriptor.
+ *
+ * Takes the already-resolved <code>isSpecV5</code> rather than the Specification Version object so the
+ * callers, which have computed it, do not repeat the comparison.
+ *
+ * @public
+ * @param {boolean} isSpecV5 Whether the custom task extension's own configured Specification Version is
+ *   5.0 or higher (unrelated to the Specification Version of the project being built)
+ * @param {object} [descriptor] The task's resolved descriptor (Specification Version 5.0 and later)
+ * @returns {{reason: string}|undefined} An exemption with its <code>reason</code>
+ *   (<code>"specVersion"</code> below 5.0, <code>"optOut"</code> for an explicit <code>cache: false</code>),
+ *   or <code>undefined</code> if the task is cacheable
+ */
+export function getCustomTaskCacheExemption(isSpecV5, descriptor) {
+	if (!isSpecV5) {
+		return {reason: "specVersion"};
+	}
+	if (descriptor && descriptor.cache === false) {
+		return {reason: "optOut"};
+	}
+	return undefined;
+}
 
 export default class TaskDefinitions {
 	#graph;
@@ -239,4 +272,56 @@ export default class TaskDefinitions {
 	async getStandardDescriptor(taskName, taskDef) {
 		return this.#resolveStandardDescriptor(taskName, taskDef);
 	}
+
+	/**
+	 * Returns the cache exemptions of the project's selected custom tasks, keyed by runtime task name
+	 * (with the <code>--&lt;number&gt;</code> suffix a repeated configuration gets). A task is listed only
+	 * when it is selected for execution and exempt from the build cache.
+	 *
+	 * Shares one factory call per custom task with {@link #getBuildSignatures} (the descriptor is memoized
+	 * per project and runtime task name on the {@link @ui5/project/specifications/extensions/Task} extension), so this
+	 * adds no extra task-module evaluation. The {@link @ui5/project/build/helpers/ProjectBuildContext} uses
+	 * it to decide whether to disable the project's result cache; the exemptions are re-derived per task by
+	 * the TaskRunner from the same descriptors for the per-stage handling and the user-facing warning.
+	 *
+	 * @public
+	 * @param {@ui5/project/build/ProjectBuilder~BuildConfiguration} [buildConfig] Build task selection
+	 * @returns {Promise<Map<string, {reason: string, extensionName: string, specVersion: string}>>}
+	 *   Exempt custom tasks and why (<code>reason</code> is <code>"specVersion"</code> or <code>"optOut"</code>,
+	 *   see {@link getCustomTaskCacheExemption})
+	 */
+	async getCustomTaskCacheExemptions(buildConfig = {includedTasks: [], excludedTasks: []}) {
+		await this._initTasks();
+		const selectedTasks = new Set(composeTaskList(
+			[...this.#standardTasks.keys(), ...this.#customTasks.keys()], buildConfig));
+		const projectName = this.#project.getName();
+		const projectNamespace = this.#project.getNamespace();
+		const exemptions = new Map();
+		for (const [taskName, {taskDef, task}] of this.#customTasks) {
+			// Task selection applies to every configuration of a repeated task, as in TaskRunner.
+			if (!selectedTasks.has(taskName.replace(/--\d+$/, ""))) {
+				continue;
+			}
+			const specVersion = task.getSpecVersion();
+			const isSpecV5 = specVersion.gte("5.0");
+			// Below 5.0 there is no descriptor; the exemption rule only needs the descriptor to read the
+			// explicit opt-out flag, which exists from 5.0.
+			const descriptor = isSpecV5 ?
+				await task.getDescriptor(taskName, {
+					projectName,
+					projectNamespace,
+					configuration: taskDef.configuration,
+					taskName
+				}) : undefined;
+			const exemption = getCustomTaskCacheExemption(isSpecV5, descriptor);
+			if (exemption) {
+				exemptions.set(taskName, {
+					reason: exemption.reason,
+					extensionName: task.getName(),
+					specVersion: specVersion.toString()
+				});
+			}
+		}
+		return exemptions;
+	}
 }

packages/project/lib/build/TaskRunner.js

@@ -2,8 +2,14 @@ import {getLogger} from "@ui5/logger";
 import composeTaskList from "./helpers/composeTaskList.js";
 import MonitoredTaskUtil from "./helpers/MonitoredTaskUtil.js";
 import StepRunner, {validateSteps} from "./helpers/StepRunner.js";
+import {getCustomTaskCacheExemption} from "./TaskDefinitions.js";
 import {createReaderCollection, createMonitor} from "@ui5/fs/resourceFactory";
 
+// Process-level guard for the single warning about custom tasks that are exempt from the build cache
+// because they declare a Specification Version below 5.0. A long-running `ui5 serve` process rebuilds
+// many times across many projects; the user needs to be told once, not on every rebuild.
+let warnedAboutUncacheableCustomTasks = false;
+
 const EMPTY_RESOURCE_REQUESTS = {paths: [], patterns: []};
 
 /**
@@ -219,15 +225,23 @@ class TaskRunner {
 		const stageTasks = allTasks.map((taskName) => {
 			const taskDef = this._tasks[taskName];
 			const steps = taskDef.descriptor?.steps;
-			if (!steps) {
-				return {taskName};
+			const stageTask = {taskName};
+			if (steps) {
+				// Validate the whole step list before it creates any stage: setTasks below derives a stage id
+				// per step name, so a duplicate name or a malformed declaration must be rejected here, not
+				// later when the step runs (by then two steps sharing a name have already created two stages
+				// with one id).
+				validateSteps(steps, {taskName});
+				taskDef.steps = Object.freeze(steps);
+				stageTask.stepNames = steps.map((step) => step.name);
+			}
+			// Only flag an exempt task; absence means cacheable (setTasks defaults to true). This keeps the
+			// payload minimal and reports the exemption to the user when such a task runs.
+			if (taskDef.cacheable === false) {
+				stageTask.cacheable = false;
+				this.#reportCacheExemption(taskDef.cacheExemption);
 			}
-			// Validate the whole step list before it creates any stage: setTasks below derives a stage id per
-			// step name, so a duplicate name or a malformed declaration must be rejected here, not later when
-			// the step runs (by then two steps sharing a name have already created two stages with one id).
-			validateSteps(steps, {taskName});
-			taskDef.steps = Object.freeze(steps);
-			return {taskName, stepNames: steps.map((step) => step.name)};
+			return stageTask;
 		});
 		this._buildCache.setTasks(stageTasks);
 
@@ -425,6 +439,9 @@ class TaskRunner {
 			stepBased,
 			descriptor,
 			options,
+			// Standard tasks are cacheable. A descriptor may still opt out via cache:false (none of the
+			// built-ins do), honored uniformly with custom tasks.
+			cacheable: descriptor ? descriptor.cache !== false : true,
 			requiredDependencies: requiresDependencies ? this._directDependencies : new Set()
 		};
 		this._taskExecutionOrder.push(taskName);
@@ -536,6 +553,11 @@ class TaskRunner {
 		// A custom task is step-based when its descriptor carries a steps array (Specification Version 5.0
 		// and later). Below 5.0 there is no descriptor and the task runs as a legacy body.
 		const stepBased = !!descriptor?.steps;
+		// Decide cache exemption from the same rule the ProjectBuildContext used to disable the result
+		// cache, over the same (memoized) descriptor and the isV5 flag already computed above (so no extra
+		// Specification Version comparison). An exempt task's stages are always executed and never stored
+		// (see ProjectBuildCache); runTasks reports the exemption to the user.
+		const cacheExemption = getCustomTaskCacheExemption(isV5, descriptor);
 		this._tasks[taskName] = {
 			task: this._createCustomTaskWrapper({
 				task,
@@ -554,6 +576,12 @@ class TaskRunner {
 			stepBased,
 			descriptor,
 			options: descriptorOptions,
+			cacheable: !cacheExemption,
+			cacheExemption: cacheExemption ? {
+				reason: cacheExemption.reason,
+				extensionName: task.getName(),
+				specVersion: specVersion.toString()
+			} : undefined,
 			requiredDependencies
 		};
 
@@ -729,6 +757,43 @@ class TaskRunner {
 		};
 	}
 
+	/**
+	 * Reports that a custom task is exempt from the build cache, so the user understands why that task runs
+	 * on every build.
+	 *
+	 * A task below Specification Version 5.0 is exempt implicitly (reason <code>"specVersion"</code>): its
+	 * legacy body may perform file I/O outside the resource model that the cache cannot track. This is
+	 * warned once per process (a long-running <code>ui5 serve</code> rebuilds repeatedly, and the user needs
+	 * telling once, not per rebuild). A Specification Version 5.0 task that set <code>cache: false</code>
+	 * (reason <code>"optOut"</code>) made a deliberate choice, so it is only logged at verbose level.
+	 *
+	 * @param {{reason: string, extensionName: string, specVersion: string}} [exemption] The task's exemption,
+	 *   or <code>undefined</code> for a cacheable task
+	 */
+	#reportCacheExemption(exemption) {
+		if (!exemption) {
+			return;
+		}
+		const projectName = this._project.getName();
+		if (exemption.reason === "specVersion") {
+			if (warnedAboutUncacheableCustomTasks) {
+				return;
+			}
+			warnedAboutUncacheableCustomTasks = true;
+			this._log.warn(
+				`Custom task ${exemption.extensionName} of project ${projectName} defines Specification ` +
+				`Version ${exemption.specVersion} and is therefore not cached. Tasks below Specification ` +
+				`Version 5.0 may read or write files outside the build's resource model, which the build ` +
+				`cache cannot track, so their results are never cached and such tasks run on every build. ` +
+				`This can slow down (incremental) builds. Update affected tasks to Specification Version 5.0 ` +
+				`to make them cacheable.`);
+		} else {
+			this._log.verbose(
+				`Custom task ${exemption.extensionName} of project ${projectName} opted out of the build ` +
+				`cache (cache: false in its task descriptor); it runs on every build.`);
+		}
+	}
+
 	/**
 	 * Builds the start/end reporting pair for one execution of a step-based task.
 	 *