Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,8 @@ Everything here is best-effort. A missing artifact, an expired one, or a token w

Because artifacts are readable and writable from every trigger, all paths behave the same: an automatic run, a `/codeboarding` command and a manual dispatch each reuse the previous analysis and publish their own. A stored analysis is discarded, and the head re-derived from the base, whenever the pinned CodeBoarding version, `.codeboardingignore`, the model selection, the analysis depth or the merge base changes — or when the base graph it grew from is not the one this run compares against.

With no analysis to continue from, a run still reuses the static analysis (the language servers' call graph, `static_analysis.pkl`) saved with the nearest base for the commit analysed or one of its last 100 first-parent ancestors, whatever configuration made it, and the engine brings it up to date. The engine checks that file itself and rebuilds one from another engine cache version, so a new CodeBoarding release or a changed model or depth costs the language-server pass only when it has to. Set `fresh_analysis: 'true'` to rebuild everything, static analysis included.

Fork pull requests never carry an analysis forward. They are reviewed on request, each review starts from the base, and nothing they produce is read by a run on this repository's own code: untrusted code must not shape state that a later run loads. The action also accepts `pull_request_target`, which runs on the base branch ref and lets both share one chain; that trigger has its own trade-offs (a PR that adds this workflow will not run it until merged, and the fork gate becomes load-bearing), so `pull_request` remains the recommended default.

### Pull requests that change nothing analysed
Expand Down Expand Up @@ -403,6 +405,7 @@ The update runs after the baseline is saved, stops itself after 10 minutes and n
| `agent_model` | both | empty | Analysis-only override for `model`. |
| `parsing_model` | both | empty | Parsing-only override for `model`. |
| `depth_cap` | both | `2` | Positive integer maximum analysis depth, including full-analysis fallbacks. Changing it rebuilds incompatible state. |
| `fresh_analysis` | both | `false` | With `'true'`, the run reuses nothing saved: every analysis is built from scratch, static analysis included. A fresh review republishes only the base analysis at its merge base; later syncs still continue from what the analysis branch holds. See [Reused analysis](#reused-analysis). |
| `github_token` | both | `${{ github.token }}` | Token for comments and sync delivery. |
| `codeboarding_analysis_location` | both | `codeboarding_branch` | Where the synced branch's analysis is stored: `codeboarding_branch` (on `codeboarding/baseline`, for the default branch; the code is never written) or `in_place` (committed as `.codeboarding/` on the synced branch). Sync saves it there and review reads it from there. |
| `warmstart_retention_days` | review | `1` | Days to keep the reusable analysis. Only the next run reads it. |
Expand Down
20 changes: 17 additions & 3 deletions action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,15 @@ inputs:
description: 'Positive integer analysis depth cap, used for both review and sync, including full fallbacks.'
required: false
default: '2'
fresh_analysis:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i am fine with fresh, although i am not sure this is the proper lingo, it sounds a bit odd.
I am more used to things like invalidate/force ;d, we can keep it fresh as well just mentioning. And it doesn't align with our core lingo where I think we call this force

description: >-
When 'true', this run reuses nothing saved: every analysis, its static analysis
included, is built from scratch. A fresh review republishes only the base at its
merge base; later syncs still continue from the analysis branch. Off by default, so
a run continues from the nearest saved analysis and, without one, still reuses the
nearest saved static analysis.
required: false
default: 'false'
review_guide_url:
description: >-
Experimental. Service base URL for POST /v1/review-guide and POST /v1/architecture.
Expand Down Expand Up @@ -477,7 +486,9 @@ runs:

# Both lookups are best effort. Without them the run derives the base from
# the committed baseline and the head from the base, which is what every run
# did before any of this existed.
# did before any of this existed. A fresh run passes no name, so no old base
# id reaches the review artifact either, while fetch-state.sh still clears
# what an earlier use of the action in this job left there.
- name: Fetch published base analysis
id: fetch_base
if: steps.guard.outputs.skip != 'true' && steps.guard.outputs.mode == 'review' && steps.state.outputs.base_name != '' && github.server_url == 'https://github.com'
Expand All @@ -488,7 +499,7 @@ runs:
GH_ENTERPRISE_TOKEN: ${{ inputs.github_token }}
GH_HOST: ${{ github.server_url }}
REPOSITORY: ${{ github.repository }}
ARTIFACT_NAME: ${{ steps.state.outputs.base_name }}
ARTIFACT_NAME: ${{ inputs.fresh_analysis != 'true' && steps.state.outputs.base_name || '' }}
DEST: ${{ runner.temp }}/cb-state/${{ github.action }}/base
# Must match the review retention below: a review references a base by
# id for its whole life, so a base with less than that left is renewed.
Expand All @@ -504,7 +515,7 @@ runs:
GH_ENTERPRISE_TOKEN: ${{ inputs.github_token }}
GH_HOST: ${{ github.server_url }}
REPOSITORY: ${{ github.repository }}
ARTIFACT_NAME: ${{ steps.state.outputs.warmstart_name }}
ARTIFACT_NAME: ${{ inputs.fresh_analysis != 'true' && steps.state.outputs.warmstart_name || '' }}
DEST: ${{ runner.temp }}/cb-state/${{ github.action }}/warmstart
run: "$GITHUB_ACTION_PATH/scripts/action/fetch-state.sh"

Expand All @@ -525,6 +536,7 @@ runs:
GH_HOST: ${{ github.server_url }}
GITHUB_SERVER_URL: ${{ github.server_url }}
DEPTH_CAP: ${{ inputs.depth_cap }}
FRESH_ANALYSIS: ${{ inputs.fresh_analysis }}
MODEL: ${{ inputs.model }}
AGENT_MODEL_INPUT: ${{ inputs.agent_model }}
PARSING_MODEL_INPUT: ${{ inputs.parsing_model }}
Expand Down Expand Up @@ -639,6 +651,7 @@ runs:
GIT_TOKEN: ${{ inputs.github_token }}
GITHUB_SERVER_URL: ${{ github.server_url }}
DEPTH_CAP: ${{ inputs.depth_cap }}
FRESH_ANALYSIS: ${{ inputs.fresh_analysis }}
MODEL: ${{ inputs.model }}
AGENT_MODEL_INPUT: ${{ inputs.agent_model }}
PARSING_MODEL_INPUT: ${{ inputs.parsing_model }}
Expand All @@ -662,6 +675,7 @@ runs:
ENGINE_VERSION: ${{ steps.state.outputs.engine_version }}
CFG_HASH: ${{ steps.state.outputs.cfg_hash }}
DEPTH_CAP: ${{ inputs.depth_cap }}
FRESH_ANALYSIS: ${{ inputs.fresh_analysis }}
MODEL: ${{ inputs.model }}
AGENT_MODEL_INPUT: ${{ inputs.agent_model }}
PARSING_MODEL_INPUT: ${{ inputs.parsing_model }}
Expand Down
5 changes: 5 additions & 0 deletions scripts/action/analysis-state.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,11 @@ if [[ ! "$DEPTH_CAP" =~ ^[1-9][0-9]*$ ]]; then
echo "::error::depth_cap must be a positive integer."
exit 1
fi
FRESH_ANALYSIS="${FRESH_ANALYSIS:-false}"
if [[ ! "$FRESH_ANALYSIS" =~ ^(true|false)$ ]]; then
echo "::error::fresh_analysis must be true or false."
exit 1
fi

# Metadata is only a compatibility check, never the source of configuration.
# Legacy baselines without a configured cap are rebuilt at the requested depth.
Expand Down
10 changes: 8 additions & 2 deletions scripts/action/analyze.sh
Original file line number Diff line number Diff line change
Expand Up @@ -28,16 +28,22 @@ run_core() {
ANALYSIS_PATH="$(awk -F= '$1 == "analysis_path" {print substr($0, index($0, "=") + 1); exit}' <<< "$output")"
}

# fresh_analysis builds from nothing, the static-analysis cache included. Not the
# head: it still grows from the base this run built, since a head analyzed apart
# from its base need not name components alike and would diff as changes.
fresh=""
if [ "${FRESH_ANALYSIS:-false}" = true ] && [ "$role" != head ]; then fresh=--fresh; fi

mkdir -p "$state"
if compatible_state "$state"; then
if [ -z "$fresh" ] && compatible_state "$state"; then
run_core incremental
if [ "$ANALYSIS_MODE" != incremental ] || { [ "$REQUIRES_FULL" != true ] && [ ! -f "$ANALYSIS_PATH" ]; }; then
fail "Core returned an invalid incremental-analysis result."
fi
[ "$REQUIRES_FULL" != true ] || echo "::notice::Core cannot update the $role analysis incrementally; analyzing in full." >&2
fi
if [ "$ANALYSIS_MODE" != incremental ] || [ "$REQUIRES_FULL" = true ]; then
run_core full --depth-cap "$DEPTH_CAP"
run_core full --depth-cap "$DEPTH_CAP" ${fresh:+"$fresh"}
if [ "$ANALYSIS_MODE" != full ] || [ ! -f "$ANALYSIS_PATH" ]; then
fail "Core returned an invalid full-analysis result."
fi
Expand Down
9 changes: 5 additions & 4 deletions scripts/action/fetch-state.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,13 @@
# Best effort: a miss, an expired bundle or a token without actions:read all
# leave the directory absent, and the caller derives from the base instead.
set -euo pipefail
[ -n "${ARTIFACT_NAME:-}" ] || exit 0

# Clear first, on every path. These destinations are fixed, so a second use of
# the action in one job would otherwise inherit the first one's files and treat
# them as this configuration's state.
# Clear first, on every path, a fresh run's empty name included. These
# destinations are fixed, so a second use of the action in one job would
# otherwise inherit the first one's files and treat them as this configuration's
# state.
rm -rf "${DEST:?}"
[ -n "${ARTIFACT_NAME:-}" ] || exit 0

api() { gh api -H 'Accept: application/vnd.github+json' "$@"; }
export GH_HOST="${GH_HOST#*://}"
Expand Down
42 changes: 25 additions & 17 deletions scripts/action/find-ancestor-base.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,20 @@
# under this configuration and downloads it into DEST, so a run without an exact
# base can catch up from it instead of analyzing from scratch. Best effort.
#
# With STATIC_ONLY=true a base saved under any configuration will do, and only
# its static-analysis cache is required, since that is all the caller takes.
#
# Prints ancestor_sha=<the commit whose analysis is now in DEST, or empty> on
# stdout and everything else on stderr.
#
# Needs the tip's first-parent history in CHECKOUT_DIR, at least CATCHUP_BOUND deep.
set -euo pipefail
: "${REPOSITORY:?}" "${CFG_HASH:?}" "${TIP_SHA:?}" "${CHECKOUT_DIR:?}" "${DEST:?}"
: "${REPOSITORY:?}" "${TIP_SHA:?}" "${CHECKOUT_DIR:?}" "${DEST:?}"
if [ "${STATIC_ONLY:-false}" = true ]; then
pattern='codeboarding-base-[0-9a-f]{16}-' needs=(static_analysis.pkl static_analysis.sha)
else
pattern="codeboarding-base-${CFG_HASH:?}-" needs=(analysis.json)
fi
BOUND="${CATCHUP_BOUND:-100}"
rm -rf "$DEST"

Expand All @@ -21,18 +29,18 @@ report() { echo "ancestor_sha=$ancestor"; }
trap report EXIT

# The merge base's first-parent history, nearest first. Distance 0 is the tip
# itself, which a review already looked up by its exact name; sync has no such
# lookup and passes INCLUDE_TIP=true.
# itself, which a review already looked up by its exact name under this
# configuration. Sync and STATIC_ONLY lookups pass INCLUDE_TIP=true: nothing
# looked the tip up for them.
walk="$(git -C "$CHECKOUT_DIR" rev-list --first-parent --max-count=$(( BOUND + 1 )) "$TIP_SHA" 2>/dev/null || true)"
[ "${INCLUDE_TIP:-false}" = true ] || walk="$(tail -n +2 <<< "$walk")"
# Prints the newest saved name for the nearest commit.
nearest() {
local commit
for commit in $walk; do
if grep -qx "$commit" <<< "$saved"; then
echo "$commit"
return 0
fi
grep -m1 -E "^$pattern$commit\$" <<< "$saved" && return 0
done
return 0
}

# The artifact listing, newest first. Its name filter is exact, so a prefix needs
Expand All @@ -50,26 +58,26 @@ trusted='.artifacts[]?
| select(.workflow_run.head_repository_id == .workflow_run.repository_id)
| select(.name | startswith("codeboarding-base-"))
| .name'
prefix="codeboarding-base-$CFG_HASH-"
saved="" page=1
saved="" name="" page=1
while [ "$page" -le "${MAX_PAGES:-50}" ]; do
if ! listing="$(api "repos/$REPOSITORY/actions/artifacts?per_page=100&page=$page" 2>/dev/null)"; then
echo "::warning::Could not list artifacts in $REPOSITORY; not looking for an older saved analysis." >&2
exit 0
fi
names="$(jq -r "$trusted" <<< "$listing" 2>/dev/null || true)"
saved="$saved$(grep "^$prefix" <<< "$names" | cut -c$(( ${#prefix} + 1 ))- || true)"$'\n'
ancestor="$(nearest)"
[ -z "$ancestor" ] || break
saved="$saved$names"$'\n'
name="$(nearest)"
[ -z "$name" ] || break
returned="$(jq -r '.artifacts | length' <<< "$listing" 2>/dev/null || echo 0)"
[ "${returned:-0}" -eq 100 ] || break
page=$(( page + 1 ))
done

[ -n "$ancestor" ] || exit 0
[ -n "$name" ] || exit 0

# Downloaded by its exact name, through the same checks as every other lookup.
if ! GITHUB_OUTPUT="" ARTIFACT_NAME="$prefix$ancestor" DEST="$DEST" \
"$(dirname "$0")/fetch-state.sh" >&2 || [ ! -f "$DEST/analysis.json" ]; then
ancestor=""
fi
GITHUB_OUTPUT="" ARTIFACT_NAME="$name" DEST="$DEST" "$(dirname "$0")/fetch-state.sh" >&2 || exit 0
for file in "${needs[@]}"; do
[ -f "$DEST/$file" ] || exit 0
done
ancestor="${name##*-}"
15 changes: 12 additions & 3 deletions scripts/action/review-generate-baseline.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,11 @@
# 4. the saved analysis of the nearest ancestor /
# 5. nothing built from scratch
#
# Built from scratch, it still reuses the static analysis of the nearest saved
# base, made under any configuration. With FRESH_ANALYSIS=true nothing saved is
# read (action.yml does not fetch source 1), and the base this run publishes
# replaces the one reviews forking there reused so far.
#
# Outputs base_analysis_path, and publish_base when this run produced a base
# other pull requests forking there should reuse.
set -euo pipefail
Expand All @@ -34,13 +39,18 @@ if compatible_state "${BASE_DIR:-}"; then
else
fetch_commit "$REVIEW_BASE_REPO" "$REVIEW_BASE_SHA"
git -C "$CHECKOUT_DIR" worktree add --detach "$checkout" "$REVIEW_BASE_SHA" >/dev/null
if from_codeboarding_baseline "$REVIEW_BASE_REPO" "$REVIEW_BASE_SHA" "$state" "$checkout" ||
seeded=false
if [ "$FRESH_ANALYSIS" = true ]; then
echo "::notice::fresh_analysis is set; analyzing ${REVIEW_BASE_SHA:0:7} from scratch."
keep_user_config "$checkout" "$state"
elif from_codeboarding_baseline "$REVIEW_BASE_REPO" "$REVIEW_BASE_SHA" "$state" "$checkout" ||
from_committed_baseline "$checkout" "$state" ||
from_ancestor_artifact "$REVIEW_BASE_REPO" "$REVIEW_BASE_SHA" "$state" false "$checkout"; then
seeded=true
else
seeded=false
echo "::notice::No saved analysis to start from; analyzing ${REVIEW_BASE_SHA:0:7} from scratch."
keep_user_config "$checkout" "$state"
reuse_static_analysis "$REVIEW_BASE_REPO" "$REVIEW_BASE_SHA" "$state"
fi
# Core never sees the token.
unset GIT_TOKEN
Expand All @@ -49,7 +59,6 @@ else
elif [ "$seeded" = true ]; then
"$ACTION_PATH/scripts/action/analyze.sh" base "$checkout" "$state" >/dev/null
else
echo "::notice::No saved analysis to start from; analyzing ${REVIEW_BASE_SHA:0:7} from scratch."
progress base
"$ACTION_PATH/scripts/action/analyze.sh" base "$checkout" "$state" >/dev/null
progress head
Expand Down
28 changes: 28 additions & 0 deletions scripts/action/seed-sources.sh
Original file line number Diff line number Diff line change
Expand Up @@ -66,3 +66,31 @@ from_ancestor_artifact() {
SEED_SHA="$ancestor"
echo "::notice::Starting from the saved analysis of ${ancestor:0:7}."
}

# With no analysis to continue from, the static analysis in the nearest saved base
# for $2 or an ancestor, under any configuration, still spares Core rebuilding it:
# Core refuses one from another cache version itself. Copies both cache files
# into $3 or neither. Best effort.
reuse_static_analysis() {
local repository="$1" tip="$2" state="$3" found ancestor dest="$RUNNER_TEMP/codeboarding-static"
[ "${ANCESTOR_LOOKUP:-false}" = true ] && [ -n "${REPOSITORY:-}" ] && [ -n "$tip" ] || return 0
fetch_commit "$repository" "$tip" "$(( CATCHUP_BOUND + 1 ))" || true
found="$(GH_TOKEN="${GIT_TOKEN:-}" GH_ENTERPRISE_TOKEN="${GIT_TOKEN:-}" TIP_SHA="$tip" DEST="$dest" \
STATIC_ONLY=true INCLUDE_TIP=true CATCHUP_BOUND="$CATCHUP_BOUND" \
"$HERE/find-ancestor-base.sh" || true)"
ancestor="$(awk -F= '$1 == "ancestor_sha" {print $2; exit}' <<< "$found")"
[ -n "$ancestor" ] || return 0
# Core reads the ignore rules from the checkout and keeps a cached file whatever
# they say later, so a cache made under other rules would keep newly ignored
# files and miss newly analysed ones. The two commits say which rules applied;
# the bundle cannot, since uploads drop dotfiles.
if ! git -C "$CHECKOUT_DIR" diff --quiet "$ancestor" "$tip" -- .codeboarding/.codeboardingignore; then
echo "::notice::The saved static analysis of ${ancestor:0:7} was made with another .codeboardingignore; not reusing it."
return 0
fi
if ! cp "$dest/static_analysis.pkl" "$dest/static_analysis.sha" "$state/"; then
rm -f "$state/static_analysis.pkl" "$state/static_analysis.sha"
return 0
fi
echo "::notice::Reusing the saved static analysis of ${ancestor:0:7}."
}
11 changes: 9 additions & 2 deletions scripts/action/sync-generate-baseline.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,10 @@
# 1. an entry on the baseline branch for this commit or an ancestor
# 2. the analysis committed on the branch, or on the parent of a commit that removed it
# 3. the saved analysis of this commit or the nearest ancestor
# 4. nothing: analyzed from scratch
# 4. nothing: analyzed from scratch, reusing the static analysis of the
# nearest saved base, made under any configuration
#
# With FRESH_ANALYSIS=true nothing saved is read.
set -euo pipefail
# shellcheck source=scripts/action/seed-sources.sh
source "$(dirname "$0")/seed-sources.sh"
Expand All @@ -21,11 +24,15 @@ if [ -n "$(git -C "$CHECKOUT_DIR" log -1 --format='%(trailers:key=CodeBoarding-S
exit 1
fi

if ! from_codeboarding_baseline "${REPOSITORY:-}" "$head_sha" "$state" "$CHECKOUT_DIR" &&
if [ "$FRESH_ANALYSIS" = true ]; then
echo "::notice::fresh_analysis is set; analyzing from scratch."
keep_user_config "$CHECKOUT_DIR" "$state"
elif ! from_codeboarding_baseline "${REPOSITORY:-}" "$head_sha" "$state" "$CHECKOUT_DIR" &&
! from_committed_baseline "$CHECKOUT_DIR" "$state" &&
! from_parent_committed_baseline "${REPOSITORY:-}" "$head_sha" "$state" "$CHECKOUT_DIR" &&
! from_ancestor_artifact "${REPOSITORY:-}" "$head_sha" "$state" true "$CHECKOUT_DIR"; then
keep_user_config "$CHECKOUT_DIR" "$state"
reuse_static_analysis "${REPOSITORY:-}" "$head_sha" "$state"
fi

# Core never sees the token.
Expand Down
29 changes: 16 additions & 13 deletions scripts/analyze_repository.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
from threading import Thread

PROG = "codeboarding"
STATIC_CACHE = {"static_analysis.pkl", "static_analysis.sha"}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a warning on this, that we have hardcoded it multiple times.

# Shared with CodeBoarding/codeboarding_cli/main.py; keep the prefix in sync.
ENGINE_ERROR_PREFIX = "CodeBoarding: "

Expand Down Expand Up @@ -133,6 +134,9 @@ def main(argv: list[str] | None = None) -> int:
parser.add_argument("--checkout", required=True, help="Path to repository checkout")
parser.add_argument("--output-dir", required=True, help="Action-owned output directory")
parser.add_argument("--depth-cap", help="Maximum hierarchy depth allowed for full analyses")
parser.add_argument(
"--fresh", action="store_true", help="Full analysis from nothing, not even the static-analysis cache"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in the core isn't this the --force parameter, i would suggest to keep the same paramtere name so it is less confusing

)

args = parser.parse_args(argv)
checkout = Path(args.checkout)
Expand All @@ -151,19 +155,18 @@ def main(argv: list[str] | None = None) -> int:

if not args.depth_cap:
raise SystemExit("--depth-cap is required for mode=full")
shutil.rmtree(output_dir)
output_dir.mkdir(parents=True)
command = [
PROG,
"full",
"--local",
str(checkout),
"--output-dir",
str(output_dir),
"--depth-cap",
args.depth_cap,
"--force",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

effectively this parameter for my comment from above.

]
# Core validates the static-analysis cache itself: one from another cache
# version is refused and rebuilt, so it survives every full run but a fresh one.
for entry in output_dir.iterdir():
if not args.fresh and entry.name in STATIC_CACHE:
continue
if entry.is_dir() and not entry.is_symlink():
shutil.rmtree(entry)
else:
entry.unlink()
command = [PROG, "full", "--local", str(checkout), "--output-dir", str(output_dir), "--depth-cap", args.depth_cap]
if args.fresh:
command.append("--force")
_run_command(command, output_dir)
analysis_path = output_dir / "analysis.json"
if not analysis_path.is_file():
Expand Down
Loading
Loading