diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 544f9285..d86fca57 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -13,7 +13,7 @@ Hypatia is the neurosymbolic CI/CD intelligence layer for the hyperpolymath ecos ## Architecture ``` -Hypatia # 170 Elixir source files under lib/ (recursive tree count, 2026-11-14) +Hypatia # 171 Elixir source files under lib/ (recursive tree count, 2026-11-16) ├── Elixir pipeline # core: pattern analysis, dispatch, learning ├── Merge orchestration # 12 modules — leases, ledger, ticker (largest subsystem) ├── Kin subsystem # 6 modules — contingency, arbiter, gate, watchdog diff --git a/CHANGELOG.adoc b/CHANGELOG.adoc index 3e87e50c..1672312a 100644 --- a/CHANGELOG.adoc +++ b/CHANGELOG.adoc @@ -12,6 +12,49 @@ https://semver.org/[Semantic Versioning]. === Changed +==== WH013 reads the canonical workflow model; `Hypatia.WorkflowYaml` is gone (2026-11-16) + +WH013 now reads `Hypatia.Workflow`, as RE001, RE002 and WH014 do. With that, +nothing reads `Hypatia.WorkflowYaml`, the line reader the hardening rules +shared, and it is deleted. Each change below has a test in +`test/workflow_hardening_test.exs` ("WH013 reads the parsed workflow model +(#971)") that fails on the line reader: + +* **A grant is a `permissions:` value.** `permissions: {contents: write}`, + the form WH013's own fix text recommends, now counts as a grant. Text that + only looks like one, such as a `run:` script that writes `contents: write` + into a file, no longer does. +* **Writes, masks and credentials are read from values and keys, not + lines.** A YAML comment naming `git push origin main` is not a write. A + `GH_TOKEN:` line that a script writes into a file is not the job's + credential. A workflow-level `env:` written after `jobs:` is inherited like + one written before it. +* **Flow-style jobs are judged per job.** The line reader could not split + `jobs: { … }`, so it judged the whole file as one body: never credentialed, + and never masked by a step's `continue-on-error`. +* **An unreadable workflow is reported, not guessed at.** As RE001 and WH014 + do, a workflow the reader refuses gets a `:warn` finding at the error's + line if its text names a write. Otherwise it is skipped. + +The tests also gain six WH014 cases (KYAML, flow `with:`, folded and quoted +run scripts, `defaults` placement, expression working directories and +unreadable files). They pass on the WH014 above. + +RE003, RE005 and `WorkflowAudit.check_sha_pinned_toolchain_without_input/1` +still read lines. Their KYAML view, `ResearchExtensions.flow_to_block/1`, +moves back into that module unchanged, so their findings, and the line +numbers the toolchain fixer rewrites by, do not move. Moving them onto the +model stays under #971. + +Measured on 2026-11-16 with Elixir 1.19.4 on OTP 27.3.4.19, comparing main +(6f3f16f) with this change on snapshots of hyperpolymath/hypatia, +rsr-template-repo, choreographic-types and mathesis, and +metadatastician/standards and kyq. Every workflow rule (WorkflowAudit, +WorkflowHardening and ResearchExtensions) gives the same findings before and +after. WH013 does not fire on these trees, so the comparison was repeated on +perturbed copies that make it fire (every `contents: write` narrowed to +`read`, 12 WH013 findings), again with the same findings. + ==== WH014 reads the canonical workflow model (2026-11-16) The masked-report/SARIF-upload rule reads jobs, steps, inputs and working diff --git a/README.adoc b/README.adoc index 31854362..e42bb8f2 100644 --- a/README.adoc +++ b/README.adoc @@ -76,7 +76,7 @@ PatternRegistry ──> TriangleRouter ──> FleetDispatcher ──> DispatchM [cols="1,3"] |=== -| `lib/` | 172 Elixir source files — pipeline, rules, VCL, neural, safety, web, merge orchestration +| `lib/` | 171 Elixir source files — pipeline, rules, VCL, neural, safety, web, merge orchestration | `lib/rules/` | 44 source files: rule modules and their helpers; see link:docs/rules/catalogue.adoc[the catalogue] | `lib/neural/` | 8 networks + blackboard, coordinator, persistence, rebalancer | `lib/vcl/` | VCL query layer over verisim-data: parser, file executor, cache, federation diff --git a/lib/hypatia/workflow_yaml.ex b/lib/hypatia/workflow_yaml.ex deleted file mode 100644 index 3d588847..00000000 --- a/lib/hypatia/workflow_yaml.ex +++ /dev/null @@ -1,637 +0,0 @@ -# SPDX-License-Identifier: MPL-2.0 -# Copyright (c) 2026 Jonathan D.A. Jewell (hyperpolymath) -defmodule Hypatia.WorkflowYaml do - @moduledoc """ - The workflow reader shared by the rules that look inside GitHub Actions - workflows: it reads a workflow the way YAML (and GitHub) does, as far as the - line- and indent-based rules need, while every line keeps its physical line - number so a finding points at the source. - - It was RE001's reader (`Hypatia.Rules.ResearchExtensions`) and moved here - unchanged so other rules (WH013 first) read workflows the same way instead - of each carrying its own approximation. - - * `read_workflow/1` and `flow_to_block/1` read a KYAML (flow-style) - workflow as the equivalent block layout, one line for one line. - * `active_lines/1` numbers the lines, drops comment lines and expands - YAML anchors and aliases (`expand_aliases/1`). - * `workflow_job_lines/1` splits the lines under `jobs:` into jobs, - `job_steps/1` a job into its steps, and `step_key_column/1` gives the - column of a step's own keys. - * `key_line?/2`, `direct_job_field?/2`, `job_field_indent/1`, - `strip_yaml_comment/2`, `nesting_at/2` and `indent_of/1` are the - primitives those are built from. - - It is a static text reader, not a YAML parser: it covers the shapes GitHub - workflows use, and it stays bounded on hostile input (alias expansion stops - after a fixed number of copied lines). - """ - - @doc """ - The workflow's lines as `{line, line_number}`, without comment lines, and - with YAML aliases expanded in place (each expanded line carries the alias's - line number). Physical line numbers are kept, so a finding never points at - a line that a header comment shifted. - """ - def active_lines(content) when is_binary(content) do - content - |> String.split("\n") - |> Enum.with_index(1) - |> Enum.reject(fn {line, _} -> String.starts_with?(String.trim_leading(line), "#") end) - |> expand_aliases() - end - - # YAML anchors and aliases, which GitHub expands in workflows: `*name` - # stands for the node anchored `&name`, so `env: *common`, `run: *script`, - # `<<: *defaults` or `deploy: *build` brings the anchored node into the job - # that holds the alias. Each alias is expanded in place before any job is - # read, and every line it brings carries the alias's line number, so a - # finding points at the alias: - # - # * an alias that is a key's whole value, or a list item, is replaced by - # what follows its anchor on the anchor's line (a scalar, a block - # scalar's `|`, a flow collection), and the anchor's indented lines - # follow at the column of the `*`; - # * a `<<:` merge puts the anchored mapping's entries into the mapping - # that holds the key, at the key's column, minus every entry whose key - # that mapping sets itself: YAML gives the explicit key precedence; - # * an alias inside a flow collection is followed by a copy of the - # anchored lines, deeper than the line. - # - # Anchors and aliases are recognised only where a node starts, outside - # quoted scalars and block-scalar text, so `echo "x *common"` and a `run: |` - # line `ls *common` stay text. The anchor is the nearest one of that name - # above the alias. - @anchor ~r/(?:^\s*(?:-\s+)*|:\s+|[\[{,]\s*)&([A-Za-z0-9_-]+)(?=\s|$)/ - @alias ~r/(?:^\s*(?:-\s+)*|:\s+|[\[{,]\s*)\*([A-Za-z0-9_-]+)\s*(?=$|[,\]}])/ - @merge_key ~r/^(\s*(?:-\s+)?)(?:<<|"<<"|'<<')\s*:/ - @mapping_key ~r/^\s*(?:-\s+)?(?:"([^"]*)"|'([^']*)'|([^\s"'#:][^:]*?))\s*:(?:\s|$)/ - @block_scalar_start ~r/(?:^\s*|[:-]\s+)(?:&[A-Za-z0-9_-]+\s+)?[>|][0-9+-]*\s*$/ - @block_scalar_header ~r/^[>|][0-9+-]*$/ - # Nested anchors can multiply copies exponentially (the "billion laughs" - # file), so expansion stops after this many copied lines per workflow. - @alias_budget 5_000 - - @doc """ - Expand YAML anchors and aliases in numbered lines (`{line, line_number}`), - as GitHub does, within what jobs read: the top-level `jobs:` and `env:` - sections. There each `*name` is replaced in place by the node anchored - `&name` (the nearest one above, wherever it is defined), a `<<:` merge adds - the anchored entries the mapping does not set itself, and every line an - alias brings carries the alias's line number. An alias in any other - top-level section (`on:`, `permissions:`, ...) is returned unchanged. - Anchors and aliases count only where a YAML node starts, never inside - quoted scalars or block-scalar text. Expansion stops - after #{@alias_budget} copied lines, so hostile input costs bounded work. - """ - def expand_aliases(lines) do - codes = yaml_codes(lines) - - anchors = - for {{{line, n}, code}, index} <- lines |> Enum.zip(codes) |> Enum.with_index(), - code != nil, - [_, {at, len}] <- Regex.scan(@anchor, code, return: :index), - node_start?(code, at) do - {binary_part(code, at, len), n, - anchored(lines, index, strip_yaml_comment(line), at + len)} - end - - # Only what jobs read is expanded in place: the jobs and the top-level - # env:. An anchor kept elsewhere is expanded where a job uses it. - {expanded, _} = - lines - |> Enum.zip(codes) - |> Enum.with_index() - |> Enum.flat_map_reduce({nil, @alias_budget}, fn {{{line, _} = entry, code}, index}, - {section, budget} -> - section = - case Regex.run(~r/^["']?([^\s"':]+)["']?\s*:/, line) do - [_, key] -> key - nil -> section - end - - if section in ["jobs", "env"] do - {out, budget} = expand_line(entry, code, lines, index, anchors, budget) - {out, {section, budget}} - else - {[entry], {section, budget}} - end - end) - - expanded - end - - defp expand_aliases(lines, anchors, budget) do - lines - |> Enum.zip(yaml_codes(lines)) - |> Enum.with_index() - |> Enum.flat_map_reduce(budget, fn {{entry, code}, index}, budget -> - expand_line(entry, code, lines, index, anchors, budget) - end) - end - - # One line with its aliases expanded. A line of block-scalar text (code nil) - # holds no alias. - defp expand_line(entry, nil, _lines, _index, _anchors, budget), do: {[entry], budget} - - defp expand_line(entry, _code, _lines, _index, _anchors, budget) when budget <= 0, - do: {[entry], budget} - - defp expand_line({line, n} = entry, code, lines, index, anchors, budget) do - matches = - for [_, {at, len}] <- Regex.scan(@alias, code, return: :index), - node_start?(code, at), - name = binary_part(code, at, len), - {_, _, node} <- - anchors |> Enum.filter(fn {a, m, _} -> a == name and m < n end) |> Enum.take(-1), - do: {at, len, node} - - # The alias that ends the line is the whole value of its key or item. - {value, in_flow} = - case List.last(matches) do - {at, len, _} = last -> - if String.trim(binary_part(code, at + len, byte_size(code) - at - len)) == "", - do: {last, Enum.drop(matches, -1)}, - else: {nil, matches} - - nil -> - {nil, []} - end - - {flow_copies, budget} = - Enum.flat_map_reduce(in_flow, budget, fn {at, _, {inline, block}}, budget -> - indent = indent_of(code) - inline_lines = if inline == "", do: [], else: [{indent_spaces(indent + 2) <> inline, n}] - {copy, budget} = anchored_block(inline, inline_lines ++ block, anchors, budget) - {renumber(reindent(copy, column(code, at - 1)), n), budget} - end) - - case value do - nil -> - {[entry | flow_copies], budget} - - {at, _, {inline, block}} -> - text = strip_yaml_comment(line) - head = String.trim_trailing(binary_part(text, 0, at - 1) <> inline) - {block, budget} = anchored_block(inline, block, anchors, budget - 1) - - block = - case Regex.run(@merge_key, code) do - [_, prefix] -> - merge_column = String.length(prefix) - - block - |> reindent(merge_column) - |> drop_entries(merge_column, sibling_keys(lines, index, merge_column)) - - nil -> - reindent(block, column(code, at - 1)) - end - - # The inline part may hold aliases of its own (`&a [*b, *c]`). - {head_lines, budget} = - if inline == "" or Regex.match?(@block_scalar_header, inline), - do: {[{head, n}], budget}, - else: expand_line({head, n}, yaml_code(head), [{head, n}], 0, anchors, budget) - - {head_lines ++ renumber(block, n) ++ flow_copies, budget} - end - end - - # An anchor's indented lines, their own aliases expanded. Under a block - # scalar's header they are text and are copied as they are. - defp anchored_block(inline, block, anchors, budget) do - if Regex.match?(@block_scalar_header, inline), - do: {block, budget - length(block)}, - else: expand_aliases(block, anchors, budget - length(block)) - end - - # The node an anchor names: what follows the anchor on its own line, and - # every line indented deeper than that line. - defp anchored(lines, index, text, after_name) do - inline = text |> binary_part(after_name, byte_size(text) - after_name) |> String.trim() - indent = indent_of(text) - - block = - lines - |> Enum.drop(index + 1) - |> Enum.take_while(fn {line, _} -> String.trim(line) == "" or indent_of(line) > indent end) - - {inline, block} - end - - # The keys the mapping holding line `index` sets at `column` itself. - defp sibling_keys(lines, index, column) do - {before, [_ | after_line]} = Enum.split(lines, index) - - in_mapping = fn ls -> - Enum.take_while(ls, fn {line, _} -> String.trim(line) == "" or indent_of(line) >= column end) - end - - (in_mapping.(Enum.reverse(before)) ++ in_mapping.(after_line)) - |> Enum.filter(fn {line, _} -> indent_of(line) == column end) - |> Enum.flat_map(fn {line, _} -> mapping_key(line) end) - |> Enum.reject(&(&1 == "<<")) - end - - # Merged entries, less those whose key the destination mapping sets. - defp drop_entries(lines, column, keys) do - lines - |> Enum.chunk_while( - [], - fn {line, _} = entry, acc -> - if indent_of(line) == column and String.trim(line) != "" and acc != [], - do: {:cont, Enum.reverse(acc), [entry]}, - else: {:cont, [entry | acc]} - end, - fn - [] -> {:cont, []} - acc -> {:cont, Enum.reverse(acc), []} - end - ) - |> Enum.reject(fn [{first, _} | _] -> Enum.any?(mapping_key(first), &(&1 in keys)) end) - |> Enum.concat() - end - - defp mapping_key(line) do - case Regex.run(@mapping_key, strip_yaml_comment(line)) do - nil -> [] - captures -> captures |> tl() |> Enum.reject(&(&1 == "")) |> Enum.take(1) - end - end - - # Each line's YAML structure, for finding anchors and aliases: the line - # without its comment and with quoted scalars blanked, byte offsets kept; - # nil for a line of block-scalar text. - defp yaml_codes(lines) do - {codes, _} = - Enum.map_reduce(lines, nil, fn {line, _}, scalar_indent -> - if scalar_indent && (String.trim(line) == "" or indent_of(line) > scalar_indent) do - {nil, scalar_indent} - else - code = yaml_code(line) - {code, if(Regex.match?(@block_scalar_start, code), do: indent_of(line))} - end - end) - - codes - end - - defp yaml_code(line), do: strip_yaml_comment(line, true) - - # A `,` separates nodes only inside a flow collection; in a plain scalar - # (`run: echo a, *b`) it is text. The name starts at `at`, after `*` or `&`. - defp node_start?(code, at) do - before = code |> binary_part(0, at - 1) |> String.trim_trailing() - not String.ends_with?(before, ",") or nesting_at(code, at) > 0 - end - - defp column(text, byte_offset), do: String.length(binary_part(text, 0, byte_offset)) - defp indent_spaces(n), do: String.duplicate(" ", n) - defp renumber(lines, n), do: Enum.map(lines, fn {line, _} -> {line, n} end) - - defp reindent(lines, target) do - base = - lines - |> Enum.map(&elem(&1, 0)) - |> Enum.reject(&(String.trim(&1) == "")) - |> Enum.map(&indent_of/1) - |> Enum.min(fn -> 0 end) - - Enum.map(lines, fn {line, n} -> - if String.trim(line) == "", - do: {"", n}, - else: {indent_spaces(indent_of(line) - base + target) <> String.trim_leading(line), n} - end) - end - - @doc """ - The column of a step's own properties: just past `- ` on its opening line, - or the indent of the first property under a bare `-`. A `-` followed only - by a comment (`- # publish`) is a bare `-`: YAML ignores the comment, so - the step's keys start on the next line. - """ - def step_key_column([{first, _} | rest]) do - case Regex.run(~r/^(\s*-\s+)[^\s#]/, first) do - [_, prefix] -> - String.length(prefix) - - _ -> - Enum.find_value(rest, indent_of(first) + 2, fn {line, _} -> - String.trim(line) != "" && indent_of(line) - end) - end - end - - def step_key_column([]), do: 0 - - @doc """ - The job's steps, each as its own physical lines. - """ - def job_steps(job_lines) do - with field_indent when is_integer(field_indent) <- job_field_indent(job_lines), - step_lines when step_lines != [] <- lines_in_steps(job_lines, field_indent), - step_indent when is_integer(step_indent) <- step_indent(step_lines, field_indent) do - step_lines - |> Enum.chunk_while( - [], - fn {line, _} = entry, acc -> - if indent_of(line) == step_indent and Regex.match?(~r/^\s*-(?:\s|$)/, line) and - acc != [] do - {:cont, Enum.reverse(acc), [entry]} - else - {:cont, [entry | acc]} - end - end, - fn - [] -> {:cont, []} - acc -> {:cont, Enum.reverse(acc), []} - end - ) - else - _ -> [] - end - end - - @doc """ - The flow-collection depth (`[`/`{` opened minus `]`/`}` closed) of `text` - before byte offset `position`; 0 outside any flow collection. - - Every bracket character counts, including one inside a quoted scalar or a - comment, so pass text with those already blanked (as the reader's own - YAML codes are); on raw YAML the depth can be wrong. - """ - def nesting_at(text, position) do - text - |> binary_part(0, position) - |> String.graphemes() - |> Enum.reduce(0, fn - c, depth when c in ["{", "["] -> depth + 1 - c, depth when c in ["}", "]"] -> depth - 1 - _, depth -> depth - end) - end - - @doc """ - True when the job (as `workflow_job_lines/1` gives it, header line first) - sets `field` as one of its own keys, at the job's field column, and not as - a deeper key of a step or a nested mapping. - """ - def direct_job_field?(job_lines, field) do - case job_field_indent(job_lines) do - nil -> - false - - field_indent -> - Enum.any?(job_lines, fn {line, _} -> - indent_of(line) == field_indent and key_line?(line, field) - end) - end - end - - @doc """ - A line that opens the mapping key `key`: YAML reads `env:`, `"env":` and - `'env':` as the same key, and so does GitHub. - """ - def key_line?(line, key) do - key = Regex.escape(key) - Regex.match?(~r/^\s*(?:#{key}|"#{key}"|'#{key}')\s*:(?:\s|$)/, line) - end - - @doc """ - The column of a job's own keys: the shallowest indent among its non-blank - lines after the header. nil for a job with no body. - """ - def job_field_indent([_header | job_lines]) do - job_lines - |> Enum.map(&elem(&1, 0)) - |> Enum.reject(&(String.trim(&1) == "")) - |> Enum.map(&indent_of/1) - |> Enum.min(fn -> nil end) - end - - def job_field_indent([]), do: nil - - @doc """ - The lines of the job's `steps:` value. Its items may sit deeper than the - key or, as an indentless sequence YAML also allows, at the key's own - column: `steps:` then `- run: …` directly beneath it. - """ - def lines_in_steps(job_lines, field_indent) do - case Enum.find_index(job_lines, fn {line, _} -> - indent_of(line) == field_indent and key_line?(line, "steps") - end) do - nil -> - [] - - index -> - job_lines - |> Enum.drop(index + 1) - |> Enum.take_while(fn {line, _} -> - String.trim(line) == "" or indent_of(line) > field_indent or - (indent_of(line) == field_indent and sequence_item?(line)) - end) - end - end - - @doc """ - The column of the step items: the shallowest `-` in the steps value, which - is the key's own column for an indentless sequence. - """ - def step_indent(step_lines, field_indent) do - step_lines - |> Enum.filter(fn {line, _} -> - indent_of(line) >= field_indent and sequence_item?(line) - end) - |> Enum.map(fn {line, _} -> indent_of(line) end) - |> Enum.min(fn -> nil end) - end - - defp sequence_item?(line), do: Regex.match?(~r/^\s*-(?:\s|$)/, line) - - @doc """ - A line without its YAML comment: a `#` at the start or after whitespace, - outside a quoted scalar. A quote opens a scalar only where one can start - (after whitespace or a flow indicator), so the apostrophe in `it's` does not. - With `blank_quoted`, each quoted scalar's text becomes spaces of the same - byte length, so what remains is structure at its original offsets. - """ - def strip_yaml_comment(line, blank_quoted \\ false) do - text = fn c -> if blank_quoted, do: String.duplicate(" ", byte_size(c)), else: c end - - line - |> String.graphemes() - |> Enum.reduce_while({[], nil, " ", false}, fn c, {acc, quote, prev, escaped} -> - cond do - quote == "\"" and escaped -> - {:cont, {[text.(c) | acc], quote, c, false}} - - quote == "\"" and c == "\\" -> - {:cont, {[text.(c) | acc], quote, c, true}} - - quote != nil and c == quote -> - {:cont, {[c | acc], nil, c, false}} - - quote != nil -> - {:cont, {[text.(c) | acc], quote, c, false}} - - c == "#" and String.trim(prev) == "" -> - {:halt, {acc, nil, c, false}} - - c in ["'", "\""] and (String.trim(prev) == "" or prev in ["{", "[", ","]) -> - {:cont, {[c | acc], c, c, false}} - - true -> - {:cont, {[c | acc], nil, c, false}} - end - end) - |> elem(0) - |> Enum.reverse() - |> Enum.join() - |> String.trim_trailing() - end - - # The top-level `jobs:` key, quoted or not, optionally carrying YAML node - # properties (an anchor `&all_jobs`, a tag `!!map`) before its block value. - @jobs_key ~r/^(?:jobs|"jobs"|'jobs')\s*:\s*(?:(?:&[A-Za-z0-9_-]+|![^\s#]*)\s*)*(?:#.*)?$/ - - @doc """ - Follow block-style jobs by indentation, retaining physical source lines. - As with the other research rules, this is a local static text analysis. - """ - def workflow_job_lines(lines) do - {_in_jobs, _indent, groups} = - Enum.reduce(lines, {false, nil, []}, fn {line, _} = entry, {in_jobs, indent, groups} -> - cond do - Regex.match?(@jobs_key, line) -> - {true, nil, groups} - - not in_jobs -> - {false, indent, groups} - - Regex.match?(~r/^\S/, line) -> - {false, nil, groups} - - true -> - header = Regex.run(~r/^(\s+)(?:[A-Za-z0-9_-]+|"[^"]+"|'[^']+'):/, line) - width = if header, do: String.length(Enum.at(header, 1)), else: nil - - cond do - width && (is_nil(indent) || width == indent) -> - {true, width, [[entry] | groups]} - - groups != [] -> - [current | rest] = groups - {true, indent, [[entry | current] | rest]} - - true -> - {true, indent, groups} - end - end - end) - - groups |> Enum.reverse() |> Enum.map(&Enum.reverse/1) - end - - @doc """ - Read a workflow for the line- and indent-based rules. A KYAML workflow - (YAML-POLICY Y-3: the estate's base form, `yq -o kyaml`) is rewritten to - the equivalent block layout one line for one line, so physical line - numbers in findings still point at the source. Block YAML is returned - byte-identical. - """ - def read_workflow(path), do: path |> File.read!() |> flow_to_block() - - @doc """ - Rewrite a KYAML document to block layout, preserving the line count: - the root `{` and every closer become blank lines, `key: {`/`key: [` - lose their opener, the first key of a `{` sequence item and each scalar - sequence item gain `- `, trailing commas go, and plain double-quoted - scalars are unquoted. Anything whose first significant line is not a - bare `{` is not KYAML and passes through untouched — block YAML's own - `on: [push]` and `permissions: {}` are never rewritten. - """ - def flow_to_block(content) when is_binary(content) do - lines = String.split(content, "\n") - - if kyaml_root?(lines) do - {out, _stack, _dash} = Enum.reduce(lines, {[], [], false}, &flow_line/2) - out |> Enum.reverse() |> Enum.join("\n") - else - content - end - end - - # True when the first line that is neither blank nor a comment is a bare `{`. - defp kyaml_root?(lines) do - lines - |> Enum.map(&String.trim/1) - |> Enum.find(&(&1 != "" and not String.starts_with?(&1, "#"))) - |> Kernel.==("{") - end - - # Rewrite one KYAML line. The stack holds the open collections (`:map` or - # `:seq`); `dash` marks that the next property opens a sequence item. - defp flow_line(line, {out, stack, dash}) do - trimmed = String.trim(line) - - if trimmed == "" or String.starts_with?(trimmed, "#") do - {[dedent(line) | out], stack, dash} - else - {head, comment} = - case Regex.run(~r/^(.*?),?(\s+#[^"]*)?$/, String.trim_trailing(line)) do - [_, head, comment] -> {head, comment} - [_, head] -> {head, ""} - end - - indent = String.duplicate(" ", indent_of(head)) - body = String.trim(head) - in_seq? = match?([:seq | _], stack) - - cond do - body in ["}", "]"] -> - {["" | out], Enum.drop(stack, 1), false} - - body == "{" -> - {["" | out], [:map | stack], in_seq?} - - String.ends_with?(body, ": {") or String.ends_with?(body, ": [") -> - kind = if String.ends_with?(body, "{"), do: :map, else: :seq - key = String.slice(body, 0..-3//1) - {[emit(indent, key, comment, dash) | out], [kind | stack], false} - - in_seq? -> - item = Regex.replace(~r/^\{\s*(.*?)\s*\}$/, body, "\\1") - {[emit(indent <> " ", unquote_scalar(item), comment, true) | out], stack, false} - - true -> - {[emit(indent, unquote_scalar(body), comment, dash) | out], stack, false} - end - end - end - - # Assemble one block-layout line, two columns left of the KYAML one, with a - # `- ` sequence marker in the two columns just above the key when `dash`. - defp emit(indent, body, comment, dash) do - prefix = if dash, do: String.slice(indent, 2..-1//1) <> "- ", else: indent - dedent(prefix <> body <> (comment || "")) - end - - # Drop the two columns of indentation the KYAML root mapping adds. - defp dedent(" " <> rest), do: rest - defp dedent(line), do: line - - # Unquote a `key: "value"` or bare `"value"` whose value holds no quote or - # escape; anything else keeps its quotes. - defp unquote_scalar(body) do - Regex.replace(~r/^((?:[^\s"':]+:\s+)?)"([^"\\]*)"$/, body, "\\1\\2") - end - - @doc "The number of leading whitespace characters of `line`." - def indent_of(line) do - case Regex.run(~r/^(\s*)/, line) do - [_, ws] -> String.length(ws) - _ -> 0 - end - end -end diff --git a/lib/rules/research_extensions.ex b/lib/rules/research_extensions.ex index 31c853ce..06793915 100644 --- a/lib/rules/research_extensions.ex +++ b/lib/rules/research_extensions.ex @@ -108,10 +108,6 @@ defmodule Hypatia.Rules.ResearchExtensions do alias Hypatia.Workflow alias Hypatia.Workflow.Step - # The rules after RE002 still read lines; they share this primitive with - # Hypatia.WorkflowYaml until they move to the model too (#971). - import Hypatia.WorkflowYaml, only: [indent_of: 1] - # ─── Workflow-file discovery ──────────────────────────────────────── @doc false @@ -1799,16 +1795,108 @@ defmodule Hypatia.Rules.ResearchExtensions do } end - # ─── KYAML (flow-style) workflows ─────────────────────────────────── - - # ─── KYAML (flow-style) workflows ──────────────────── + # ─── KYAML (flow-style) workflows, for the line-based rules ───────── - # Kept here for existing callers; the reader is Hypatia.WorkflowYaml. + # RE003 and RE005 still read workflow text line by line, and so does + # `WorkflowAudit.check_sha_pinned_toolchain_without_input/1`, whose findings + # feed a fixer that rewrites files by line. Until they read + # `Hypatia.Workflow` (#971), a KYAML workflow is shown to them as its block + # layout, one line for one line. The rules that read the model (RE001, RE002, + # WH013, WH014) do not use this. @doc false - defdelegate read_workflow(path), to: Hypatia.WorkflowYaml - + def read_workflow(path), do: path |> File.read!() |> flow_to_block() + + # Rewrite a KYAML document to block layout, preserving the line count: + # the root `{` and every closer become blank lines, `key: {`/`key: [` + # lose their opener, the first key of a `{` sequence item and each scalar + # sequence item gain `- `, trailing commas go, and plain double-quoted + # scalars are unquoted. Anything whose first significant line is not a + # bare `{` is not KYAML and passes through untouched — block YAML's own + # `on: [push]` and `permissions: {}` are never rewritten. @doc false - defdelegate flow_to_block(content), to: Hypatia.WorkflowYaml + def flow_to_block(content) when is_binary(content) do + lines = String.split(content, "\n") + + if kyaml_root?(lines) do + {out, _stack, _dash} = Enum.reduce(lines, {[], [], false}, &flow_line/2) + out |> Enum.reverse() |> Enum.join("\n") + else + content + end + end + + # True when the first line that is neither blank nor a comment is a bare `{`. + defp kyaml_root?(lines) do + lines + |> Enum.map(&String.trim/1) + |> Enum.find(&(&1 != "" and not String.starts_with?(&1, "#"))) + |> Kernel.==("{") + end + + # Rewrite one KYAML line. The stack holds the open collections (`:map` or + # `:seq`); `dash` marks that the next property opens a sequence item. + defp flow_line(line, {out, stack, dash}) do + trimmed = String.trim(line) + + if trimmed == "" or String.starts_with?(trimmed, "#") do + {[dedent(line) | out], stack, dash} + else + {head, comment} = + case Regex.run(~r/^(.*?),?(\s+#[^"]*)?$/, String.trim_trailing(line)) do + [_, head, comment] -> {head, comment} + [_, head] -> {head, ""} + end + + indent = String.duplicate(" ", indent_of(head)) + body = String.trim(head) + in_seq? = match?([:seq | _], stack) + + cond do + body in ["}", "]"] -> + {["" | out], Enum.drop(stack, 1), false} + + body == "{" -> + {["" | out], [:map | stack], in_seq?} + + String.ends_with?(body, ": {") or String.ends_with?(body, ": [") -> + kind = if String.ends_with?(body, "{"), do: :map, else: :seq + key = String.slice(body, 0..-3//1) + {[emit(indent, key, comment, dash) | out], [kind | stack], false} + + in_seq? -> + item = Regex.replace(~r/^\{\s*(.*?)\s*\}$/, body, "\\1") + {[emit(indent <> " ", unquote_scalar(item), comment, true) | out], stack, false} + + true -> + {[emit(indent, unquote_scalar(body), comment, dash) | out], stack, false} + end + end + end + + # Assemble one block-layout line, two columns left of the KYAML one, with a + # `- ` sequence marker in the two columns just above the key when `dash`. + defp emit(indent, body, comment, dash) do + prefix = if dash, do: String.slice(indent, 2..-1//1) <> "- ", else: indent + dedent(prefix <> body <> (comment || "")) + end + + # Drop the two columns of indentation the KYAML root mapping adds. + defp dedent(" " <> rest), do: rest + defp dedent(line), do: line + + # Unquote a `key: "value"` or bare `"value"` whose value holds no quote or + # escape; anything else keeps its quotes. + defp unquote_scalar(body) do + Regex.replace(~r/^((?:[^\s"':]+:\s+)?)"([^"\\]*)"$/, body, "\\1\\2") + end + + # The width of a line's leading whitespace, in characters. + defp indent_of(line) do + case Regex.run(~r/^(\s*)/, line) do + [_, ws] -> String.length(ws) + _ -> 0 + end + end # ─── Internals ────────────────────────────────────────────────────── diff --git a/lib/rules/workflow_hardening.ex b/lib/rules/workflow_hardening.ex index a34ba6de..6a1be055 100644 --- a/lib/rules/workflow_hardening.ex +++ b/lib/rules/workflow_hardening.ex @@ -75,9 +75,11 @@ defmodule Hypatia.Rules.WorkflowHardening do ## Implementation note - WH014 reads job/step structure and scalar values through `Hypatia.Workflow`. - Its shell analysis remains a bounded literal-path analysis. The remaining - rules still use the legacy readers below; their migration is tracked in #971. + WH013 and WH014 read job/step structure and scalar values through + `Hypatia.Workflow`: they judge a job by its own keys and values, and a + workflow the reader refuses is reported, not guessed at. WH014's shell + analysis remains a bounded literal-path analysis. The remaining rules still + use the legacy readers below; their migration is tracked in #971. Those remaining rules use regex-on-YAML-text rather than full YAML parsing because: @@ -96,7 +98,8 @@ defmodule Hypatia.Rules.WorkflowHardening do require Logger - alias Hypatia.{Kyaml, Workflow, WorkflowYaml} + alias Hypatia.{Kyaml, Workflow} + alias Hypatia.Workflow.Step # ─── Workflow-file discovery ──────────────────────────────────────── @@ -1136,60 +1139,155 @@ defmodule Hypatia.Rules.WorkflowHardening do Masking stays a defect there: `git push … 2>/dev/null || echo` cannot turn the job red when the credential expires or loses scope. That shape is reported at `:warn` with its own fix recipe, because no grant is missing. + + ## Reading the workflow + + The workflow is read through `Hypatia.Workflow`: grants are `permissions:` + values, writes and masks are read from the jobs' string values, a + credential from their `token:`/`GH_TOKEN:`/`GITHUB_TOKEN:` keys, and a + step's own `continue-on-error: true`, in any layout. A workflow the reader + refuses is reported at `:warn`, at the error's line, when its text names a + write, and skipped otherwise. """ def wh013_permission_starved_write(repo_path) do repo_path |> workflow_files() |> Enum.flat_map(fn path -> - content = WorkflowYaml.read_workflow(path) + bytes = File.read!(path) rel = Path.relative_to(path, repo_path) - {credentialed, token_writes} = - content - |> writing_jobs() - |> Enum.split_with(& &1.credentialed?) + case Workflow.parse(bytes, rel) do + {:ok, nil} -> + [] - masked_credentialed = - if Enum.any?(credentialed, &masked_job?/1), - do: [wh013_masked_credentialed_finding(rel)], - else: [] + {:ok, workflow} -> + wh013_workflow(rel, workflow) - wh013_starved_findings(rel, content, token_writes) ++ masked_credentialed + # A workflow the reader refuses is reported when its text names a + # write, and skipped otherwise: its grants and jobs cannot be read, + # and a silent skip of a writing workflow would read as a pass. + {:error, error} -> + if performs_contents_write?(bytes), do: [wh013_unreadable(rel, error)], else: [] + end end) end - # Each job that performs a write, as `%{text, steps, credentialed?}`, read - # by the shared workflow reader: comment lines dropped, aliases expanded, - # steps only under the job's own `steps:` key. A job is credentialed by its - # own text or by a workflow-level `env:` it inherits. A file whose jobs the - # reader cannot split is judged as one body and is never credentialed: one - # job's PAT must not excuse another job's write. - defp writing_jobs(content) do - case content |> WorkflowYaml.active_lines() |> WorkflowYaml.workflow_job_lines() do - [] -> - if performs_contents_write?(content), - do: [%{text: content, steps: [], credentialed?: false}], - else: [] + defp wh013_workflow(rel, workflow) do + {credentialed, token_writes} = + workflow + |> writing_jobs() + |> Enum.split_with(& &1.credentialed?) - jobs -> - inherited? = own_credential_job?(workflow_preamble(content)) + masked_credentialed = + if Enum.any?(credentialed, &masked_job?/1), + do: [wh013_masked_credentialed_finding(rel)], + else: [] - for job <- jobs, text = lines_text(job), performs_contents_write?(text) do - %{ - text: text, - steps: WorkflowYaml.job_steps(job), - credentialed?: inherited? or own_credential_job?(text) - } - end + wh013_starved_findings(rel, workflow, token_writes) ++ masked_credentialed + end + + defp wh013_unreadable(rel, error) do + %{ + rule: "WH013", + file: rel, + severity: :warn, + line: error.line, + reason: + "#{rel} could not be read as KYAML (#{error.code} at line #{error.line}, column " <> + "#{error.column}: #{error.message}), so WH013 cannot check that its writes " <> + "(push/commit/release/PR) are granted `contents: write`", + action: :report, + detail: %{ + line: error.line, + fix: + "Rewrite the construct the reader names in the KYAML subset (kyq " <> + "docs/KYAML_SPEC.adoc; standards YAML-POLICY). GitHub Actions refuses some of " <> + "these too, such as the `<<` merge key." + } + } + end + + # Each job that performs a write, as `%{values, steps, credentialed?}`, read + # from the parsed model (`Hypatia.Workflow`): `values` are the job's string + # values (YAML values, so a comment is never one; in a block scalar, lines + # that open with `#` are dropped as shell comments), `steps` its own steps. + # A job is credentialed by its own keys or by a workflow-level key it + # inherits (`env:`). A workflow with no jobs is judged as one body and is + # never credentialed: one job's PAT must not excuse another job's write. + defp writing_jobs(%Workflow{jobs: [], doc: doc}) do + values = scalar_values(doc) + + if writes?(values), + do: [%{values: values, steps: [], credentialed?: false}], + else: [] + end + + defp writing_jobs(%Workflow{doc: doc, jobs: jobs}) do + inherited? = + doc + |> Workflow.entries() + |> Enum.reject(&(&1.key.value == "jobs")) + |> Enum.flat_map(&entry_key_lines/1) + |> Enum.any?(&own_credential_job?/1) + + for job <- jobs, values = scalar_values(job.node), writes?(values) do + %{ + values: values, + steps: job.steps, + credentialed?: inherited? or Enum.any?(key_lines(job.node), &own_credential_job?/1) + } end end - defp lines_text(lines), do: Enum.map_join(lines, "\n", &elem(&1, 0)) + defp writes?(values), do: Enum.any?(values, &performs_contents_write?/1) - # Everything above the top-level `jobs:` key (name, on, permissions, env). - # The key may be quoted (`"jobs":`, `'jobs':`), as WorkflowYaml reads it. - defp workflow_preamble(content) do - content |> String.split(~r/^(?:jobs|"jobs"|'jobs')\s*:/m, parts: 2) |> hd() + # The string values under a node in document order, aliases read through. + # The workflow was read within kyq's node budget, so the walk is bounded. + defp scalar_values(nil), do: [] + + defp scalar_values(node) do + case Kyaml.deref(node) do + %{kind: :map, entries: entries} -> Enum.flat_map(entries, &scalar_values(&1.value)) + %{kind: :seq, items: items} -> Enum.flat_map(items, &scalar_values/1) + %{kind: :string, style: :block, value: value} -> [drop_comment_lines(value)] + %{kind: :string, value: value} -> [value] + _ -> [] + end + end + + defp drop_comment_lines(text) do + text + |> String.split("\n") + |> Enum.reject(&String.starts_with?(String.trim_leading(&1), "#")) + |> Enum.join("\n") + end + + # The keys a job (or the workflow) supplies a credential in, as one line of + # text each (`token: ${{ secrets.FARM_PAT }}`) for `own_credential_job?/1`, + # aliases read through. Keys are YAML keys, quoted or not, at any depth + # (`with:`, `env:`); text inside a scalar (a `run:` script that writes a + # file) is not a key. + @credential_keys ~w(token GH_TOKEN GITHUB_TOKEN) + + defp key_lines(node) do + case node && Kyaml.deref(node) do + %{kind: :map, entries: entries} -> Enum.flat_map(entries, &entry_key_lines/1) + %{kind: :seq, items: items} -> Enum.flat_map(items, &key_lines/1) + _ -> [] + end + end + + defp entry_key_lines(%{key: key, value: value}) do + own = + case {key, Workflow.text(value)} do + {%{kind: :string, value: name}, text} when name in @credential_keys and is_binary(text) -> + ["#{name}: #{text}"] + + _ -> + [] + end + + own ++ key_lines(value) end # `token:` / `GH_TOKEN:` / `GITHUB_TOKEN:` set to a secret other than @@ -1233,17 +1331,18 @@ defmodule Hypatia.Rules.WorkflowHardening do } end - # The grant is judged file-wide (`content`); starvation and masking only + # The grant is judged workflow-wide; starvation and masking only # over the jobs whose writes ride on GITHUB_TOKEN (`token_writes`). - defp wh013_starved_findings(rel, content, token_writes) do + defp wh013_starved_findings(rel, workflow, token_writes) do cond do token_writes == [] -> [] # A grant exists at either level. This rule deliberately does not try - # to prove the grant is on the RIGHT job — that needs a real YAML - # parse, and a false "you are broken" is worse than a missed one. - grants_contents_write_anywhere?(content) -> + # to prove the grant is on the RIGHT job (a job-level block replaces + # the workflow-level one, and an absent block inherits it): a false + # "you are broken" is worse than a missed one. + grants_contents_write_anywhere?(workflow) -> [] # No grant anywhere, and the write is masked so it can never go red. @@ -1297,12 +1396,31 @@ defmodule Hypatia.Rules.WorkflowHardening do @doc """ Return true when `contents: write` (or `write-all`) is granted at the - workflow level or at any job level. + workflow level or at any job level: a `permissions:` value of `write-all`, + or a `permissions:` mapping whose `contents` is `write`, in any layout + (`permissions: {contents: write}` is the same grant as the block form). + Text that only looks like a grant, such as a `run:` script that writes a + workflow file, is not one. + + It takes the parsed model or workflow text. Text the reader refuses grants + nothing. """ def grants_contents_write_anywhere?(content) when is_binary(content) do - Regex.match?(~r/^[ \t]*contents:\s*write\b/m, content) or - Regex.match?(~r/^[ \t]*permissions:\s*write-all\b/m, content) or - Regex.match?(~r/^[ \t]*write-all:\s*true\b/m, content) + case Workflow.parse(content) do + {:ok, %Workflow{} = workflow} -> grants_contents_write_anywhere?(workflow) + _ -> false + end + end + + def grants_contents_write_anywhere?(%Workflow{permissions: permissions, jobs: jobs}) do + Enum.any?([permissions | Enum.map(jobs, & &1.permissions)], &grants_contents_write?/1) + end + + defp grants_contents_write?(nil), do: false + + defp grants_contents_write?(permissions) do + Workflow.text(permissions) == "write-all" or + Workflow.text(Kyaml.get(permissions, "contents")) == "write" end # A write whose failure is swallowed: stderr redirected away, or the @@ -1314,11 +1432,6 @@ defmodule Hypatia.Rules.WorkflowHardening do ~r/(?:git\s+push|git\s+commit|gh\s+pr\s+(?:create|merge)|gh\s+release\s+(?:create|upload|edit))[^\n]*\|\|\s*echo/ ] - # `continue-on-error: true` as a key of the step, quoted or not. The prefix - # (indent, plus `- ` on the step's opening line) is captured so the key's - # column can be compared with the step's own key column. - @step_continue_on_error ~r/^(\s*(?:-\s+)?)(?:continue-on-error|"continue-on-error"|'continue-on-error')\s*:\s*true\b/ - @doc """ Return true when a write operation's failure is masked, so a denied write cannot turn the job red: a shell mask on the write itself, or @@ -1328,40 +1441,33 @@ defmodule Hypatia.Rules.WorkflowHardening do (standards' github-backup-mirror-reusable.yml documents and surfaces it). `content` is a workflow (or a fragment of one). Its steps are read with - `Hypatia.WorkflowYaml`, so only the items of each job's own `steps:` count; - a fragment without `jobs:` is judged by its shell masks alone. + `Hypatia.Workflow`, so only the items of each job's own `steps:` count; a + fragment the reader cannot read as a workflow is judged by its shell masks + alone. """ def masked_write?(content) when is_binary(content) do shell_masked_write?(content) or - content - |> WorkflowYaml.active_lines() - |> WorkflowYaml.workflow_job_lines() - |> Enum.flat_map(&WorkflowYaml.job_steps/1) - |> Enum.any?(&masked_step?/1) - end + case Workflow.parse(content) do + {:ok, %Workflow{jobs: jobs}} -> + Enum.any?(jobs, fn job -> Enum.any?(job.steps, &masked_step?/1) end) - defp masked_job?(%{text: text, steps: steps}) do - shell_masked_write?(text) or Enum.any?(steps, &masked_step?/1) + _ -> + false + end end - defp shell_masked_write?(text), do: Enum.any?(@write_masks, &Regex.match?(&1, text)) - - defp masked_step?(step) do - step_continue_on_error?(step) and performs_contents_write?(lines_text(step)) + defp masked_job?(%{values: values, steps: steps}) do + Enum.any?(values, &shell_masked_write?/1) or Enum.any?(steps, &masked_step?/1) end - # `continue-on-error: true` at the step's own key column. Deeper text (a - # `run: |` script that writes a workflow file, a `with:` value) is not the - # step's flag. - defp step_continue_on_error?(step) do - key_column = WorkflowYaml.step_key_column(step) + defp shell_masked_write?(text), do: Enum.any?(@write_masks, &Regex.match?(&1, text)) - Enum.any?(step, fn {line, _} -> - case Regex.run(@step_continue_on_error, line) do - [_, prefix] -> String.length(prefix) == key_column - nil -> false - end - end) + # `continue-on-error: true` as a key of the step itself, on a step that + # writes. Text deeper in the step (a `run: |` script that writes a workflow + # file, a `with:` value) is not the step's flag. + defp masked_step?(%Step{node: node}) do + match?(%{kind: :bool, value: true}, Kyaml.deref(Kyaml.get(node, "continue-on-error"))) and + writes?(scalar_values(node)) end # ─── Internals ────────────────────────────────────────────────────── diff --git a/test/workflow_hardening_test.exs b/test/workflow_hardening_test.exs index 59ecbdbf..de7cde14 100644 --- a/test/workflow_hardening_test.exs +++ b/test/workflow_hardening_test.exs @@ -1953,6 +1953,8 @@ defmodule Hypatia.Rules.WorkflowHardeningTest do end test "literal working directories distinguish same-named report files" do + # The jq program goes in a block scalar: as a plain scalar, its `: ` + # makes the line invalid YAML, which GitHub (and the reader) refuse. for {input, expected?} <- [{"findings.json", false}, {"../scanner/findings.json", true}] do repo = create_repo_with_workflow(""" @@ -1972,6 +1974,7 @@ defmodule Hypatia.Rules.WorkflowHardeningTest do on_exit(fn -> File.rm_rf!(repo) end) findings = WorkflowHardening.wh014_masked_scanner_upload(repo) + assert Enum.all?(findings, &(&1.severity == :high)) assert findings != [] == expected? end end @@ -2109,4 +2112,353 @@ defmodule Hypatia.Rules.WorkflowHardeningTest do end end end + + # ─── WH013 and WH014 read the parsed model (#971) ─────────────────── + # + # Each test below, except the refused workflow that names no write, fails + # on the line reader the two rules used before (deleted with this change; + # see CHANGELOG.adoc); each pins a place where it read the workflow + # differently from YAML. + + describe "WH013 reads the parsed workflow model (#971)" do + test "the grant check still takes workflow text, and text the reader refuses grants nothing" do + assert WorkflowHardening.grants_contents_write_anywhere?( + "on: push\njobs:\n a:\n permissions: {contents: write}\n runs-on: x\n" + ) + + refute WorkflowHardening.grants_contents_write_anywhere?( + "on: push\njobs:\n a:\n runs-on: x\n steps:\n - run: echo 'contents: write' > p.yml\n" + ) + + refute WorkflowHardening.grants_contents_write_anywhere?("jobs: [unclosed\n") + end + + test "ResearchExtensions.read_workflow/1 stays callable from outside the module" do + path = + Path.join( + System.tmp_dir!(), + "wh013-read-workflow-#{System.unique_integer([:positive])}.yml" + ) + + File.write!(path, "on: push\n") + assert Hypatia.Rules.ResearchExtensions.read_workflow(path) == "on: push\n" + File.rm!(path) + end + + test "a flow-style grant is a grant" do + repo = + create_repo_with_workflow(""" + name: Release + permissions: + contents: read + jobs: + publish: + runs-on: ubuntu-latest + permissions: { contents: write } + steps: + - run: git push origin HEAD + """) + + on_exit(fn -> File.rm_rf!(repo) end) + assert [] = WorkflowHardening.wh013_permission_starved_write(repo) + end + + test "text that only looks like a grant is not one" do + for grant <- ["contents: write", "write-all: true"] do + repo = + create_repo_with_workflow(""" + name: Release + permissions: + contents: read + jobs: + publish: + runs-on: ubuntu-latest + steps: + - run: | + cat > generated.yml <<'YML' + permissions: + #{grant} + YML + git commit -am generated && git push origin HEAD + """) + + on_exit(fn -> File.rm_rf!(repo) end) + + assert [%{severity: :high, fix_recipe: "grant-contents-write-to-writing-job"}] = + WorkflowHardening.wh013_permission_starved_write(repo), + grant + end + end + + test "a YAML comment that names a write is not a write" do + repo = + create_repo_with_workflow(""" + name: CI + permissions: + contents: read + jobs: + test: + runs-on: ubuntu-latest + steps: + - run: mix test # the release job used to git push origin main + """) + + on_exit(fn -> File.rm_rf!(repo) end) + assert [] = WorkflowHardening.wh013_permission_starved_write(repo) + end + + test "a credential a script writes into a file is not the job's credential" do + repo = + create_repo_with_workflow(""" + name: Release + permissions: + contents: read + jobs: + publish: + runs-on: ubuntu-latest + steps: + - run: | + cat > deploy.yml <<'YML' + GH_TOKEN: ${{ secrets.DEPLOY_PAT }} + YML + git push origin HEAD + """) + + on_exit(fn -> File.rm_rf!(repo) end) + + assert [%{severity: :high, fix_recipe: "grant-contents-write-to-writing-job"}] = + WorkflowHardening.wh013_permission_starved_write(repo) + end + + test "flow-style jobs are judged per job, with their credentials and step flags" do + # One job releases with its own PAT; the other pushes with GITHUB_TOKEN + # from a step marked continue-on-error. Written in flow style, the line + # reader judged the file as one body: never credentialed, never masked. + repo = + create_repo_with_workflow(""" + name: Release + permissions: { contents: read } + jobs: { + notes: { runs-on: ubuntu-latest, steps: [ + { env: { GH_TOKEN: "${{ secrets.RELEASE_PAT }}" }, run: "gh release create v1" } ] }, + record: { runs-on: ubuntu-latest, steps: [ + { continue-on-error: true, run: "git push origin HEAD" } ] } + } + """) + + on_exit(fn -> File.rm_rf!(repo) end) + + assert [%{severity: :high, fix_recipe: "grant-contents-write-to-writing-job-and-unmask"}] = + WorkflowHardening.wh013_permission_starved_write(repo) + end + + test "a workflow-level env written after `jobs:` still covers every job" do + repo = + create_repo_with_workflow(""" + name: Release + permissions: + contents: read + jobs: + publish: + runs-on: ubuntu-latest + steps: + - run: gh release create v1 dist/* + env: + GH_TOKEN: ${{ secrets.RELEASE_PAT }} + """) + + on_exit(fn -> File.rm_rf!(repo) end) + assert [] = WorkflowHardening.wh013_permission_starved_write(repo) + end + + test "a writing workflow the reader refuses is reported at the error, not guessed at" do + repo = + create_repo_with_workflow(""" + name: Release + x-defaults: &defaults + runs-on: ubuntu-latest + jobs: + publish: + <<: *defaults + permissions: + contents: write + steps: + - run: git push origin HEAD + """) + + on_exit(fn -> File.rm_rf!(repo) end) + + assert [%{rule: "WH013", severity: :warn, line: 6, reason: reason}] = + WorkflowHardening.wh013_permission_starved_write(repo) + + assert reason =~ "could not be read" + end + + test "a refused workflow that names no write is not reported" do + repo = + create_repo_with_workflow(""" + x-defaults: &defaults + runs-on: ubuntu-latest + jobs: + test: + <<: *defaults + steps: + - run: mix test + """) + + on_exit(fn -> File.rm_rf!(repo) end) + assert [] = WorkflowHardening.wh013_permission_starved_write(repo) + end + end + + describe "WH014 reads the parsed workflow model (#971)" do + test "a KYAML workflow is read, with its flow-style `with:`" do + repo = + create_repo_with_workflow(""" + { + jobs: { + scan: { + "runs-on": "ubuntu-latest", + steps: [ + { + run: "hypatia scan . > findings.json || true", + }, + { + run: "jq '{runs: [{results: .}]}' findings.json > out.sarif", + }, + { + uses: "github/codeql-action/upload-sarif@v4", + with: {sarif_file: "out.sarif"}, + }, + ], + }, + }, + } + """) + + on_exit(fn -> File.rm_rf!(repo) end) + + assert [%{rule: "WH014", severity: :high, detail: %{jobs: ["scan"]}}] = + WorkflowHardening.wh014_masked_scanner_upload(repo) + end + + test "a folded or quoted run script is read as its YAML value" do + for run <- [ + ">-\n hypatia scan . > findings.json\n || true", + ~s("hypatia scan . > findings.json || true"), + "'hypatia scan . > findings.json || true'" + ] do + repo = + create_repo_with_workflow(""" + jobs: + scan: + runs-on: ubuntu-latest + steps: + - run: #{run} + - run: | + jq '{runs: [{results: .}]}' findings.json > out.sarif + - uses: github/codeql-action/upload-sarif@v4 + with: + sarif_file: out.sarif + """) + + on_exit(fn -> File.rm_rf!(repo) end) + + assert [%{rule: "WH014", severity: :high}] = + WorkflowHardening.wh014_masked_scanner_upload(repo), + run + end + end + + test "a job's defaults.run.working-directory counts wherever the key is written" do + for {sarif_file, expected?} <- [{"out.sarif", false}, {"scanner/out.sarif", true}] do + repo = + create_repo_with_workflow(""" + jobs: + scan: + runs-on: ubuntu-latest + steps: + - run: hypatia scan . > findings.json || true + - run: | + jq '{runs: [{results: .}]}' findings.json > out.sarif + - uses: github/codeql-action/upload-sarif@v4 + with: + sarif_file: #{sarif_file} + defaults: + run: + working-directory: scanner + """) + + on_exit(fn -> File.rm_rf!(repo) end) + findings = WorkflowHardening.wh014_masked_scanner_upload(repo) + assert findings != [] == expected?, sarif_file + end + end + + test "a reusable workflow's `working-directory` input is not a default directory" do + repo = + create_repo_with_workflow(""" + on: + workflow_call: + inputs: + working-directory: + type: string + default: . + jobs: + scan: + runs-on: ubuntu-latest + steps: + - run: hypatia scan . > findings.json || true + - run: | + jq '{runs: [{results: .}]}' findings.json > out.sarif + - uses: github/codeql-action/upload-sarif@v4 + with: + sarif_file: out.sarif + """) + + on_exit(fn -> File.rm_rf!(repo) end) + assert [%{rule: "WH014"}] = WorkflowHardening.wh014_masked_scanner_upload(repo) + end + + test "a step whose working directory is an expression is not followed" do + repo = + create_repo_with_workflow(""" + jobs: + scan: + runs-on: ubuntu-latest + steps: + - working-directory: ${{ inputs.dir }} + run: hypatia scan . > findings.json || true + - run: | + jq '{runs: [{results: .}]}' findings.json > out.sarif + - uses: github/codeql-action/upload-sarif@v4 + with: + sarif_file: out.sarif + """) + + on_exit(fn -> File.rm_rf!(repo) end) + assert [] = WorkflowHardening.wh014_masked_scanner_upload(repo) + end + + test "an uploading workflow the reader refuses is reported at the error" do + repo = + create_repo_with_workflow(""" + x-defaults: &defaults + runs-on: ubuntu-latest + jobs: + scan: + <<: *defaults + steps: + - run: hypatia scan . > findings.json || true + - uses: github/codeql-action/upload-sarif@v4 + with: + sarif_file: findings.json + """) + + on_exit(fn -> File.rm_rf!(repo) end) + + assert [%{rule: "WH014", severity: :warn, line: 5}] = + WorkflowHardening.wh014_masked_scanner_upload(repo) + end + end end