Dedent list items when quoting them into fields

CHANGELOG.md

@@ -10,6 +10,7 @@
 - A draft is filled in by `in2lambda draft question add`, `in2lambda draft part add QUESTION` and `in2lambda draft question solution QUESTION`. Each takes `--text` to copy the wording out of the frozen source, as a block id such as `b3` or as lines such as `s10:14`, or `--literal TEXT` where the source does not say it in a form the field can take, which records the field as edited and written by layer 4 rather than 3. Question and part numbers are worked out from the fields already written rather than given, so a replay arrives at the same ids. `in2lambda draft split block BLOCK AT` cuts a block the parser made one of two things into `b3a` and `b3b`, so that each half can be quoted on its own. A command writing a field that is already written, or quoting lines another field was taken from, is refused naming both fields.
 - `in2lambda draft field replace FIELD OLD NEW` changes the wording inside a field that is already written, for the faults only an edit can fix - a brace the OCR dropped out of some maths, which no range of the source says correctly. OLD has to occur in the field exactly once, or the command is refused saying how many times it occurs; `--regex` reads it as a regular expression and NEW as what to replace it with. The field is left quoting the lines it was taken from, at the layer that wrote it, but recorded as edited and by whoever replaced the wording, so the change can be shown against the source.
 - `in2lambda spec run SPEC` runs a YAML file of selectors over the frozen source: it says which blocks are questions, parts and solutions, which to ignore, what to strip off the front of each one, and which of the four filters lays the solutions out. It fills in the draft's fields with the markdown of the lines each was taken from, records the spec's name and hash in the log so a replay runs the same file, and reports every block it made nothing of. Running an edited spec over a draft it has already filled in is refused, as freezing a document that has changed is: `in2lambda source add --start-over` begins the draft again. Reading a spec needs pyyaml, which the `convert` extra now installs alongside panflute. See [the spec page](https://lambda-feedback.github.io/in2lambda/spec.html) for the selectors and layouts.
+- A field quoted out of a list item is now dedented as commonmark reads the item: the marker comes off the first line and as much of the same width off every line under it. So a question written `1.  ` no longer carries its number, a continuation line no longer arrives indented far enough to be rendered as a code block, and a spec's `strip` is left with what pandoc does not read as a marker. Values written by `in2lambda spec run`, `in2lambda draft question add`, `in2lambda draft part add` and `in2lambda draft question solution` change accordingly; the ranges behind them still name the same source lines.
 - `in2lambda validate` checks a draft over as a whole and writes what it finds into it as a `report`: source blocks in no field and not marked ignore, two fields taken from the same lines, gaps in the numbering of the questions or their parts, parts nothing answers, and fields holding nothing. Each finding names the field and the lines it is about, so it can be acted on without reading the draft. Finding something is not a failure and the command still exits 0; the report is replaced by the next run of the checks and dropped by the next command that changes the draft, since it describes the draft as it stood.
 - `in2lambda build` writes the draft in this directory out as a Lambda Feedback set: one question per `qN.text` field, holding the parts written for it and the worked solutions, with the images those fields refer to under `media/`, as `in2lambda convert` writes a set - a field naming an image that is not beside the draft is refused saying which file is missing, since the checks read the draft and not the folder it is in, and a question's own solution written beside a solution for every part it has becomes a part of its own holding just that solution, as `convert` pairs them up. It is refused unless `in2lambda validate` has been run since the draft last changed - every command that changes one drops its report - and found nothing, and the refusal prints what the report says so it can be acted on without opening the draft. `in2lambda render` writes each question as a PDF instead, compiled as Lambda Feedback's own PDF generator compiles it, which needs pandoc and xelatex; it is gated on nothing, since looking at a draft is how what the checks found gets fixed. Both take `-o/--out`, as `convert` does.
 - Importing `in2lambda.katex_convert` no longer writes a file called `log` into the working directory. What it has to say about a converted expression goes to the `in2lambda.katex_convert` logger, which is silent unless the application configures logging.

docs/source/spec.md

@@ -28,7 +28,7 @@ $ in2lambda spec run spec.yaml
 question: Header level=2 text~'^Question'
 part:     ListItem
 solution: after Header text=Solutions, label~'^\d+(\([a-z]\))?$'
-strip:    ['^#+ ', '^\([a-z]\) ', '^\d+(\([a-z]\))? ']
+strip:    ['^#+ ', '^\d+(\([a-z]\))? ']
 ignore:   Header level=1
 layout:   PartsSepSol
 ```
@@ -40,8 +40,9 @@ layout:   PartsSepSol
 - **`ignore`** selects the blocks that are none of them - a running header, a page of
   instructions - and marks them as `in2lambda draft mark ignore` would, so they are not reported
   as left out.
-- **`strip`** is a list of patterns taken off the front of every value: the `(a) ` or `1. ` that
-  labels a part in the document, but not in the question.
+- **`strip`** is a list of patterns taken off the front of every value: the `Q1. ` or
+  `Solution: ` that labels a block in the document, but not in the question. A list marker is
+  not one of them, since a value quoted out of a list item arrives dedented.
 - **`predicates`** names a Python file beside the spec, for the selectors that cannot say what
   they mean in constraints alone. See below.
 - **`layout`** is one of the [filters](filters/index), and says which solution answers which
@@ -76,9 +77,11 @@ A regular expression goes in single quotes. YAML reads `\(` inside double quotes
 and complains, and `'^\([a-z]\)'` is the same string without the argument.
 
 A selector matches what **pandoc** makes of the document, while a field holds the **markdown** of
-the lines it came from. That is worth knowing in two places: a part written `(a) Find the load.`
-is a `ListItem`, because pandoc reads `(a)` as a list marker, and `strip` still has to take the
-`(a) ` off the front of the value, because the line it was copied from still has it.
+the lines it came from. That is worth knowing where a part is written `(a) Find the load.`: it is
+a `ListItem`, because pandoc reads `(a)` as a list marker, and the value comes dedented the way
+pandoc reads the item - the marker off the first line and as much of the same width off every
+line under it - so `strip` is only for what pandoc does not read as a marker, the `Q1. ` and the
+`Solution: `.
 
 ## Predicates
 

in2lambda/draft/__init__.py

@@ -24,6 +24,7 @@
     _elements,
     _require_conversion_tools,
     blocks,
+    dedented,
     frozen,
     save,
     serialise,
@@ -425,13 +426,29 @@ def _fill(
     return record(
         draft,
         key,
-        "\n".join(markdown.splitlines()[start - 1 : end]),
+        _quoted(draft, markdown, start, end),
         layer=3,
         ranges=[[start, end]],
         by=by,
     )
 
 
+def _quoted(draft: dict[str, Any], markdown: str, start: int, end: int) -> str:
+    """Lines of the frozen source as a field takes them.
+
+    Lines quoted out of a list item are dedented by the item's own indentation, which
+    is the markdown's rather than the author's; the range is still the source lines.
+    The block the lines fall in says whether they are, rather than the text itself, so
+    that a paragraph reading like a list item is quoted as it is written.
+    """
+    text = "\n".join(markdown.splitlines()[start - 1 : end])
+    # Blocks do not overlap, so the one the first line falls in is the one the lines are
+    # part of - a nested item among them included, since only a top-level item is a
+    # block of its own and a range is how one of those is quoted.
+    block = next((b for b in draft["blocks"] if b["start"] <= start <= b["end"]), None)
+    return dedented(text) if block and block["type"] == "list item" else text
+
+
 def _next(draft: dict[str, Any], prefix: str) -> str:
     """The first of ``{prefix}1``, ``{prefix}2``... the draft has no text for.
 

in2lambda/source/__init__.py

@@ -344,6 +344,48 @@ def blocks(markdown: str) -> list[Block]:
     return [block for block, _ in _elements(markdown)]
 
 
+_MARKER = re.compile(r" *(?:[-+*]|\(?(?:\d+|[ivxlcdm]+|[IVXLCDM]+|[A-Za-z])[.)]) {1,4}")
+"""A list item's marker on its first line, as `commonmark_x` reads one."""
+
+
+def dedented(text: str) -> str:
+    r"""Some lines of a list item, with the item's own indentation off every one.
+
+    A field quoted out of a list item would otherwise carry the marker and the
+    continuation indent the markdown needed to hold it together, and four leading
+    spaces after a blank line are a code block wherever the field is rendered.
+
+    Args:
+        text: The lines as the source writes them, the first of them holding the
+            item's marker.
+
+    Returns:
+        The same lines with the marker off the first and as much of the same width
+        off each of the rest as it has to give, so that a list nested inside the item
+        keeps its own relative indent. Text whose first line has no marker on it comes
+        back unchanged, but a paragraph reading like one - ``A. Smith says`` - would be
+        dedented, so what this is called on is decided by the block's type rather than
+        by its text.
+
+    Examples:
+        >>> from in2lambda.source import dedented
+        >>> dedented("1.  A person walks\n    to the edge.")
+        'A person walks\nto the edge.'
+        >>> dedented("    (a) Find the speed\n        afterwards.")
+        'Find the speed\nafterwards.'
+        >>> dedented("Some words\n  wrapped.")
+        'Some words\n  wrapped.'
+    """
+    if (marker := _MARKER.match(text)) is None:
+        return text
+    width = marker.end()
+    first, *rest = text.split("\n")
+    return "\n".join(
+        [first[width:]]
+        + [line[min(width, len(line) - len(line.lstrip(" "))) :] for line in rest]
+    )
+
+
 def _elements(markdown: str) -> list[tuple[Block, Any]]:
     """Every block of some markdown, each beside the panflute element it was taken from.
 

in2lambda/spec/__init__.py

@@ -34,7 +34,7 @@
 from typing import Any, NamedTuple, Optional
 
 from in2lambda.filters import builtin_filters
-from in2lambda.source import Block, SourceError
+from in2lambda.source import Block, SourceError, dedented
 
 _KEYS = ("question", "part", "solution", "strip", "ignore", "layout", "predicates")
 """Everything a spec may say. Anything else in one is a typo, and is refused as one."""
@@ -547,6 +547,10 @@ def _stripped(spec: Spec, lines: list[str], block: Block) -> str:
     emphasis and the images in a question survive into the field.
     """
     text = "\n".join(lines[block.start - 1 : block.end])
+    # The list marker and the indent under it are the markdown's, not the author's, so
+    # they come off before the spec's patterns, which are for what is left.
+    if block.type == "list item":
+        text = dedented(text)
     for pattern in spec.strip:
         text = pattern.sub("", text)
     return text.strip()

tests/fixtures/drafts/README.md

@@ -28,6 +28,9 @@ into `media/` - the only folder here with a file the fields point at.
 `question_solution_beside_part_solutions` is the only one filled in by a spec rather than by
 commands one at a time, and the only one whose question has two parts: both are answered by the
 spec's own solutions, so the `question solution` after it answers nothing, and the export carries
-it as a part of its own rather than dropping the wording. `question_without_parts` is a question
+it as a part of its own rather than dropping the wording. `nested_list` is a numbered question with two lettered parts nested inside it, quoted by line
+range because only the top-level item is a block: each field is dedented by its own depth, four
+spaces for the question and eight for the parts, while its range still names the source lines.
+`question_without_parts` is a question
 and nothing else, which the checks have nothing to say about: it is here because a question with
 no parts is what the export has to write out as an empty part rather than as the template's.

tests/fixtures/drafts/nested_list/commands.json

@@ -0,0 +1,32 @@
+[
+  {
+    "args": {
+      "block": "b1"
+    },
+    "by": "tests",
+    "command": "mark ignore"
+  },
+  {
+    "args": {
+      "text": "s3:4"
+    },
+    "by": "tests",
+    "command": "question add"
+  },
+  {
+    "args": {
+      "question": "q1",
+      "text": "s6:7"
+    },
+    "by": "tests",
+    "command": "part add"
+  },
+  {
+    "args": {
+      "question": "q1",
+      "text": "s9"
+    },
+    "by": "tests",
+    "command": "part add"
+  }
+]

tests/fixtures/drafts/nested_list/expected.json

@@ -0,0 +1,50 @@
+{
+  "b1.ignore": {
+    "by": "tests",
+    "edited": false,
+    "layer": 3,
+    "ranges": [
+      [
+        1,
+        1
+      ]
+    ],
+    "value": true
+  },
+  "q1.p1.text": {
+    "by": "tests",
+    "edited": false,
+    "layer": 3,
+    "ranges": [
+      [
+        6,
+        7
+      ]
+    ],
+    "value": "Find the angular speed\nafterwards."
+  },
+  "q1.p2.text": {
+    "by": "tests",
+    "edited": false,
+    "layer": 3,
+    "ranges": [
+      [
+        9,
+        9
+      ]
+    ],
+    "value": "Find the energy lost."
+  },
+  "q1.text": {
+    "by": "tests",
+    "edited": false,
+    "layer": 3,
+    "ranges": [
+      [
+        3,
+        4
+      ]
+    ],
+    "value": "A person walks from the centre\nto the edge of a horizontal turntable."
+  }
+}

tests/fixtures/drafts/nested_list/report.json

@@ -0,0 +1,39 @@
+[
+  {
+    "check": "uncovered",
+    "field": "b2",
+    "message": "b2 (lines 5-5, 8-8) is in no field and not marked ignore.",
+    "ranges": [
+      [
+        5,
+        5
+      ],
+      [
+        8,
+        8
+      ]
+    ]
+  },
+  {
+    "check": "no-solution",
+    "field": "q1.p1",
+    "message": "q1.p1 (lines 6-7) has no solution: neither q1.p1.solution nor q1.solution is written.",
+    "ranges": [
+      [
+        6,
+        7
+      ]
+    ]
+  },
+  {
+    "check": "no-solution",
+    "field": "q1.p2",
+    "message": "q1.p2 (lines 9-9) has no solution: neither q1.p2.solution nor q1.solution is written.",
+    "ranges": [
+      [
+        9,
+        9
+      ]
+    ]
+  }
+]

tests/fixtures/drafts/nested_list/source.md

@@ -0,0 +1,9 @@
+# Problem sheet 9
+
+1.  A person walks from the centre
+    to the edge of a horizontal turntable.
+
+    (a) Find the angular speed
+        afterwards.
+
+    (b) Find the energy lost.

tests/fixtures/drafts/part_without_solution/expected.json

@@ -21,7 +21,7 @@
         5
       ]
     ],
-    "value": "(a) Give the drag coefficient you used."
+    "value": "Give the drag coefficient you used."
   },
   "q1.text": {
     "by": "tests",

tests/fixtures/drafts/question_solution_beside_part_solutions/commands.json

@@ -1,7 +1,7 @@
 [
   {
     "args": {
-      "hash": "sha256:de8814ff58a658669eadd41c48481a5082b2bfd4242d43ad751ec79a66640108",
+      "hash": "sha256:fe12526ad18f41fd98e31ab256511bcb2d16009720acce193560893637c30639",
       "spec": "spec.yaml"
     },
     "by": "tests",

tests/fixtures/drafts/question_solution_beside_part_solutions/spec.yaml

@@ -1,6 +1,6 @@
 question: Para text~'^Q\d+\.'
 part:     ListItem
 solution: Para text~'^Solution:'
-strip:    ['^Q\d+\. ', '^\([a-z]\) ', '^Solution: ']
+strip:    ['^Q\d+\. ', '^Solution: ']
 ignore:   Header
 layout:   PartPartSolSol

tests/fixtures/specs/README.md

@@ -24,5 +24,8 @@ bold each of them starts with, which is markup rather than text. Its log entry n
 and hashes it as it does the spec, which is what makes a changed predicate refuse to replay.
 
 A selector matches what pandoc parses, and a field holds the markdown of the lines it was taken
-from. That is why a part written `(a) Find the load.` is a `ListItem` - pandoc reads `(a)` as a
-list marker - while `strip` still has to take the `(a) ` off the front of the value.
+from, dedented where those lines are a list item's: `numbered_questions` is the sheet whose
+questions are written `1.  ` and run on over several lines, and its spec strips nothing but the
+`Solution: `, so it pins that the marker and the indent under it come off by themselves. That
+is why no spec here strips a `(a) ` - pandoc reads `(a)` as a list marker - while the `Q1. `
+and the `Solution: `, which it does not, are still `strip`'s to take off.

tests/fixtures/specs/numbered_questions/expected.json

@@ -0,0 +1,62 @@
+{
+  "b1.ignore": {
+    "by": "tests",
+    "edited": false,
+    "layer": 1,
+    "ranges": [
+      [
+        1,
+        1
+      ]
+    ],
+    "value": true
+  },
+  "q1.solution": {
+    "by": "tests",
+    "edited": false,
+    "layer": 1,
+    "ranges": [
+      [
+        8,
+        8
+      ]
+    ],
+    "value": "angular momentum is conserved, so $\\omega$ falls."
+  },
+  "q1.text": {
+    "by": "tests",
+    "edited": false,
+    "layer": 1,
+    "ranges": [
+      [
+        3,
+        6
+      ]
+    ],
+    "value": "A person walks from the centre\nto the edge of a horizontal turntable.\n\n$$I = \\frac{1}{2} M R^2$$"
+  },
+  "q2.solution": {
+    "by": "tests",
+    "edited": false,
+    "layer": 1,
+    "ranges": [
+      [
+        12,
+        12
+      ]
+    ],
+    "value": "$\\Delta E = \\frac{1}{2} (I_1 \\omega_1^2 - I_2 \\omega_2^2)$."
+  },
+  "q2.text": {
+    "by": "tests",
+    "edited": false,
+    "layer": 1,
+    "ranges": [
+      [
+        10,
+        10
+      ]
+    ],
+    "value": "Find the energy lost as the person walks out."
+  }
+}