Quote a source range into an existing field (t50)

Quote a source range into an existing field

Author: peterbjohnson

CHANGELOG.md

@@ -9,6 +9,7 @@
 - A draft now holds a `log` of every command that changed it and a `fields` map of what those commands wrote, each field recording which layer wrote it (1 a spec, 2 a predicate, 3 a line range, 4 a literal), the source ranges it was copied from, whether it has been edited and by whom. `in2lambda draft mark ignore BLOCK` is the first such command, and `in2lambda draft replay` rebuilds the draft from the frozen markdown and the log, refusing unless what it builds is the draft that is there, byte for byte. A draft written before this has no `log` in it and is refused as one nothing here wrote; `in2lambda source add --start-over` freezes the document again.
 - A draft is filled in by `in2lambda draft question add`, `in2lambda draft part add QUESTION` and `in2lambda draft question solution QUESTION`. Each takes `--text` to copy the wording out of the frozen source, as a block id such as `b3` or as lines such as `s10:14`, or `--literal TEXT` where the source does not say it in a form the field can take, which records the field as edited and written by layer 4 rather than 3. Question and part numbers are worked out from the fields already written rather than given, so a replay arrives at the same ids. `in2lambda draft split block BLOCK AT` cuts a block the parser made one of two things into `b3a` and `b3b`, so that each half can be quoted on its own. A command writing a field that is already written, or quoting lines another field was taken from, is refused: the first naming the field, the second naming both.
 - `in2lambda draft field replace FIELD OLD NEW` changes the wording inside a field that is already written, for the faults only an edit can fix - a brace the OCR dropped out of some maths, which no range of the source says correctly. OLD has to occur in the field exactly once, or the command is refused saying how many times it occurs; `--regex` reads it as a regular expression and NEW as what to replace it with. The field is left quoting the lines it was taken from, at the layer that wrote it, but recorded as edited and by whoever replaced the wording, so the change can be shown against the source.
+- `in2lambda draft field set FIELD --text RANGE` quotes lines of a frozen source into a field that is already written, for a field quoted from the wrong lines - a spec matching the label line `Q4` alone writes an empty `q4.text`, which `in2lambda validate` reports as empty. The field is written again at layer 3, with the range of the lines quoted and `edited` false, as `in2lambda draft question add --text` writes one; the range it named before is dropped, and `in2lambda validate` reports those lines as in no field. There is no `--literal`: `in2lambda draft field replace` is the command for text no line of the source says.
 - `in2lambda spec run SPEC` runs a YAML file of selectors over the frozen source: it says which blocks are questions, parts and solutions, which to ignore, what to strip off the front of each one, and which of the four filters lays the solutions out. It fills in the draft's fields with the markdown of the lines each was taken from, records the spec's name and hash in the log so a replay runs the same file, and reports every block it made nothing of. Running an edited spec over a draft it has already filled in is refused, as freezing a document that has changed is: `in2lambda source add --start-over` begins the draft again. Reading a spec needs pyyaml, which the `convert` extra now installs alongside panflute. See [the spec page](https://lambda-feedback.github.io/in2lambda/spec.html) for the selectors and layouts.
 - A spec's `question`, `part`, `solution` and `ignore` now each take a list of selectors as well as one selector, written as a YAML list under the key, and a block has that role where any one of them matches it. A run that would write a field twice - a document of nothing but solutions has more solutions than the layout has questions to answer - now leaves the second block in no field and reports it, naming the field and the block that holds it, instead of refusing the whole run. In the Python API, `in2lambda.spec.Spec` holds a list of selectors per role rather than one or None, and `in2lambda.spec.fields` returns a third list, of `in2lambda.spec.Doubled`, beside its fields and its ignored blocks.
 - A field quoted out of a list item is now dedented as commonmark reads the item: the marker comes off the first line and as much of the same width off every line under it. So a question written `1.  ` no longer carries its number, a continuation line no longer arrives indented far enough to be rendered as a code block, and a spec's `strip` is left with what pandoc does not read as a marker. Values written by `in2lambda spec run`, `in2lambda draft question add`, `in2lambda draft part add` and `in2lambda draft question solution` change accordingly; the ranges behind them still name the same source lines.

docs/source/drafts.md

@@ -361,6 +361,15 @@ The layer and the ranges are left as they were, so the field still names the lin
 from, and `edited` says the field no longer holds what those lines say. The wording replaced has
 to occur in the field exactly once, and `--regex` reads it as a regular expression.
 
+Where the source does say the wording and the field was quoted from the wrong lines - a spec
+matching the label line `Q4` alone writes an empty `q4.text`, which `in2lambda validate` reports -
+`in2lambda draft field set q4.text --text b3` quotes block `b3` into the field instead. The field
+is written again at layer 3, with the range of the lines quoted and `edited` false, as
+`in2lambda draft question add --text` writes one. The lines the field named before are dropped,
+and `in2lambda validate` reports them as in no field until `in2lambda draft mark ignore` says they
+are nothing to take a question from. `in2lambda draft field set` takes no `--literal`:
+`in2lambda draft field replace` is the command for text no line of the source says.
+
 ## Replay and check again
 
 ```bash

in2lambda/draft/__init__.py

@@ -155,15 +155,16 @@ def record(
 
     Raises:
         AlreadyFilled: the field is written already, or the lines it was to be copied
-            from are where another field of the same source came from. Nothing here
-            writes a field twice - `field replace` changes the wording of one rather
-            than writing it again - so either is a mistake, and worth naming both
-            halves of.
+            from are where another field of the same source came from. A command that
+            fills a field writes it once - `field replace` changes the wording of one
+            and `field set` quotes other lines into one - so either is a mistake, and
+            worth naming both halves of.
     """
     if key in draft["fields"]:
         raise AlreadyFilled(
-            f"{key} is already written, and no command here writes a field twice. Run "
-            "in2lambda draft field replace to change the wording it holds, or "
+            f"{key} is already written, and no command fills a field twice. Run "
+            "in2lambda draft field replace to change the wording it holds, "
+            "in2lambda draft field set to quote other lines into it, or "
             "in2lambda source add --start-over to begin the draft again."
         )
     for filled, field in draft["fields"].items():
@@ -496,9 +497,27 @@ def _fill(
             by=by,
             edited=True,
         )
-    source, start, end = _lines(
-        draft, sources, _argument(args, "text", command), command
+    return _quote(
+        draft, sources, _argument(args, "text", command), by, command=command, key=key
     )
+
+
+def _quote(
+    draft: dict[str, Any],
+    sources: list[str],
+    where: str,
+    by: str,
+    *,
+    command: str,
+    key: str,
+) -> str:
+    """Writes a field from the lines of a frozen source that `where` names.
+
+    Raises:
+        NoSuchBlock, NoSuchLines: `where` is not somewhere in a source.
+        AlreadyFilled: the field, or the lines it names, are taken.
+    """
+    source, start, end = _lines(draft, sources, where, command)
     return record(
         draft,
         key,
@@ -561,6 +580,22 @@ def _require_question(draft: dict[str, Any], question: str, command: str) -> Non
         )
 
 
+def _text_field(draft: dict[str, Any], key: str, command: str) -> dict[str, Any]:
+    """The field of that name, which a command writing into one has to find.
+
+    Raises:
+        NoSuchField: the draft has no field of that name, or the one it has holds
+            something other than text - `b3.ignore` holds true.
+    """
+    field = draft["fields"].get(key)
+    if field is None or not isinstance(field.get("value"), str):
+        raise NoSuchField(
+            f"There is no field {key} holding text in the draft: {command} writes into "
+            "a field one of the commands before it has written."
+        )
+    return field
+
+
 @command("mark ignore")
 def _mark_ignore(
     draft: dict[str, Any],
@@ -679,12 +714,7 @@ def _field_replace(
     # argument for it, as every other option of a command does.
     regex = "regex" in args and _argument(args, "regex", "field replace", bool)
 
-    field = draft["fields"].get(key)
-    if field is None or not isinstance(field.get("value"), str):
-        raise NoSuchField(
-            f"There is no field {key} holding text in the draft: field replace changes "
-            "the wording of a field one of the commands before it has written."
-        )
+    field = _text_field(draft, key, "field replace")
     value = field["value"]
     try:
         found = len(re.findall(old, value)) if regex else value.count(old)
@@ -714,6 +744,44 @@ def _field_replace(
     return key
 
 
+@command("field set")
+def _field_set(
+    draft: dict[str, Any],
+    sources: list[str],
+    args: dict[str, Any],
+    by: str,
+    directory: str,
+) -> str:
+    """Quotes lines of a frozen source into a field that is written already.
+
+    A spec matching the label line `Q4` alone writes an empty `q4.text`, which
+    `in2lambda validate` reports; the wording the field should hold is somewhere else in
+    the source. So the field is written again from the lines that do say it, at layer 3
+    with their ranges and `edited` false, the way `question add --text` writes one.
+    `field replace` is the command for text nothing in the source says.
+
+    The ranges the field named before are dropped, because its value is no longer
+    copied from those lines. `in2lambda validate` then reports them as in no field, and
+    `in2lambda draft mark ignore` says a block is nothing to take a question from.
+
+    Raises:
+        MalformedCommand: the command has no ``field`` or no ``text``.
+        NoSuchField: the draft has no field of that name holding text.
+        NoSuchBlock, NoSuchLines: its ``text`` is not somewhere in a source.
+        AlreadyFilled: the lines it names are where another field of the same source
+            came from.
+    """
+    key = _argument(args, "field", "field set")
+    where = _argument(args, "text", "field set")
+    _text_field(draft, key, "field set")
+    # Out of the fields before it is written again, so that `record` checks the lines
+    # against the other fields of the source rather than refusing the key it is to
+    # write. Nothing reaches the file until the command has run, since `execute` saves
+    # the draft only once `apply` has returned.
+    del draft["fields"][key]
+    return _quote(draft, sources, where, by, command="field set", key=key)
+
+
 @command("split block")
 def _split_block(
     draft: dict[str, Any],

in2lambda/main.py

@@ -315,6 +315,14 @@ def draft_group() -> None:
 """Who ran a draft command, which every one of them records."""
 
 
+_WHERE = (
+    "Where in a frozen source the text is: a block id such as b3, or lines such as "
+    "s10:14, with the source's number in front - 2/b3, 2/s10:14 - for any but the "
+    "first. Run in2lambda source show to see both."
+)
+"""What --text names, which every command taking one names the same way."""
+
+
 def _text_or_literal(command: Callable[..., None]) -> Callable[..., None]:
     """The two ways to fill a field: quoted from the frozen source, or typed out."""
     for option in (
@@ -323,12 +331,7 @@ def _text_or_literal(command: Callable[..., None]) -> Callable[..., None]:
             help="The text itself, where the source does not say it in a form the "
             "field can take. Marks the field as edited.",
         ),
-        click.option(
-            "--text",
-            help="Where in a frozen source the text is: a block id such as b3, or "
-            "lines such as s10:14, with the source's number in front - 2/b3, 2/s10:14 "
-            "- for any but the first. Run in2lambda source show to see both.",
-        ),
+        click.option("--text", help=_WHERE),
     ):
         command = option(command)
     return command
@@ -445,7 +448,7 @@ def draft_split_block(block: str, at: int, by: str, draft: Optional[str]) -> Non
 
 @draft_group.group("field")
 def draft_field() -> None:
-    """Changes the wording of a field the draft has written already."""
+    """Writes a field the draft has written already, from the source or by hand."""
 
 
 @draft_field.command("replace")
@@ -471,6 +474,16 @@ def draft_field_replace(
     )
 
 
+@draft_field.command("set")
+@click.argument("field")
+@click.option("--text", required=True, help=_WHERE)
+@_by
+@_draft
+def draft_field_set(field: str, text: str, by: str, draft: Optional[str]) -> None:
+    """Writes FIELD again, quoting the lines --text names into it."""
+    _run("field set", {"field": field, "text": text}, by, draft)
+
+
 @draft_group.command("replay")
 @_draft
 def draft_replay(draft: Optional[str]) -> None:

tests/fixtures/drafts/README.md

@@ -24,6 +24,9 @@ is why the block it was typed from is marked ignore rather than left unaccounted
 brace back and another, with `--regex`, writes a `\tfrac` over the division in the solution, so
 both fields end up edited while their ranges still name the lines they were quoted from, and the
 backslash in what the second one writes is written rather than read as a replacement template.
+`field_set` is the repair for the field `empty_field` leaves empty: `question add --text s2` quotes
+a blank line into `q1.text`, which the checks report as empty, and `field set q1.text --text b2`
+writes the paragraph into the field instead, at layer 3 with the block's range and `edited` false.
 `part_without_solution` and `empty_field` are the smallest drafts the other two checks have
 anything to say about; an overlap and a gap in the numbering are not here, because no run of
 commands can make one. `figure_in_a_question` is a question whose text runs on into an image, so

tests/fixtures/drafts/field_set/commands.json

@@ -0,0 +1,24 @@
+[
+  {
+    "args": {
+      "block": "b1"
+    },
+    "by": "tests",
+    "command": "mark ignore"
+  },
+  {
+    "args": {
+      "text": "s2"
+    },
+    "by": "tests",
+    "command": "question add"
+  },
+  {
+    "args": {
+      "field": "q1.text",
+      "text": "b2"
+    },
+    "by": "tests",
+    "command": "field set"
+  }
+]

tests/fixtures/drafts/field_set/expected.json

@@ -0,0 +1,26 @@
+{
+  "b1.ignore": {
+    "by": "tests",
+    "edited": false,
+    "layer": 3,
+    "ranges": [
+      [
+        1,
+        1
+      ]
+    ],
+    "value": true
+  },
+  "q1.text": {
+    "by": "tests",
+    "edited": false,
+    "layer": 3,
+    "ranges": [
+      [
+        3,
+        3
+      ]
+    ],
+    "value": "Find the pressure at the bottom of the tank."
+  }
+}

tests/fixtures/drafts/field_set/report.json

@@ -0,0 +1,14 @@
+[
+  {
+    "check": "no-solution",
+    "field": "q1",
+    "level": "warning",
+    "message": "q1 (lines 3-3) has no solution: q1.solution is not written, and it has no parts.",
+    "ranges": [
+      [
+        3,
+        3
+      ]
+    ]
+  }
+]

tests/fixtures/drafts/field_set/source.md

@@ -0,0 +1,3 @@
+# Pressure in a tank
+
+Find the pressure at the bottom of the tank.

tests/test_draft.py

@@ -404,6 +404,10 @@ def test_the_refusal_names_the_lines_that_are_in_the_way(
         (["draft", "field", "replace", "q1.text", "steam", "water"], "occurs 0 times"),
         (["draft", "field", "replace", "q9.text", "a", "b"], "q9.text"),
         (["draft", "field", "replace", "b1.ignore", "a", "b"], "b1.ignore"),
+        (["draft", "field", "set", "q9.text", "--text", "s3"], "q9.text"),
+        (["draft", "field", "set", "b1.ignore", "--text", "s3"], "b1.ignore"),
+        # Line 16 is where q1's solution came from, so it is not also q2's text.
+        (["draft", "field", "set", "q2.text", "--text", "s16"], "q1.solution"),
         (
             ["draft", "field", "replace", "q1.text", "(", "X", "--regex"],
             "not a regular expression",
@@ -424,6 +428,9 @@ def test_the_refusal_names_the_lines_that_are_in_the_way(
         "wording the field does not say",
         "a field nothing has written",
         "a field that is not text",
+        "a field set naming a field nothing has written",
+        "a field set naming a field that is not text",
+        "a field set quoting lines another field was taken from",
         "a regex that is not one",
     ],
 )
@@ -483,6 +490,22 @@ def test_a_command_says_what_it_wrote(tmp_path: Path, monkeypatch) -> None:
     # --regex is an option, so a command nobody passed it to logs no argument for it.
     assert "regex" not in draft["log"][-1]["args"]
 
+    # And a field set names the field it quoted into, which holds the lines it names and
+    # nothing of the range it named before: q1.text was lines 5-6 and is now line 5.
+    result = runner.invoke(
+        cli, ["draft", "field", "set", "q1.text", "--text", "s5", "--by", "ocr"]
+    )
+
+    assert result.exit_code == 0, result.output
+    assert result.output == "Wrote q1.text.\n"
+    assert json.loads(draft_path.read_text())["fields"]["q1.text"] == {
+        "value": "Water flows through a horizontal pipe of diameter $d$ at speed $v$.",
+        "layer": 3,
+        "ranges": [[5, 5]],
+        "edited": False,
+        "by": "ocr",
+    }
+
 
 def test_a_field_is_quoted_from_a_later_source_by_its_number(
     tmp_path: Path, monkeypatch