Make the server backend-free: the packaging supplies the render program

.github/workflows/ci.yml

@@ -20,8 +20,9 @@ jobs:
 
     - name: Run tests
       run: |
+        SERVER=$(nix build --no-link --print-out-paths .#dynamic)/bin
         TESTS=$(nix build --no-link --print-out-paths .#tests)/bin/lsp-export-server-test
-        "$TESTS" --print
+        PATH="$SERVER:$PATH" "$TESTS" --print
 
   no-gmp:
     runs-on: arc-public-small

CHANGELOG.md

@@ -1,36 +1,14 @@
 # Changelog
 
-## 0.4.1.2
-
-- Release artifacts for aarch64-linux, x86_64-darwin and aarch64-darwin alongside x86_64-linux.
-
-## 0.4.1.1
-
-- A rebuild request before any dependency is known no longer restarts the first render.
-
-## 0.4.1.0
-
-- A `pandoc` backend: pandoc converts the buffer, and the LaTeX-based writers go through the LaTeX backend's quick and settle passes.
-- Tasks and one-shot renders see the output extension as the `format` option.
-
-## 0.4.0.0
-
-- A rebuild request only renders when some on-disk dependency's content changed; the reason names the files.
-- The settle pass is silent in the log unless it fails or changes the output.
-
-## 0.3.0.0
-
-- Statuses carry the pass (`phase`: quick or settle) and the `reason` the render ran.
-- The settle pass no longer reports `rendering`.
-- Marker lines in the log say when each pass started, why, how long it took and what was sent.
-
-## 0.2.0.0
-
-- Renamed from render-server.
-- Renders stamp a fixed per-task SOURCE_DATE_EPOCH, and an output identical to the last one is not published.
-- Reported dependencies leave out the buffers the client sends.
-- Logs the debounce period when a task starts.
-
 ## 0.1.0.0
 
-- First release: the LaTeX backend.
+- First release. A language server that renders a document from the buffers the client sends and
+  streams the result back as xdelta3 diffs: `codedown.render.start/stop/rebuild/setPaused/resync`
+  with `codedown/render/status`, `output` and `log` notifications.
+- `--render EXE`, and optionally `--settle EXE`, name the programs that render. They run in the
+  document's directory with the task's layout and options in `RENDER_*` variables, write
+  `RENDER_OUTPUT`, and may list what they read in `RENDER_DEPS`.
+- An edit gets a quick render once edits pause for the debounce period, and the settle pass follows
+  once they stay quiet for longer. An output identical to the last one is not sent, and renders
+  stamp a fixed per-task `SOURCE_DATE_EPOCH` so unchanged input gives unchanged bytes.
+- A rebuild request renders only when some on-disk dependency's content changed.

README.md

@@ -4,18 +4,53 @@ A language server that renders documents from the buffers a client sends it, and
 result back as xdelta3 diffs. The codedown runner uses it for live exports: an edit reaches the
 server as `didChange` and a new render comes back without a trip through the disk.
 
+The server knows nothing about document formats. The packaging names the program that renders:
+
 ```
-lsp-export-server serve --backend latex -o engine=pdflatex     # the LSP, on stdio
-lsp-export-server render --backend latex input.tex output.pdf  # once, from disk
+lsp-export-server serve --render latex-render --settle latex-render -o engine=pdflatex   # the LSP, on stdio
+lsp-export-server render --render latex-render input.tex output.pdf                      # once, from disk
 ```
 
+Each edit gets a quick render with the `--render` program once edits pause for the debounce
+period. With a `--settle` program, a settle pass follows once edits stay quiet for longer (a full
+latexmk build, say); only the quick pass reports `rendering`, and an output identical to the last
+one is not sent.
+
+## The render program
+
+It runs in the document's directory, so relative paths resolve from the document's real tree,
+with these variables:
+
+| Variable | |
+|---|---|
+| `RENDER_PHASE` | `quick` or `settle` |
+| `RENDER_DOCUMENT` | the real document's absolute path |
+| `RENDER_ENTRY` | the document's current buffer, written to `RENDER_SRC` |
+| `RENDER_OVERLAY` | the other open buffers, at their paths relative to the document's directory; a file here is newer than its disk copy |
+| `RENDER_OUT` | scratch space kept across renders, for aux files and the like |
+| `RENDER_WORK` | the task's work directory holding `src/`, `overlay/` and `out/` |
+| `RENDER_JOB`, `RENDER_FORMAT` | the document's base name and the output extension |
+| `RENDER_OUTPUT` | where to write the result: `RENDER_OUT/RENDER_JOB.RENDER_FORMAT` |
+| `RENDER_DEPS` | optional: write the files read, one path per line, absolute or relative to the directory |
+| `RENDER_EPOCH` | a Unix time to use for timestamps in the output (`SOURCE_DATE_EPOCH`), fixed per task so unchanged input gives unchanged bytes |
+| `RENDER_OPT_<key>` | each `-o key=value` from the command line and each option in the start request |
+
+Its stdout and stderr are streamed to the client as the log. A non-zero exit reports an error with
+that exit code. Paths listed in `RENDER_DEPS` under `RENDER_SRC` or `RENDER_OVERLAY` are mapped
+back to the documents they stand for; the client is told to watch the ones it does not hold open,
+and a `rebuild` renders again only when one of them changed.
+
+## Protocol
+
 Commands (`workspace/executeCommand`): `codedown.render.start`, `codedown.render.stop`,
 `codedown.render.rebuild`, `codedown.render.setPaused`, `codedown.render.resync`. Notifications:
 `codedown/render/status`, `codedown/render/output`, `codedown/render/log`. See `src/Render/Protocol.hs`.
-
-Each edit gets a quick single-pass render, then a latexmk settle pass once edits stay quiet; only
-the quick pass reports `rendering`, and an output identical to the last one is not sent. Statuses
-carry the pass (`phase`) and what triggered it (`reason`), and `──` marker lines in the log say
-when each pass started, why, how long it took and what was sent.
+Statuses carry the pass (`phase`) and what triggered it (`reason`), and `──` marker lines in the
+log say when each pass started, why, how long it took and what was sent.
 
 `cbits/` is xdelta3, Apache-2.0 (see `cbits/LICENSE`).
+
+## Developing
+
+`nix develop -c stack build --test`. The tests drive the built executable through lsp-test with
+shell scripts as render programs.

app/Main.hs

@@ -3,13 +3,11 @@ module Main (main) where
 import qualified Data.ByteString as B
 import qualified Data.Map as M
 import Data.Text (Text)
-import Data.Time.Clock.POSIX (getPOSIXTime)
 import qualified Data.Text as T
 import qualified Data.Text.IO as T
+import Data.Time.Clock.POSIX (getPOSIXTime)
 import Options.Applicative
-import Render.Backend
-import Render.Backend.Latex (latexBackend)
-import Render.Backend.Pandoc (pandocBackend)
+import Render.Command
 import Render.Protocol (LogStream(..))
 import Render.Server (runRenderServer)
 import System.Directory
@@ -20,29 +18,28 @@ import System.IO.Temp
 import UnliftIO (try)
 
 
-backends :: [Backend]
-backends = [latexBackend, pandocBackend]
-
 data Common = Common {
-  commonBackend :: String
+  commonPrograms :: Programs
   , commonOptions :: [(Text, Text)]
   }
 
-data Command =
+data Invocation =
   Serve Common
   | RenderOnce Common FilePath FilePath
 
 commonParser :: Parser Common
 commonParser = Common
-  <$> strOption (long "backend" <> metavar "NAME" <> help "Which backend renders: latex, pandoc")
-  <*> many (option (eitherReader keyValue) (long "option" <> short 'o' <> metavar "KEY=VALUE" <> help "A backend option, e.g. engine=xelatex"))
+  <$> (Programs
+    <$> strOption (long "render" <> metavar "EXE" <> help "The program that renders: run in the document's directory with the RENDER_* variables; writes RENDER_OUTPUT")
+    <*> optional (strOption (long "settle" <> metavar "EXE" <> help "The program for the settle pass once edits stay quiet; without one there is no settle pass")))
+  <*> many (option (eitherReader keyValue) (long "option" <> short 'o' <> metavar "KEY=VALUE" <> help "An option passed on as RENDER_OPT_KEY"))
   where
     keyValue s = case break (== '=') s of
       (k, '=' : v) | not (null k) -> Right (T.pack k, T.pack v)
       _ -> Left "expected KEY=VALUE"
 
-commandParser :: Parser Command
-commandParser = hsubparser (
+invocationParser :: Parser Invocation
+invocationParser = hsubparser (
   command "serve" (info (Serve <$> commonParser) (progDesc "Run the language server on stdio"))
   <> command "render" (info (RenderOnce <$> commonParser
                                         <*> strArgument (metavar "INPUT")
@@ -53,43 +50,37 @@ commandParser = hsubparser (
 main :: IO ()
 main = do
   hSetBuffering stderr LineBuffering
-  cmd <- execParser (info (commandParser <**> helper) (fullDesc <> progDesc "Renders documents, live from a language client or once from disk"))
-  case cmd of
-    Serve common -> do
-      backend <- pickBackend common
+  execParser (info (invocationParser <**> helper) (fullDesc <> progDesc "Renders documents, live from a language client or once from disk")) >>= \case
+    Serve (Common {..}) -> do
       tmp <- getCanonicalTemporaryDirectory
       withTempDirectory tmp "lsp-export-server" $ \workRoot ->
-        runRenderServer backend workRoot (M.fromList (commonOptions common)) >>= \case
+        runRenderServer commonPrograms workRoot (M.fromList commonOptions) >>= \case
           0 -> return ()
           n -> exitWith (ExitFailure n)
-    RenderOnce common input output -> do
-      backend <- pickBackend common
+    RenderOnce (Common {..}) input output -> do
       inputPath <- makeAbsolute input
       text <- T.readFile inputPath
       tmp <- getCanonicalTemporaryDirectory
       epoch <- round <$> getPOSIXTime
       withTempDirectory tmp "lsp-export-server" $ \workDir -> do
-        result <- try $ backendRender backend RenderRequest {
+        result <- try $ render commonPrograms RenderRequest {
           requestEntryPath = inputPath
           , requestEntryText = text
           , requestOverlays = []
           , requestWorkDir = workDir
-          , requestOptions = M.insert "format" (T.pack (drop 1 (takeExtension output))) (M.fromList (commonOptions common))
+          , requestOptions = M.insert "format" (T.pack (drop 1 (takeExtension output))) (M.fromList commonOptions)
           , requestSettle = True
           , requestEpoch = epoch
-          , requestLog = \stream line -> T.hPutStrLn (if stream == LogStdout then stdout else stderr) line
+          , requestLog = \stream line -> T.hPutStrLn (handleFor stream) line
           }
         case result of
           Left (RenderFailure code msg) -> do
             T.hPutStrLn stderr msg
-            exitWith (ExitFailure (if code == 0 then 1 else code))
+            exitWith (ExitFailure (case code of { 0 -> 1; n -> n }))
           Right (RenderResult {resultBytes}) -> do
             createDirectoryIfMissing True (takeDirectory output)
             B.writeFile output resultBytes
-
-pickBackend :: Common -> IO Backend
-pickBackend (Common {commonBackend}) = case [b | b <- backends, backendName b == T.pack commonBackend] of
-  (b : _) -> return b
-  [] -> do
-    hPutStrLn stderr ("unknown backend: " <> commonBackend <> "; known: " <> unwords (map (T.unpack . backendName) backends))
-    exitWith (ExitFailure 2)
+  where
+    handleFor = \case
+      LogStdout -> stdout
+      LogStderr -> stderr

lsp-export-server.cabal

@@ -5,7 +5,7 @@ cabal-version: 2.2
 -- see: https://github.com/sol/hpack
 
 name:           lsp-export-server
-version:        0.4.1.2
+version:        0.1.0.0
 synopsis:       A language server that renders documents from their in-memory buffers
 homepage:       https://github.com/codedownio/lsp-export-server#readme
 bug-reports:    https://github.com/codedownio/lsp-export-server/issues
@@ -37,9 +37,7 @@ source-repository head
 
 library
   exposed-modules:
-      Render.Backend
-      Render.Backend.Latex
-      Render.Backend.Pandoc
+      Render.Command
       Render.Process
       Render.Protocol
       Render.Server
@@ -60,6 +58,7 @@ library
       FlexibleInstances
       LambdaCase
       ViewPatterns
+      RankNTypes
       MultiWayIf
       TupleSections
       NumericUnderscores
@@ -92,6 +91,53 @@ library
     , unliftio
   default-language: Haskell2010
 
+library test-lib
+  exposed-modules:
+      TestLib.Events
+      TestLib.Main
+      TestLib.Session
+  other-modules:
+      Paths_lsp_export_server
+  autogen-modules:
+      Paths_lsp_export_server
+  hs-source-dirs:
+      test-lib
+  default-extensions:
+      OverloadedStrings
+      QuasiQuotes
+      NamedFieldPuns
+      RecordWildCards
+      ScopedTypeVariables
+      FlexibleContexts
+      FlexibleInstances
+      LambdaCase
+      ViewPatterns
+      RankNTypes
+      MultiWayIf
+      TupleSections
+      NumericUnderscores
+      TypeApplications
+      DataKinds
+  ghc-options: -Wunused-packages -Wall -fno-warn-name-shadowing
+  build-depends:
+      aeson
+    , base >=4.7 && <5
+    , base64-bytestring
+    , bytestring
+    , containers
+    , directory
+    , filepath
+    , lens
+    , lsp-export-server
+    , lsp-test
+    , lsp-types
+    , process
+    , sandwich
+    , string-interpolate
+    , text
+    , time
+  default-language: Haskell2010
+
 executable lsp-export-server
   main-is: Main.hs
   other-modules:
@@ -110,15 +156,15 @@ executable lsp-export-server
       FlexibleInstances
       LambdaCase
       ViewPatterns
+      RankNTypes
       MultiWayIf
       TupleSections
       NumericUnderscores
       TypeApplications
       DataKinds
   ghc-options: -Wunused-packages -Wall -fno-warn-name-shadowing -threaded -rtsopts -with-rtsopts=-N -O2 -W
   build-depends:
-      aeson
-    , base >=4.7 && <5
+      base >=4.7 && <5
     , bytestring
     , containers
     , directory
@@ -135,6 +181,11 @@ test-suite lsp-export-server-test
   type: exitcode-stdio-1.0
   main-is: Main.hs
   other-modules:
+      Spec
+      Spec.Deps
+      Spec.Protocol
+      Spec.Server
+      Spec.Xdelta
       Paths_lsp_export_server
   autogen-modules:
       Paths_lsp_export_server
@@ -150,20 +201,27 @@ test-suite lsp-export-server-test
       FlexibleInstances
       LambdaCase
       ViewPatterns
+      RankNTypes
       MultiWayIf
       TupleSections
       NumericUnderscores
       TypeApplications
       DataKinds
   ghc-options: -Wunused-packages -Wall -fno-warn-name-shadowing -threaded -rtsopts -with-rtsopts=-N -W
+  build-tool-depends:
+      sandwich:sandwich-discover
+    , lsp-export-server:lsp-export-server
   build-depends:
       aeson
     , base >=4.7 && <5
     , bytestring
     , containers
+    , filepath
     , lsp-export-server
+    , lsp-test
     , sandwich
     , string-interpolate
+    , test-lib
     , text
     , unliftio
   default-language: Haskell2010