SIP-72: dedented triple-quoted string literals

[lihaoyi]

Implements scala/improvement-proposals#112

• The initial Lexing/Scanning is lenient, only looking for the opening ''''* and equivalent closing delimiter. This matches how we can expect this to be implemented in other tools that have more restricted lexing frameworks (IntelliJ w/ JFlex, VSCode w/ TextMate Grammars, NeoVim w/ TreeSitter)

• All other validation (opening delimiter must be followed by newline, closing delimiter must be preceded by whitespace only) and de-denting is left to the parsing phase, which is the only time we have a complete string "literal" when an interpolator is present, and thus are able to look at the trailing delimiter's preceding indent whitespace and trim it from all earlier STRINGLIT/STRINGPART tokens

• def interpolatedString needed to be refactored to support dedenting: rather than constructing the trees immediately, we first assemble all the strings parts, then use the last string part to compute the dedent that we apply to all other parts, and only then do we construct the trees

• Covered by neg/ tests and run/ tests for all the major features and edge cases I could think of:

  • All indentation removed
  • Some indentation preserved
  • Empty strings
  • Single-line strings
  • Blank lines in the string
  • Leading and trailing blank lines
  • Varying indentation
  • Extensible delimiters with 4 and 5 quotes
  • Funky operator and unicode characters in the string
  • Tab-based indentation
  • Interpolation with s and f
  • Single- and Multi-line pattern matching with and without interpolation
  • In larger expressions: lists, infix operators, etc.
  • As singleton-type ascriptions and singleton-type parameters
  • As literals passed to @compileTimeOnly

• I haven't managed to reliably run tests for some reason, I think I'm bumping into https://contributors.scala-lang.org/t/current-testcompilation/7256. But I tested it manually by copy-pasting the run/neg test files into the bin/scala REPL and compared the output manually with the .check files on disk, and the output is identical

[odersky]

That was fast!

[lihaoyi]

Not ready to review yet! Still need a bit more vibing haha

[Gedochao]

Not ready to review yet! Still need a bit more vibing haha

Ah right, I'll convert it to draft then

[He-Pin]

Should be able to detect this in one loop

[lihaoyi]

@noti0na1 resolved your comments

[lihaoyi]

also wired up the REPL incomplete literal detection to work for ''' strings

[lihaoyi]

Bump on this since been a while

[Gedochao]

Bump on this since been a while

No worries, we remember. Work on getting 3.8 to RC1 may be delaying things, sorry about the compiler team being perhaps less a tad responsive these couple days.

[lihaoyi]

Just another bump here since it's been two weeks since the last bump, I would like to try and get this into 3.8.0 if possible

[Gedochao]

I would like to try and get this into 3.8.0 if possible

I mean, the SIP is still in progress, 3.8.0-RC1 is already late (the artifacts should become available later today, if all goes well) and this is a rather hefty review (which we haven't jumped on immediately, as finalising 3.8.0-RC1 has devoured most of the team's resources).
So it's almost certainly something that'll wait until 3.10.
That being said, we'll try to get back to the review as soon as possible.

[lihaoyi]

The SIP is in progress because it's waiting for an implementation, which is this PR. So in other words the SIP is currently blocked on this review to make progress

Fine if it's impossible to get it in to 3.8.0, just want to make sure my expectations are set appropriately. If you say "a tad less responsive", don't be surprised if I keep pinging you every so often. If you say "come back in a few months", then I'll come back in a few months and stop bothering you haha

[Gedochao]

The SIP is in progress because it's waiting for an implementation, which is this PR. So in other words the SIP is currently blocked on this review to make progress

Fine if it's impossible to get it in to 3.8.0, just want to make sure my expectations are set appropriately. If you say "a tad less responsive", don't be surprised if I keep pinging you every so often. If you say "come back in a few months", then I'll come back in a few months and stop bothering you haha

By all means, do ping us if we don't come back 😅

Additional note: 3.10 sounds very distant, but in reality, since 3.9.0 is planned as the new LTS, 3.10.0 will likely follow shortly after, with 3.9.x patches being reserved for backports.
So still distant, just not extremely so.

[lihaoyi]

If this goes in experimentslly in 3.10 it won't be usable by the community until non-experimentally in 3.11, which would be in 2027 i assume? That's maybe a year later than going in in 3.8 and becoming usable in 3.9, which is my motivation for trying to get it in. Delaying the SIP a year is a long time for something that's already approved and marked for implementation, and in fact has an implementation ready to go. So you can understand why I'd like to get it in sooner if at all possible

[He-Pin]

If this can get merged in 3.8.0 and marked as experimental , then it will always free to change.

[Gedochao]

If this goes in experimentslly in 3.10 it won't be usable by the community until non-experimentally in 3.11, which would be in 2027 i assume?

Yes.

That's maybe a year later than going in in 3.8 and becoming usable in 3.9, which is my motivation for trying to get it in.

The current intention is to not promote any experimental features to stable or preview between 3.8 and 3.9 (as part of the new LTS feature freeze). Only preview features could get stabilised.

So you can understand why I'd like to get it in sooner if at all possible

I am aware many features will get delayed as a result of the feature freeze implemented for the new 3.9 LTS' sake.
We knew this.
That is the cost of making an LTS version to users in a stable state.

Scala 3.8.0-RC1 is long overdue, its release is being prepared as we speak and we will not be delaying it any further. There is always that one more thing that we could potentially get if we wait another couple of days. One has to draw the line at some point. That point is now. Has been for a bit, in fact, as the 3.8 branch has been cut some time ago, with select changes being back ported there.

Even though we will likely have a 3.8.0-RC2 (if not 3.8.0-RC3, given just how big and complex 3.8 came out to be), subsequent RCs are reserved strictly for fixes and improvements, not new features.

And so, it currently seems unlikely we'd include this PR before 3.10.
Thank you for your understanding.

[WojciechMazur]

We're in the future freeze for the last 2 weeks. It needs to wait for 3.10, be accepted by SIP, made preview in 3.11 and maybe stable in 3.12.

[odersky]

It is accepted for implementation so we are OK as far the SIP is concerned. But the fact that we have been in feature freeze for two weeks is true. So the timing does not work out, I am afraid.

[lihaoyi]

Got it. If it can't go into 3.8, could it go into 3.9 as experimental? @Gedochao said the restriction for 3.9 is no new stable features, but are experimental features also restricted as @WojciechMazur seems to suggest?

[odersky]

Yes, we have a policy of no new experimental features in 3.9. But I don't expect the 3.9 period to last long anyways. Essentially we just want to get to a stable state that we can support long term with backports. Once we are there the head will move to 3.10.

[odersky]

@lihaoyi I think it's easiest if I work on the PR a bit to make it more conformant to our internal conventions. I just invited you to https://github.com/dotty-staging/dotty/. Can you push your PR there? Then several people can collaborate on it.

[lihaoyi]

Got it! Will wait for 3.10 then. I pushed this PR to dotty-staging@dedented-strings as you mentioned

[lihaoyi]

This PR also has Allow edits and access to secrets by maintainers enabled, so feel free to push changes directly here if you would like