refactor: allow multiline and comments in python expressions

[JuroOravec]

Follow up to #17

This adds support for multiline Python expressions and comments within these expressions:

{% component "table"
  data=([
    1,  # This is bla
    2,  # Also bla
    3,  # bla bla
  ])
/ %}

[JuroOravec]

Unfortunately Ruff's Python AST doesn't capture info on comments. So I had to parse the comments manually by searching for # and taking comments as everything from # until the end of line.

But then we naturally want to avoid those # inside strings. But Python strings aren't simply anything between two quotes, because strings can contain escaped quotes like " this is \" escaped".

So this function parses the string, detects strings, while being mindful of escaped quotes inside strings. And only when outside of strings, it captures # as comments.

[JuroOravec]

The logic for extracting comments goes character by character and changes the state based on whether we're inside a string, entering/leaving a string, etc.

[JuroOravec]

Because the Python expression is parsed by the Ruff Python AST parser, it raises a different error message than what Python would have raised if the expression was evaluated with eval().

So if we have a expression (1, 2], in Python we get an error:

>>> (1, 2]
  File "<stdin>", line 1
    (1, 2]
         ^
SyntaxError: closing parenthesis ']' does not match opening parenthesis '('

But the AST parser will raise an error message like so:

Parse error: Expected an expression or a ']' at byte range 7..8

Now, to make these error messages easier to interpret, this function modifies the error messages to also show the exact syntax that caused the error:

Parse error: Expected an expression or a ']' at byte range 7..8: ')'

[JuroOravec]

Another thing that this function does is that when the Python expression is passed to Ruff's Python AST parser, the expression is wrapped in extra parentheses and newlines - (\n...\n).

This is necessary to allow the Python expressions to be defined across multiple lines (e.g. imagine a comment on a separate line before or after the actual expression:

{% component "table"
  data=(
    # bla bla
    [1, 2, 3]
  )
/ %}

But the side effect is that all line and col positions are shifted.

And when there's a syntax error, the Ruff Python AST parser reports also the start/end indices of the problematic syntax.

So this function also fixes the indices so that they refer to the position in the original expression, and NOT in the one wrapped in (\n...\n).

[JuroOravec]

Another thing to enable the Python expressions to span multiple lines was to switch from eval() to exec(), because eval() doesn't allow newlines.

One good thing about eval() was that it allows only a single Python expression. "expression" is something that can be assinged to a variable. On the other hand, "stataments" are syntax that cannot be assigned to variables (e.g. import). So eval() disallowed code with statements or multiple expressions. So something like these would NOT be allowed:

# Statement
eval("""
from os import abc
class X:
    ...
""")

# Multiple expressions
eval("""
2 + 2
my_var
""")

However, the logic is still safe, because the same safety measure is implemented on the level of Ruff's Python AST parser:

• There, when walking the AST, we raise error if we come across statements, effectively banning them.
• The Ruff AST parser also has a setting to parse the code as "expression", which raises when multiple expressions or statements are encountered. See parse_expression

[JuroOravec]

cc @EmilStenstrom