Ticket #5018: Color pairs beyond 256

[egmontkob]

Proposed changes

editor: Allow more contexts per syntax highlighting rule
tty: Allow more than 128 / 256 color pairs

• Resolves: mcedit: some foreground / background color combinations look incorrect #5018

Checklist

• I have referenced the issue(s) resolved by this PR (if any)
• I have signed-off my contribution with git commit --amend -s
• Lint and unit tests pass locally with my changes (make indent && make check)
• I have added tests that prove my fix is effective or that my feature works
• I have added the necessary documentation (if appropriate)

[egmontkob]

I've tested with slang, ncurses 6.5 with widechar, ncurses 5.9 without widechar; combined with TERM=xterm, TERM=linux and more modern ones (256+ colors). Not all combinations of these, but a lot of them.

In 8-color terminals the situation still isn't perfect (isn't expected to be either), but there's no regression.

In 256+-color terminals the issues are now fixed, the test file of the original bugreport appeared as expected.

IMO let's land this first, I'll rebase #4995 on top of this.

[zyv]

Great work, thank you! 🚀

I don't think there was a deep thinking behind limiting the number of contexts. My guess was that one had to pick something in the first place, and 256 was picked out of memory considerations along the lines of "640K should be enough for everybody".

I can't claim I understand (or took the time to understand) the system in-depth, but the idea was to allow for an extremely fast recovering streaming parser, which would be flexible enough to cover popular formats in a halfway sane way.

For this, you have "contexts" and within those "contexts" you have "keywords". The latter would roughly correspond, e.g. to standard library function names in a programming language. The former is more like areas within these function names that should be processed in different ways (like comment blocks). That's about it (and, by the way, the way the context idea is implemented is the genetic reason for #2142).

We definitively have syntax files with more "keywords" per "context" than 256, but I wasn’t sure that we have any with more than 256 contexts. A quick grep shows that we only have files with up to 60 contexts or so:

for i in *.syntax ; do grep -E "^context" $i | wc -l ; done

I don't think that relaxing this limitation would cause any serious issues though, other than giving users yet another opportunity to shoot themselves in the foot, of which they already have plenty anyways...