Performance enhancements

[ahankinson]

It's great to see that this project has some life in it now. I have been using it for a while and maintaining my own fork with some performance enhancements.

Primarily, the main enhancements have been to enable packratting (which seems to now be enabled) and pre-declaring and compiling the regexes in the natural language processor.

If the current benchmarks are to be believed, this leads to a significant speedup. All the tests pass, so I guess it's OK?

https://rism-digital.github.io/python-edtf/dev/bench/

I've gone through and tried to integrate the latest changes, but that was a big PR so another set of eyes would be helpful.

[ahankinson]

Not sure why it's failing -- the tests all pass and work on my end.

[ahankinson]

It occurs to me that a big part of the reason for the speed-up is the use of the functools lru_cache, which will cache the result of the same input string and skip the computation of that value, instead returning only the cached value. While this is useful in real-world contexts, it's probably not so useful in benchmark runs!

[ahankinson]

@aweakley does this PR look OK?

[aweakley]

This looks great! Thank you very much. I've just put a couple of comments - I think some things were duplicated in a merge at some point maybe? and one regex looks different.

[ahankinson]

I've done a bit of cleanup, but the parser_classes file is still quite buggy. Unfortunately I don't quite know how to trigger the bugs, but I'm at least confident that I didn't make it much worse.

[aweakley]

Thank you. I'll check it out.

[aweakley]

@ahankinson The tests are passing for me, are there particular bugs you were worried about in your comment above?

[ahankinson]

Yeah the tests pass but I don’t think they exercise the parser classes that much. I have the ruff checker running on that and it still highlights a bunch of stuff

[ahankinson]

BTW, ruff has also updated some rules so I have updated it accordingly now.

[ahankinson]

This seems to be going nowhere. I'm going to be adding more changes on my fork, so to prevent it from running here I'll close it.

[aweakley]

Gah! sorry @ahankinson , I missed your comments. I'd like to merge but if I reopen this I think I'll pick up your most recent changes on your branch - are you happy for me to merge them too?

[ahankinson]

Do you have any preference on whether I keep the lru_cache in the library? The performance improvements are significant, but I can understand not wanting a caching mechanism in a library.

The other changes I've been adding are fairly benign, so I'm happy for them to be merged.

[aweakley]

I think let's keep it in please.

[ahankinson]

I probably won’t be able to check it over until next week. I’m the meantime if you see something you want removed let me know.

[ahankinson]

One thing I did in my branch is remove support in the CI for python 3.9, since a) it's EOL later this year, and b) from 3.10 onwards the type annotation Optional[foo] can be written foo | None, which I prefer.

[ahankinson]

This should be ready for review now. I'm not sure why it's failing though.

[aweakley]

That's great, thank you very much indeed.

[aweakley]

I think the change to the signature here:

python-edtf/edtf/parser/grammar.py

Lines 346 to 351 in 95ebb72

	def parse_edtf(
	input_string: str,
	parse_all: bool = True,
	fail_silently: bool = False,
	debug: bool | None = None,
	):

(parseAll->parse_all) means that we should release this as a new major version, so I'll prepare that.