feat: add pretty run report

[kaeun97]

Resolves #398.

Here is an example code:

from __future__ import annotations
from egglog import *

egraph = EGraph()

class Num(Expr):
    def __init__(self, n: i64Like) -> None: ...
    def __add__(self, other: Num) -> Num: ...
    def __mul__(self, other: Num) -> Num: ...

x, y = vars_("x y", Num)
egraph.register(rewrite(x + y).to(y + x))
egraph.register(Num(1) + Num(2))
report = egraph.run(10)
print(report)

Output before:

RunReport { iterations: [IterationReport { rule_set_report: RuleSetReport { changed: true, rule_reports: {"(rewrite (__main___Num___add__ _x _y) (__main___Num___add__ _y _x))": [RuleReport { plan: None, search_and_apply_time: 2.625µs, num_matches: 1 }]}, search_and_apply_time: 5.375µs, merge_time: 583ns }, rebuild_time: 1.125µs }, IterationReport { rule_set_report: RuleSetReport { changed: false, rule_reports: {"(rewrite (__main___Num___add__ _x _y) (__main___Num___add__ _y _x))": [RuleReport { plan: None, search_and_apply_time: 1.125µs, num_matches: 1 }]}, search_and_apply_time: 2.75µs, merge_time: 1.041µs }, rebuild_time: 0ns }], updated: true, search_and_apply_time_per_rule: {"(rewrite (__main___Num___add__ _x _y) (__main___Num___add__ _y _x))": 3.75µs}, num_matches_per_rule: {"(rewrite (__main___Num___add__ _x _y) (__main___Num___add__ _y _x))": 2}, search_and_apply_time_per_ruleset: {"": 8.125µs}, merge_time_per_ruleset: {"": 1.624µs}, rebuild_time_per_ruleset: {"": 1.125µs} }

Output after:

PrettyRunReport(iterations=[PrettyIterationReport(rule_set_report=PrettyRuleSetReport(changed=True, rule_reports={'rewrite(x + y).to(y + x)': [PrettyRuleReport(plan=None, search_and_apply_time=datetime.timedelta(0), num_matches=1)]}, search_and_apply_time=datetime.timedelta(0), merge_time=datetime.timedelta(0)), rebuild_time=datetime.timedelta(0)), PrettyIterationReport(rule_set_report=PrettyRuleSetReport(changed=False, rule_reports={'rewrite(x + y).to(y + x)': [PrettyRuleReport(plan=None, search_and_apply_time=datetime.timedelta(0), num_matches=1)]}, search_and_apply_time=datetime.timedelta(0), merge_time=datetime.timedelta(0)), rebuild_time=datetime.timedelta(0))], updated=True, search_and_apply_time_per_rule={'rewrite(x + y).to(y + x)': datetime.timedelta(0)}, num_matches_per_rule={'rewrite(x + y).to(y + x)': 2}, search_and_apply_time_per_ruleset={'': datetime.timedelta(0)}, merge_time_per_ruleset={'': datetime.timedelta(0)}, rebuild_time_per_ruleset={'': datetime.timedelta(0)})

[saulshanabrook]

Thank you for this! Added a few comments. Could you also add this to the changelog file with a link to this PR?

[codspeed-hq]

Merging this PR will degrade performance by 67.23%

❌ 4 regressed benchmarks
✅ 8 untouched benchmarks
⏩ 2 skipped benchmarks¹

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	test_jit[add]	636.6 ms	946.6 ms	-32.75%
❌	Simulation	test_jit[lda]	11.6 s	35.3 s	-67.23%
❌	WallTime	test_jit[lda]	15.8 s	33.2 s	-52.39%
❌	WallTime	test_jit[add]	813.9 ms	1,051.9 ms	-22.62%

---

_{Comparing kaeun97:kaeun97/pretty-report (3d7bec7) with main (8812ec9)}

Footnotes

1. 2 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[saulshanabrook]

Thanks for the fixes, I left a few small comments. There are also some mypy and formatting issues I think.

There is a bigger question about performance, if the codspeed is correct it looks like this slows things down by a ton!

Taking almost 40% of the time in a bigger benchmark just to translate bindings.

It makes me wonder about a different approach, where we set each rewrite and rule with a manual name like 1, 2, 3, ... and then we don't have to do the name searching and mangling and can just parse the name as an int then look it up? And if it's a birewrite just take off the <= or >=?

It would make the egglog file a bit more verbose, but makes parsing the reports more straightforward and more performant which seems like a good tradeoff?

I was also going back and forth on whether the RunReport should store a RewriteOrRule or the decl? If we just store the RewriteOrRule it's easier to pretty print, can just use the builtin one, and it's easier for users to grab that off and compare it or use it... But most of the other exposed objects just store the decls, so I will leave it up to you!

EDIT: It looks like the docs failures also highlight some other exceptions from this. I imagine also if we name the rules here that might also help since it seems like it's hitting on looking up the string?

[saulshanabrook]

If a name is provided, we don't need to save the normalized version right?

[saulshanabrook]

nit: remove this extra function

[saulshanabrook]

Could you leave a comment here that the birewrite is de-sugared into these two names in egglog which is why we have both of them?

[saulshanabrook]

A birewrite will actually show up twice right? So if we want to keep both of them as one Python rule, then I think then we would need to add the timings for both of them?

[saulshanabrook]

Could you add to the tests asserting the string of the rule is in rewrite output, not just that it doesn't container some strings?

[saulshanabrook]

We don't have to do this, but we could also change these strings into actual ruleset objects? I am not sure if that would be too hard or awkward, and isn't essential for this PR.

[kaeun97]

@saulshanabrook Thanks for the thorough review! I do agree that the performance looks concerning. The numeric name approach you mentioned would work for bindings with a "name" field - so not for, RewriteDecl nor BiRewriteDecl. That would require the rust side change. Happy to prioritize that before continuing on with this PR. Also, we can do lazy loading (translate when the user needs it) to have minimal impact to performance.

[saulshanabrook]

The numeric name approach you mentioned would work for bindings with a "name" field - so not for, RewriteDecl nor BiRewriteDecl. That would require the rust side change. Happy to prioritize that before continuing on with this PR.

Ah yeah I kept forgetting about this! I just talked to some other folks on the egglog team and they said that sounds like a great feature to add, just something we hadn't gotten around to yet. It should also I think be relatively straightforward so a good first PR to egglog core if you don't mind doing that...

Then once that is merged hopefully should just be able to update the pin here and can use that feature. I believe the version of egglog we depend on here is pretty recent, so hopefully won't be other changes we have to adapt to.