[homersimpsons]

• Implement command
• Structured CanonicalData
• Use twig to render exercises
  • Custom functions to ease writing template:
    • export: export any variable
    • testfn: write a test function name
    • sort_input: sort tests inputs
• Test mode

Part of #1066

[github-actions]

This PR touches files which potentially affect the outcome of the tests of an exercise. This will cause all students' solutions to affected exercises to be re-tested.

If this PR does not affect the result of the test (or, for example, adds an edge case that is not worth rerunning all tests for), please add the following to the merge-commit message which will stops student's tests from re-running. Please copy-paste to avoid typos.

[no important files changed]

For more information, refer to the documentation. If you are unsure whether to add the message or not, please ping @exercism/maintainers-admin in a comment. Thank you!

[mk-mxp]

Thanks for showing this PoC. I know Twig, so I have no doubt Twig is a way to make the test generator. In my eyes, putting all the logic into Twig simply moves it away from the language all our contributors should know best: PHP. So, no doubt, Application.php will be less complex. But the Twig templates must contain the complexity then. I agree, sprintf format strings don't look as nice as Twig does. But they have no dependencies and no special knowledge is required.

As I tried to explain in the README of my suggestion, it is not about the templating. The thing to ensure is, that all of the possible variations of canonical-data.json including the unknown / upcoming variations must be visible in the output of the test generator. That is the most relevant part of it.

The next relevant thing is: To ensure all output is thought about by the user of the generator. There is no use in a test generator that does what ChatGPT does: Simply put together code that somehow is running. The problem specifications are made for humans, and humans must interpret comments and values to form the test code accordingly. That is the core of Erik's warning about taking it to far: Do not think generating code means being done.

You are right, Twig is tested well. But does the Twig template actually do what is intended to do? That is, what I think tests are for. They ensure, that all the loops and variations behave as intended. So, no, testing cannot be skipped. One may use golden tests, which mean making up all the variations and comparing the results. But please, not only manually while developing, but runnable and repeatable.

That said, I would not mind combining the parsing of the canonical-data.json with Twig templates. It is required to separate out the known from the unknown parts and render all of those parts into some kind of template. There must be a way to force users to look into the code and not pretend they are done when there was no error thrown by the generator. How does that look within your suggestion?

Exercises to look at for the variations: list-ops is the most easy kind, bank-account has errors and multiple calls to different functions per test, complex-numbers has nested cases and relevant comments to show to the user (of the generator).

[mk-mxp]

@homersimpsons There is a discussion going on about the test generation in the forum: https://forum.exercism.org/t/test-generators-for-tracks/10615

I think, this is what is different between my attempt and yours. You want to have an updating generator to produce production-ready test code. I want to have a first-time generator to generate code from scratch. These are 2 different goals and thus need 2 different tools.

[IsaacG]

Per #663 @mk-mxp said "Please feel free to take this forward."

@homersimpsons Would you like to move forward with this? I think it would add value to the track.

[homersimpsons]

@homersimpsons Would you like to move forward with this? I think it would add value to the track.

Yes definitely, I will revisit this in 2 weeks (a bit busy on other topics).

I will consider using PHP for templating, but my early researches were not really conclusive on this topic.

[IsaacG]

If Twig makes things easier, I think that's fine. If we decide to prefer PHP, that can always be changed later.

[IsaacG]

Any updates here, @homersimpsons ?

[IsaacG]

@homersimpsons Ping?

[homersimpsons]

@homersimpsons Ping?

Yes sorry, I started to look into this, but the other merge request with logic to kick start a new exercise was merged and so I wanted to integrate both codes but it embeds a full Symfony application instead of just the console component etc... so I started to do a bit of yak shaving but did not had a lot of time available for this in the previous weeks to complete this work.

[IsaacG]

No worries. Life happens. I just wanted to make sure this wasn't forgotten or abandoned!

[mk-mxp]

Thanks a lot for making this real!

[IsaacG]

Nice! Congrats on getting this done!