Add uniqueness restriction on names #567
Conversation
| It is good practice to use a numbered prefix such as `00`, `01`, `02`, `03`, and so on, to get the desired order of test cases, while keeping the file names descriptive. | ||
| Remember that the numbered prefixes should be zero padded to the same length to get the expected lexicographical order. | ||
|
|
||
| Test case file names (the base name of the `.in` file) must be unique across the entire `data/` directory, unless the test cases are equivalent. |
There was a problem hiding this comment.
Maybe we should be a tiny bit more specific with the definition of equivalence? Surely we don't care about output validator flags in this definition, right?
For any two test cases, if the contents of their .in and .files directory are equivalent, as well as the args sequence in the .yaml file, then the input of the two test cases is equivalent. For any two test cases, if their input, output validator arguments and the contents of their .ans files are equivalent, then the test cases are equivalent.
At the very least, we should say "if their inputs are equivalent". Additionally, we should probably either copy paste the definition or link to it.
There was a problem hiding this comment.
Surely we don't care about output validator flags in this definition, right?
Agreed, I want to be able to reuse a .in file with different output validator flags.
There was a problem hiding this comment.
Ok, maybe we do, but then we have two different kinds of equivalence. The one you want to use is "the inputs are equivalent", the one we already have defined and that I used is "the test cases are equivalent". The latter allows a judge system to reuse the results of judging the test case, the former does not. This is why I would like to use that definition.
There was a problem hiding this comment.
Ok, maybe we do, but then we have two different kinds of equivalence.
Yes. For a concrete example: Sweden has a problem asking "find the min and max possible thing for a given input", with subtask "you only need to find the max correctly". I would argue that the most correct solution in this instance is that this property is part of the group via output_validator_flags, not any testcase itself, and we want to be able to reuse them (problem in question: https://po2punkt0.kattis.com/problems/robottavlingen)
There was a problem hiding this comment.
Ah... I see you have not read the entirety of me and @thorehusfeldt's discussion in #523 😏.
In that discussion the consensus seems to go towards output_validator_flags being part of "the test case". I think @thorehusfeldt's is arguing from a point of "the sameness of a test case should imply the sameness of the judgement of said test case", and I would agree with that. I feels strange to say that you could pass a test case and then fail "the same" test case? They are quite obviously not the same then.
So, IMO, what you are talking about is identical input, not identical test cases. I would argue that that can be sufficiently handled by symlinks of copying?
Closes #539
I don't feel we had consensus on the issue yet, but this is how it could be done.
Please chime in...