111 create yaml template for problem submission #121
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| # Please enter the relevant information. | ||
| # Fields that are not relevant can be removed (delete the whole line), or left empty. | ||
| - name: | ||
| short: BBOB | ||
| full: Real-Parameter Black-Box Optimization Benchmarking | ||
| suite/generator/single: suite | ||
| objectives: | ||
| number: '1' # Can be scalable, enter a single number, a range: '1-5', or 'scalable', if it can be any value | ||
| variables: | ||
| types: continuous # Can be one or more of (continuous, integer, binary, categorical, mixed) in case of mixed, describe which mixes are possible | ||
| conditional: 'no' | ||
| dimensionality: scalable | ||
|
There was a problem hiding this comment.
|
||
| constraints: | ||
| present: 'no' | ||
|
Collaborator
There was a problem hiding this comment. This is confusing to enter, because they have boundary/box: yes.
Collaborator
Author
There was a problem hiding this comment. Yes, makes sense, we can also automatically derive the 'present' field from other fields later if we need it.
Collaborator
Author
There was a problem hiding this comment. Actually, for backwards compatibility with the existing template it might make sense to keep it for the moment. What do you think @CIGbalance |
||
| soft: '0' | ||
| hard: '0' | ||
| boundary/box: 'yes' | ||
|
There was a problem hiding this comment. We don't consider discrete, categorical variables here, right? In case of non-categorical variables, we expect the number of boxes to be equal to the number of variables. Could it be different? |
||
| permutation: 'no' | ||
|
There was a problem hiding this comment. A In the case of a mixed-integer problem, can there be a permutation present in the (parts of) discrete part? |
||
| dynamic: | ||
| present: 'no' # Enter, yes, no, or optional | ||
| types: '' | ||
| noise: 'no' | ||
|
There was a problem hiding this comment. With respect to consistency, I suggest to follow the |
||
| modality: | ||
| types: 'unimodal, multimodal' # Enter all that are present | ||
| evaluations: | ||
| multi-fidelity: 'no' | ||
| partial possible: 'no' | ||
| independent objectives: 'no' | ||
| reference: | ||
| links: | ||
| - https://doi.org/10.1080/10556788.2020.1808977 | ||
| authors: 'Nikolaus Hansen, Anne Auger, Raymond Ros, Olaf Mersmann, Tea Tušar, Dimo Brockhoff' | ||
| contact person: '' | ||
| implementations: | ||
| - name: COCO | ||
| link: https://github.com/numbbo/coco | ||
| languages: 'C, Python' | ||
| evaluation time: 'less than a second' | ||
|
Collaborator
There was a problem hiding this comment. This is going to be wild to analyse if we don't give some structure
Collaborator
Author
There was a problem hiding this comment. What would you suggest? @CIGbalance My initial idea would be to ask for something like: Are there good standards we can refer to/use?
Collaborator
Author
There was a problem hiding this comment. We did something similar for the real-world benchmarks questionnaire. There we had less than a second/minute/hour/day, or more than a day. Perhaps the |
||
| specific requirements: 'no' # Can be things like memory/GPU requirements, or the need for a specific simulator. Enter 'no' if none. | ||
| source: | ||
| real-world: | ||
kvdblom marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| domain: '' # E.g., automotive | ||
| open/closed: '' # Is the problem open source or not | ||
| artificial: 'yes' | ||
| other: 'no' | ||
| textual description: | ||
| general info: '' | ||
| motivation: 'evaluate algorithm performance for typical difficulties that occur in continuous problems' | ||
| challenge/key characteristics: '' | ||
| limitations: '' | ||
| other info: '' | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I still think it would be best to enter problems separately, but I understand the necessity of streamlining this process. So if we want to allow entering suites, maybe we have different templates? Because some values might only make sense for a suite? Otherwise, for a suite, a lot of the input would be "varies" for number of objectives, variables, constraints, dynamic, noise, etc?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it would be nice to have problems separately, I have also prototyped how it might work to have both the suite and component problems, see here:
OPL/problems.yaml
Line 1107 in 84b34b6
I am not sure about separate templates for suites or problems, but we can think about this and discuss what makes sense. I would imagine, e.g., the BBOB sphere still has a bunch of "varies", because it is scalable in the number of variables for example. For a suite, I would want an exhaustive list (e.g., noise: yes, no / optional, or something like that) of the options, rather than a vague "varies". I will describe this in a comment in the template.