Contest: prepare HW split - #100
Conversation
To prepare a split between HW and the rest. Signed-off-by: Matthieu Baerts <matttbe@kernel.org>
This file is read by another service (faker), possibly simultaneously. Signed-off-by: Matthieu Baerts <matttbe@kernel.org>
It looks like it is not / no longer used. So drop that. Note that the combined 'info' file will now only be used by the UI (status page), not by pw_contest.py anymore. Signed-off-by: Matthieu Baerts <matttbe@kernel.org>
To support the HW branches, deal with multiple input branch URLs, and multiple output combined files. This sounds better than having to deal with different services writing in the same DB. Config changes: - branch_url -> branch_urls: with multiple URLs, separated by ',' - combined -> combined_files: output files, separated by ',' Signed-off-by: Matthieu Baerts <matttbe@kernel.org>
This limitation was only done for in progress work, but we don't need the old results. Old results are still on the different runners (and in the DB). (Probably the different runners should also trim their 'results.json' file.) Because of that, 'all-results.json' is currently over 410MB, which takes a bit of time to process here and in pw_contest. The other consequence is that some files for the UI contains info about old branches that are not needed, but are still processed on the client side, e.g. branch-results.json. Signed-off-by: Matthieu Baerts <matttbe@kernel.org>
56495aa to
e910d1f
Compare
Generated in another file by the dedicated pw-contest service. Signed-off-by: Matthieu Baerts <matttbe@kernel.org>
e910d1f to
558538d
Compare
| info = config.get("output", "info") | ||
| with open(info, 'w') as fp: | ||
| json.dump(state["info"], fp) | ||
| write_json_atomic(config.get("output", "info"), state["info"]) |
There was a problem hiding this comment.
Is this going to work? Last time we did this the writer did not have write permissions to the directory where the files are. So we could not do an atomic swap. Please double check if this will be a problem for branches info?
There was a problem hiding this comment.
I already checked the directory permission (and created the new files), but I will sure double check, thank you for the reminder
| dir=/path/to/output | ||
| url_pfx=relative/within/server | ||
| info=/path/to/info.json | ||
| """ |
There was a problem hiding this comment.
I don't understand this commit. If you want to get rid of the faker why not delete the whole thing? You're removing the main reason for its existence. Combining infos can move to the brancher probably?
There was a problem hiding this comment.
The combined infos is still used by the UI of I remember well (status I think, I can check when I have access)
| dir=/path/to/output | ||
| url_pfx=relative/within/server | ||
| combined=name-of-manifest.json | ||
| combined_files=name-of-manifest1.json,name-of-manifest2.json |
There was a problem hiding this comment.
Sure but also - why? Is it that bad to have two instances running?
There was a problem hiding this comment.
I was feeling this safer, not too have two confident writers (for the DB, even if it is supposed to be safe), and to keep the remote DB as is.
But I can revert back to two services (what I had in the v1)
|
|
||
| branches.add(v.branch); | ||
|
|
||
| if (v.remote == "brancher") { |
There was a problem hiding this comment.
Isn't this entry here what faker is producing?
There was a problem hiding this comment.
Yes I think so, the combined info part. I can check when I have access.
2 participants
This prepares the future split, dealing with HW results in a different 'contest'.
The plan is:
contest/results-collector.py:pw_contest.py:results-collector.py, output one for status.htmlfaker: not needed any more for pw_contest, but still for status.html. (Note that the faker part was not used apparently, so removed in this PR)Note: of course, this shouldn't be deployed until the corresponding modifications are done on the hosts.