Crawl pvi screens

[tomkane-dls]

produce a correct tree by resolving links like Phoebus does

The main result of this PR is jsonmap entries now look like this:

-          <IOC>https://t01-opis.diamond.ac.uk/bl01t-mo-motor-01</IOC>
+          <IOC>../../../../bl01t-mo-motor-01</IOC>

removing the generate-jsonmap command from the CI of b01-1-services has allowed for total removal of any URLs. A beta version is deployed on b01-1

---

Changes:

• All specific references to pvi/remote screens are gone, they are simply resolved relative to the entry point as seen above. PVI regex is gone, beamline_url is no longer used, any refernces to urls dropped. The $(IOC) macro maybe could have a better name something like $(service_dir)?

• resolve_link() expands macros in the link text and resolves the result relative to the screen containing the link. This replaces resolve_link_path/find_local_screen, which matched $(IOC)/ with a regex and resolved everything against a fixed link_base_dir using the component's service name.

• Macros inherit down the crawl — CrawlContext now carries the screen being crawled and the accumulated macros - inherit_macros() expands a link's macro values against the parent's before overriding. Previously macros were only attached to the resulting node, never used for resolution. CrawlContext.with_screen_component became with_screen and service_name is gone.

• ScreenNode.duplicate removed as it is unused.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.25%. Comparing base (4966f69) to head (b1d0557).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #272      +/-   ##
==========================================
+ Coverage   96.23%   96.25%   +0.02%     
==========================================
  Files          17       17              
  Lines        1088     1094       +6     
==========================================
+ Hits         1047     1053       +6     
  Misses         41       41              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[OCopping]

When running this against i19-services, some files get malformed file paths, e.g. for S3:

"file": "techui-support/bob/slits/slit-4axis.bob",
"children": [
    {
        "file": "techui-support/bob/slits/slit-4axis-individual-embed.bob",
        "children": [
            {
                "file": "techui-support/bob/pmac/$(IOC)/pmacAxis.pvi.bob",        <-----------
                "exists": false,
                "macros": {
                    "P": "BL19I-AL-SLITS-03",
                    "M": ":Y:POS",
                    "IOC": "$(IOC)",
                    "label": "Y Pos"
                },
                "displayName": "X Center"
            },

[OCopping]

A similar thing happens with DCM.bob (NOTE: a validated screen) on i19-services:

"file": "DCM.bob",
"children": [
    {
        "file": "techui-support/bob/pmac/https:/i19-opis.diamond.ac.uk/bl19i-mo-dcm-01/pmacAxis.pvi.bob",      <--------
        "exists": false,
        "macros": {
            "P": "BL19I-MO-DCM-01",
            "M": ":BRAGG",
            "label": "BRAGG",
            "IOC": "https://i19-opis.diamond.ac.uk/bl19i-mo-dcm-01"
        },
        "displayName": "BRAGG"

[tomkane-dls]

A bit more context, as i understand it -the changes here in this PR would remove all references to the opis url. I will refer to them as generated screens, as within this context nothing is really 'remote' from techui-builders perspective.

All paths, to locally created screens (like SPEC_1.bob in b01-1), and to screens generated by other services from fastcs or pvi (like bl01c-ea-flip-01/BL01C-EA-FLIP-01.bob). are relative. The IOC macro passed down allows any screen to navigate back to the top level directory.

for example - from inside SPEC_1.bob, go down into techui-support to the motor screen, and pass down the relative location from that screen, back to the service directory for the pmac ioc, where this screen will find its pvi generated screens:

        <file>techui-support/bob/pmac/motor_embed.bob</file>
        <macros>
          <M>:Z</M>
          <P>BL01C-MO-SPEC-01</P>
          <label>Z</label>
          <IOC>../../../../bl01c-mo-ioc-01</IOC>
        </macros>

When this is ran in kubernetes on ioc startup, all the generated screens from other services are mounted in the pod, so techui-builder can crawl everything locally inside the opis/ folder which has this structure:

opis/
  bl01c-di-dcam-01/
  bl01c-di-dcam-02/
  bl01c-di-dcam-03/
  bl01c-di-dcam-04/
  bl01c-di-dcam-05/
  bl01c-di-locum-01/
  bl01c-ea-align-01/
  bl01c-ea-flip-01/
  bl01c-ea-flip-02/
  bl01c-ea-flip-03/
  bl01c-ea-panda-01/
  bl01c-mo-ioc-01/
  bl01c-mo-ppanda-01/
  bl01c-mo-sim-01/
  bl01c-synoptic/

The relative paths are put into entries for every screen in the JsonMap. Daedalus can then navigate in the same way relatively, seeing the same structure as inside the opis/ above from its entry point:

https://b01-1-opis.diamond.ac.uk/bl01c-synoptic/

With absolute paths, we would need to fetch the screens generated by other services to be able to crawl them, otherwise we can only tell daedalus about the first screen (like https://b01-1-opis.diamond.ac.uk/bl01c-di-dcam-01/ADUVC.pvi.bob), we cannot crawl it to find the screens that it might link to.

With relative paths this does mean that when running techui-builder locally we will not find any of the generated screens as they are not contained in b01-1-services. The JsonMap will be much smaller and running daedalus locally from this file would not find any of the generated screens, but this is still similar to what was happening previously where we were only creating the path to the first pvi screen, and were not able to crawl it. I believe to have the same testing locally we would need to fetch all the screens from https://b01-1-opis.diamond.ac.uk so they could be crawled.

[tomkane-dls]

A similar thing happens with DCM.bob (NOTE: a validated screen) on i19-services:

"file": "DCM.bob",
"children": [
    {
        "file": "techui-support/bob/pmac/https:/i19-opis.diamond.ac.uk/bl19i-mo-dcm-01/pmacAxis.pvi.bob",      <--------
        "exists": false,
        "macros": {
            "P": "BL19I-MO-DCM-01",
            "M": ":BRAGG",
            "label": "BRAGG",
            "IOC": "https://i19-opis.diamond.ac.uk/bl19i-mo-dcm-01"
        },
        "displayName": "BRAGG"

hmm I would have expected this to become like the below - i will investigate why this isnt happening here

"file": "DCM.bob",
"children": [
    {
        "file": "techui-support/bob/pmac/pmacAxis.pvi.bob",      <--------
        "exists": false,
        "macros": {
            "P": "BL19I-MO-DCM-01",
            "M": ":BRAGG",
            "label": "BRAGG",
            "IOC": "../../../../bl19i-mo-dcm-01"
        },
        "displayName": "BRAGG"

[tomkane-dls]

Ok i didn't encounter these problems as they are not present on b01-1 or in t01 example.

DCM/Validated screens
I believe the issue with DCM is that the saved screen still has the old URL, and that the validator only checks macro names, not values. The validated screens would need to either have their IOC macros changed by hand, or delete and regenerate the screens and reimplement the hand crafted changes again. Maybe it could be good to add a warning if a change in the IOC macro is detected when checking against validated screens.

S3/Screens using autofill - note i don't see S3 but assuming its the same as S1
Screens that link a file in techui.yaml don't pass through the generator, so they never get an IOC macro added. I believe quick fix is to add the IOC macro into the techui.yaml. I think given how dynamic everything is with techui-builder currently, this is maybe the right call for now.

macros:
  P: BL19I-AL-SLITS-01
  IOC: ../../../../bl19i-mo-ioc-01                <-----

I haven't tested this yet, so i will check these fixes work as i say they do

[tomkane-dls]

Okay yes i believe the above fixes, adding an IOC macro in techui.yaml for the slits (if we are happy with this as a fix if choosing to bypass the generator) and regenerating validated screens both work when i tested locally:

DCM:

{
    "file": "DCM.bob",
    "children": [
        {
            "file": "../bl19i-mo-dcm-01/pmacAxis.pvi.bob",
            "macros": {
                "M": ":BRAGG",
                "P": "BL19I-MO-DCM-01",
                "label": "BRAGG",
                "IOC": "../../../../bl19i-mo-dcm-01"
            },
            "displayName": "BRAGG"
        },

SLITS:

"file": "techui-support/bob/slits/slit-4axis-individual-embed.bob",
"children": [
    {
        "file": "../bl19i-mo-ioc-01/pmacAxis.pvi.bob",
        "exists": false,
        "macros": {
            "P": "BL19I-AL-SLITS-01",
            "IOC": "../../../../bl19i-mo-ioc-01",
            "M": ":Y:POS",
            "label": "Y Pos"
        },
        "displayName": "X Center"
    },

[OCopping]

Ah yeah, if you look in validator.py I had started to look in to the macro overwriting but at the time I wrote it I didn't have a good solution. Now that a lot has changed (e.g. things moving in to utils.py) maybe this will be an easy addition?
I can have a look in the next couple days, unless you want to.

techui-builder/src/techui_builder/validator.py

Lines 106 to 119 in 2dd3dc6

	# ---------- This is how we could overwrite macros in the future ----------
	# for expected_macro in diff_expected_macros:
	# macro_element = Element(expected_macro)
	# # Get the macro value from generated pwidget macros
	# macro_element.text = pmacros[expected_macro]
	# print(pmacros[expected_macro])
	# # Convert xml.etree.Element to ObjectifiedElement
	# new_macro = fromstring(tostring(macro_element))
	# file_widget.macros.append(new_macro)
	# write_bob("")

[tomkane-dls]

• Okay i have decided to leave renaming the IOC macro (i think either to (OPI_DIR) or (SERVICE_DIR)) for another PR as it will involve coordinating with techui-support and services repos.

@OCopping I am busy on the ViSR sprint from now, so i won't have time to have a look at the validator stuff currently. If you could look at it that would be great - do you think that belongs on another PR or on this one?