instanceID is no longer recognized as a valid reference in v4.x

[tiritea]

Previous (AFAICT pyxform v3.x and earlier) the ${instanceID} reference was correctly interpreted by pyxform, and would resolve to the meta/instanceID element, allowing you to basically reference the computed submission ID from within the form. For example, the following XLSForm

SubmissionID.xlsx

produces the expected (and desired):

% which xls2xform

/Library/Frameworks/Python.framework/Versions/3.12/bin/xls2xform

% pip show pyxform
Name: pyxform
Version: 3.1.0
Summary: A Python package to create XForms for ODK Collect.
Home-page: https://pypi.python.org/pypi/pyxform/
Author: 
Author-email: "github.com/xlsform" <support@getodk.org>
License: 
Location: /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages
Requires: defusedxml, openpyxl, xlrd
Required-by: 

% xls2xform ~/Desktop/SubmissionID.xlsx      
Conversion complete!

% cat  SubmissionID.xml | xmllint --format -
<?xml version="1.0"?>
<h:html xmlns="http://www.w3.org/2002/xforms" xmlns:h="http://www.w3.org/1999/xhtml" xmlns:ev="http://www.w3.org/2001/xml-events" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:jr="http://openrosa.org/javarosa" xmlns:orx="http://openrosa.org/xforms" xmlns:odk="http://www.opendatakit.org/xforms">
    <h:head>
        <h:title>Submission ID</h:title>
        <model odk:xforms-version="1.0.0">
            <instance>
                <data id="SubmissionID">
                    <mygroup>
                        <name/>
                        <isParent/>
                        <id/>
                    </mygroup>
                    <meta>
                        <instanceID/>
                    </meta>
                </data>
            </instance>
            <instance id="yes_no">
                <root>
                    <item>
                        <name>0</name>
                        <label>no</label>
                    </item>
                    <item>
                        <name>1</name>
                        <label>yes</label>
                    </item>
                </root>
            </instance>
            <bind nodeset="/data/mygroup/name" type="string"/>
            <bind nodeset="/data/mygroup/isParent" type="string"/>
            <bind nodeset="/data/mygroup/id" type="string" calculate=" /data/meta/instanceID " readonly="true()"/>
            <bind nodeset="/data/meta/instanceID" type="string" readonly="true()" jr:preload="uid"/>
        </model>
    </h:head>
    <h:body>
        <group appearance="field-list" ref="/data/mygroup">
            <input ref="/data/mygroup/name">
                <label>What is your name?</label>
            </input>
            <select1 ref="/data/mygroup/isParent">
                <label>Do you have any children?</label>
                <itemset nodeset="instance('yes_no')/root/item">
                    <value ref="name"/>
                    <label ref="label"/>
                </itemset>
            </select1>
            <input ref="/data/mygroup/id">
                <label>Submission ID</label>
            </input>
        </group>
    </h:body>
</h:html>

However, under the latest pyxform 4.1 master, this form now throws an error about an unidentified 'instanceID' reference

% pip uninstall pyxform
% pip install git+https://github.com/XLSForm/pyxform.git@master#egg=pyxform

Collecting pyxform
...
Successfully installed lark-1.3.1 openpyxl-3.1.5 pyxform-4.1.0

% pip show pyxform

Name: pyxform
Version: 4.1.0
Summary: A Python package to create XForms for ODK Collect.
Home-page: https://pypi.python.org/pypi/pyxform/
Author: 
Author-email: "github.com/xlsform" <support@getodk.org>
License: 
Location: /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages
Requires: defusedxml, lark, openpyxl, xlrd
Required-by: 

% xls2xform ~/Desktop/SubmissionID.xlsx      
Conversion complete!
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.12/bin/xls2xform", line 8, in <module>
    sys.exit(main_cli())
             ^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyxform/xls2xform.py", line 259, in main_cli
    warnings = xls2xform_convert(
               ^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyxform/xls2xform.py", line 135, in xls2xform_convert
    result = convert(
             ^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyxform/xls2xform.py", line 99, in convert
    pyxform_data = workbook_to_json(
                   ^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyxform/xls2json.py", line 1447, in workbook_to_json
    validate_pyxform_references_in_workbook(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyxform/validators/pyxform/pyxform_reference.py", line 285, in validate_pyxform_references_in_workbook
    validate_pyxform_reference_syntax(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyxform/validators/pyxform/pyxform_reference.py", line 201, in validate_pyxform_reference_syntax
    raise PyXFormError(
pyxform.errors.PyXFormError: [row : 5] On the 'survey' sheet, the 'calculation' value is invalid. Reference variables must contain a name from the 'survey' sheet. Could not find the name 'instanceID'.

This can also be verified by fact this form appears to presently deploy fine under Central, and runs fine under latest ODK Collect fine

but the latest XLSForm Online with throw an error for it:

I assume ${instanceID} hasn't now been explicitly excluded for some reason, so this appears to be a regression of some manner?

[lognaturel]

I agree this is unintentional! @lindsay-stevens let's take this one on soon as well.

[lindsay-stevens]

@lognaturel There doesn't seem to be any specs, documentation or tests that mention or show that metadata fields (e.g. instanceID) or hidden questions (e.g. username) can be used in reference variables - only user-specified question, group, or repeat names. So perhaps this is more of an issue about officially supporting that? Is the intended behaviour that any name (system or user specified) in the primary instance can be referenced with a ${} lookup? In the meantime the workaround would be to write the expected XPath instead of ${instanceID}.