Skip to content

fixed bug #2116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
niveditasing wants to merge 9 commits into
base: master
Choose a base branch
from
+22 −2
Open

fixed bug #2116

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
dc35b26
fixed bug
niveditasing Jul 28, 2026
cad7776
fixed bug
niveditasing Jul 28, 2026
63e0d21
Merge branch 'master' into Bug_fix_validator_goldens
niveditasing Aug 4, 2026
c9c53ee
Merge branch 'master' into Bug_fix_validator_goldens
niveditasing Aug 5, 2026
5c544e6
added runner.py
niveditasing Aug 5, 2026
b000070
import_validation: simplify list normalization in goldens validation
niveditasing Aug 5, 2026
b981816
statvar_importer: fix list sorting and bracketed list normalization i…
niveditasing Aug 5, 2026
26036e4
statvar_importer: revert list normalization changes in mcf_diff.py
niveditasing Aug 5, 2026
738e52e
added test
niveditasing Aug 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion tools/import_validation/runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,8 @@ def _load_differ_df_from_mcf(self, input_dir: str) -> pd.DataFrame:
stats[current_var][diff_type] += 1

if not stats:
return pd.DataFrame()
return pd.DataFrame(
columns=['StatVar', 'ADDED', 'DELETED', 'MODIFIED'])

rows = []
for var, counts in stats.items():
Expand Down
6 changes: 5 additions & 1 deletion tools/statvar_importer/mcf_file_util.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -826,7 +826,7 @@ def normalize_list(value: str, sort: bool = True) -> str:
value_list = get_value_list(value)
has_quotes = True
else:
value_list = value.split(',')
value_list = [v.strip() for v in value.split(',')]

@ajaits ajaits Aug 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the normalize_value() call below does remove extra namespace and other normalizations too like namespaces. this can be removed.

@niveditasing niveditasing Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

normalize_value() strips, but it runs after sort(). Without stripping here, leading spaces alter the sort order and break our golden files

values = []
if sort:
value_list = sorted(value_list)
Expand Down Expand Up @@ -931,6 +931,10 @@ def normalize_value(
if value[0] == '"' and value[-1] == '"' and len(value) > 100:
# Retain very long strings, such as geoJsonCoordinates, as is.
return value
if value.startswith('[') and value.endswith(']') and ',' in value:
Comment thread
niveditasing marked this conversation as resolved.
inner_list = value[1:-1].strip()
normalized_list = normalize_list(inner_list)
return f'[{normalized_list}]'
if ',' in value and maybe_list:
return normalize_list(value)
if value[0] == '[':
Expand Down
15 changes: 15 additions & 0 deletions tools/statvar_importer/mcf_file_util_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,21 @@ def test_normalize_range(self):
self.assertEqual('dcid:Upto20',
mcf_file_util.normalize_value('[ - 20 ]', True))

def test_normalize_value_list_with_brackets(self):
# Test bracketed lists of values with commas.
self.assertEqual(
'[dcid:A,dcid:B]',
mcf_file_util.normalize_value('[dcid:B,dcid:A]'),
)
self.assertEqual(
'[dcid:A,dcid:B]',
mcf_file_util.normalize_value('[ dcid:B, dcid:A ]'),
)
self.assertEqual(
'["ValueA","ValueB"]',
mcf_file_util.normalize_value('[ "ValueB", "ValueA" ]'),
)

def test_load_mcf_file(self):
mcf_nodes = mcf_file_util.load_mcf_nodes(
os.path.join(_module_dir_, 'test_data',
Expand Down
Loading