Clarification: dataset row count differs from EMBER2024 docs (3,232,000 vs 6,464,000)

[tvquynh]

Hi authors,
The EMBER2024 notebook/paper lists the dataset size as 3,232,000 rows, but the file I just downloaded shows 6,464,000 rows (rows doubled, columns unchanged).

Is this the intended updated release, or did I download the wrong artifact?

If intended, could you confirm the expected size and update the docs/changelog (and ideally provide a version tag/checksum)?

Thanks!

[tvquynh]

I found the duplicated sha256 in the same .jsonl file. Please help to check at your side.

Thanks!

[sadia4444a]

Hi,
I also found the same issue after downloading the EMBER2024 dataset. The .jsonl file contains duplicated sha256 entries, which caused the total number of rows to become 6,464,000 instead of the 3,232,000 rows mentioned in the paper.
To proceed with my research, I preprocessed the .jsonl file and removed duplicate samples based on the sha256 field to restore the expected unique entries.