Skip to content

Add forensic metascience checks to the reproduction intro (#57) - #76

Open
LukasWallrich wants to merge 2 commits into
mainfrom
content/issue-57-forensic-metascience
Open

Add forensic metascience checks to the reproduction intro (#57)#76
LukasWallrich wants to merge 2 commits into
mainfrom
content/issue-57-forensic-metascience

Conversation

@LukasWallrich

Copy link
Copy Markdown
Contributor

The "Reproduction before Replication" section of planning.qmd previously described only p-value recomputation (statcheck, papercheck) for the case where neither code nor data are available. It now covers forensic metascience more broadly.

planning.qmd

  • The paragraph states that a reproduction in the strict sense is impossible without data, but that the internal consistency of the reported results can still be checked.
  • Adds GRIM (whether a reported mean can arise from integer or granular data given sample size and reported precision), GRIMMER (the same logic for standard deviations), SPRITE (hypothetical samples compatible with the reported summary statistics and with response-scale bounds), and cross-paper consistency checks such as subgroup sizes summing to the stated sample size.
  • States what such checks can and cannot show: a flag identifies a result that is inconsistent or impossible under the tool's assumptions and is worth raising with the original authors, it can also arise from rounding or transcription slips, and it does not by itself establish that a finding is wrong. Passing the checks says nothing about replicability.
  • Cites @Heathers2025 for an accessible introduction and links to the Uncertainty section, where these techniques already appear as an aid to choosing a target.

Figure 4.1 (images/WBn_Image_3.png)

  • The right-hand box now reads "Check consistency of reported results (forensic metascience tools, e.g. statcheck)" instead of "Check correctness of reported results via statcheck.io". The box was extended downwards to fit four lines; borders, arrows and font size match the rest of the figure (Arial at the same size as the original text). There is no vector source for this figure in the repository, so the PNG was edited directly.

execution_reproductions.qmd, Gathering resources

  • Adds a short paragraph on data integrity checks once the data are obtained: value ranges, variable codes, labels and missing-value conventions against the codebook, sample size and exclusions against the report, anomalies such as impossible values, duplicated rows and missing-value codes read as observations, and recomputation of the reported descriptive statistics. Discrepancies should be checked against documented exclusions and transformations before being raised with the authors.

references.bib

  • Heathers2025 url updated to https://jamesheathers.curve.space/; the Zenodo DOI is unchanged.

Both chapters render without citation or cross-reference warnings.

Closes #57

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add brief discussion of forensic metascience / techniques without data to reproduction intro and refer to that in Figure 4.1

1 participant