From 82adad1f26a53dbde3b5410ea0a9542811ed3e49 Mon Sep 17 00:00:00 2001 From: Lukas Wallrich Date: Sun, 2 Aug 2026 12:50:56 +0200 Subject: [PATCH] FAQ: the reproduction axes aggregate in their own words, and chains name a target MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two rules the extraction pipeline codes against but the FAQ did not state. Aggregation across several analyses was written only in the replication vocabulary, so "conflicting results" pointed at *mixed* — a value reproductions do not have. Each axis now aggregates in its own terms. Replication chains had no entry at all: the rule lived in a project doc as a fragment ("most likely the closest successor study") whose direction is ambiguous read alone. Stated with an example instead. Co-Authored-By: Claude Fable 5 --- output/flora_faq.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/output/flora_faq.md b/output/flora_faq.md index fb9e890..2872ff9 100644 --- a/output/flora_faq.md +++ b/output/flora_faq.md @@ -43,6 +43,12 @@ Then it will be listed twice in FLoRA, once as a replication and once as a repro Each row is one pair of references. A replication study can have multiple studies but their results are aggregated (e.g., based on the authors’ judgment, who may report aggregated results or conflicting results, which we consider as mixed). +For reproductions, the same aggregation applies to each outcome axis separately, in that axis’s own vocabulary: conflicting results across several analyses mean *computational issues* on the computational axis, and *robustness challenges* on the robustness axis. There is no *mixed* value for reproductions. + +### Which study is the target when a finding has already been replicated by others? + +The target is the study the replication actually re-tests — the one closest to it in the chain of replications — not the chain’s original source. For example, a 2024 paper re-testing a 2010 finding as it was measured in a 2018 replication is entered against the 2018 study. + ### When is a finding mixed? We rely on what replication authors say in the abstract or the report, subject to our interpretation of this. If studies are part of a meta-paper, we still rely on the individual reports where available, and otherwise on the main criterion used by the paper authors.