Skip to content

reproduce

Phase 4 — re-run determinism check: every statistic must return essentially identical.

reproduce

Phase 4 — reproducibility.

Re-run the same Python analysis a second time (with the same seed, if any) and confirm every statistic comes back essentially identical. This tests determinism within one process — it is not a guarantee of reproducibility on a different machine, OS, or BLAS/library build.

The default tolerance is extremely tight (rtol = 1e-12) rather than bit-exact, so genuinely deterministic code still passes when a multithreaded BLAS reduces sums in a slightly different order between two calls (last-ULP drift). Set reproducibility: {atol, rtol} in the project to tighten or loosen it, or pin OMP_NUM_THREADS=1 if you require bit-for-bit equality.

Note on randomness: a shared seed makes a same-tool re-run reproducible, but it does NOT align random streams across Python and R (the two use different RNGs), so the Phase 5 cross-tool comparison is meaningful only for deterministic estimators, not seed-matched random draws.

reproducibility(run1, run2, tol=None)

Compare two runs of the same analysis and flag any statistic that drifted.

Walks the union of keys across run1 and run2. A key present in only one run fails as a presence check; a key present in both is compared with :func:~crossverify.checks.is_close under the resolved tolerance. The default tolerance is extremely tight (rtol = 1e-12) rather than bit-exact, so genuinely deterministic code still passes when a multithreaded BLAS reduces sums in a slightly different order between two calls (last-ULP drift).

Parameters:

Name Type Description Default
run1

Mapping of statistic name to value from the first run.

 required
run2

Mapping of statistic name to value from the second (re-)run.

 required
tol

Optional tolerance mapping; reads atol (default DEFAULT_ATOL) and rtol (default DEFAULT_RTOL). None is treated as an empty mapping.

 None

Returns:

Type Description

A list of phase-4 CheckResult records, one per statistic in the union of

run1 and run2: a failed presence check when a key is missing from either

run, otherwise a pass/fail on whether the two values agree within tolerance.
Source code in crossverify/reproduce.py 
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
def reproducibility(run1, run2, tol=None):
    """Compare two runs of the same analysis and flag any statistic that drifted.

    Walks the union of keys across ``run1`` and ``run2``. A key present in only one
    run fails as a presence check; a key present in both is compared with
    :func:`~crossverify.checks.is_close` under the resolved tolerance. The default
    tolerance is extremely tight (``rtol = 1e-12``) rather than bit-exact, so genuinely
    deterministic code still passes when a multithreaded BLAS reduces sums in a slightly
    different order between two calls (last-ULP drift).

    Args:
        run1: Mapping of statistic name to value from the first run.
        run2: Mapping of statistic name to value from the second (re-)run.
        tol: Optional tolerance mapping; reads ``atol`` (default ``DEFAULT_ATOL``) and
            ``rtol`` (default ``DEFAULT_RTOL``). ``None`` is treated as an empty mapping.

    Returns:
        A list of phase-4 ``CheckResult`` records, one per statistic in the union of
        ``run1`` and ``run2``: a failed presence check when a key is missing from either
        run, otherwise a pass/fail on whether the two values agree within tolerance.
    """
    tol = tol or {}
    atol = tol.get("atol", DEFAULT_ATOL)
    rtol = tol.get("rtol", DEFAULT_RTOL)
    out = []
    for k in sorted(set(run1) | set(run2)):
        a, b = run1.get(k), run2.get(k)
        if k not in run1 or k not in run2:
            out.append(
                CheckResult(
                    "4",
                    f"repro:{k}",
                    f"{k} present on both runs",
                    False,
                    f"run1={fmt(a)} run2={fmt(b)}",
                )
            )
            continue
        ok = is_close(a, b, atol=atol, rtol=rtol)
        out.append(
            CheckResult(
                "4", f"repro:{k}", f"Re-run identical: {k}", ok, f"run1 = {fmt(a)}, run2 = {fmt(b)}"
            )
        )
    return out