Skip to content

triangulate

Phase 5 — cross-tool triangulation of Python vs R statistics within tolerance.

triangulate

Phase 5 — cross-tool triangulation.

Compare the Python results against an independent R implementation, statistic by statistic, within tolerance. This is the step that catches results which are artifacts of one tool's defaults rather than properties of the data. Set abs: true in a statistic's tolerance to compare magnitudes only, for quantities whose sign is implementation-defined (e.g. PCA loadings).

A mismatch fails the build by default. For a statistic that legitimately differs across tools for a defensible reason (robust-SE variant, ddof/denominator choice, contrast coding), declare severity: info in its per-key tolerance so the divergence is reported as INFO rather than FAIL — the run stays green and the disagreement is surfaced for a human to interpret, instead of pressuring the analyst to force one tool to mimic the other. A statistic that is simply absent in one tool is always a hard failure (the replication is incomplete), regardless of severity.

triangulate(py_results, r_results, tolerance)

Compare Python and R results statistic by statistic within tolerance.

Iterates the union of statistic keys. A statistic absent from either side is always a hard failure (the replication is incomplete). For statistics present in both, applies the per-key tolerance (including magnitude-only comparison when abs is set); a mismatch fails the build unless the statistic declares severity: info, in which case it is reported as INFO instead.

Parameters:

Name Type Description Default
py_results

Mapping of statistic name to value from the Python implementation.

 required
r_results

Mapping of statistic name to value from the R implementation.

 required
tolerance

Tolerance configuration mapping, typically Project.tolerance.

 required

Returns:

Type Description

A (checks, rows) tuple where checks is a list of

class:~crossverify.checks.CheckResult and rows is a list of comparison-row

dicts (keys stat, python, r, delta, match, note) for the

report table.
Source code in crossverify/triangulate.py 
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
def triangulate(py_results, r_results, tolerance):
    """Compare Python and R results statistic by statistic within tolerance.

    Iterates the union of statistic keys. A statistic absent from either side is always a
    hard failure (the replication is incomplete). For statistics present in both, applies
    the per-key tolerance (including magnitude-only comparison when ``abs`` is set); a
    mismatch fails the build unless the statistic declares ``severity: info``, in which
    case it is reported as INFO instead.

    Args:
        py_results: Mapping of statistic name to value from the Python implementation.
        r_results: Mapping of statistic name to value from the R implementation.
        tolerance: Tolerance configuration mapping, typically ``Project.tolerance``.

    Returns:
        A ``(checks, rows)`` tuple where ``checks`` is a list of
        :class:`~crossverify.checks.CheckResult` and ``rows`` is a list of comparison-row
        dicts (keys ``stat``, ``python``, ``r``, ``delta``, ``match``, ``note``) for the
        report table.
    """
    checks = []
    rows = []
    for k in sorted(set(py_results) | set(r_results)):
        a, b = py_results.get(k), r_results.get(k)
        atol, rtol, use_abs = tol_for(tolerance, k)

        if k not in py_results or k not in r_results:
            note = "missing in Python" if k not in py_results else "missing in R"
            checks.append(CheckResult("5", f"triangulate:{k}", f"Python vs R: {k}", False, note))
            rows.append(
                {"stat": k, "python": a, "r": b, "delta": None, "match": False, "note": note}
            )
            continue

        ok = is_close(a, b, atol, rtol, use_abs)
        delta = abs(abs(a) - abs(b)) if use_abs else abs(a - b)
        advisory = severity_for(tolerance, k) == "info"
        # On a mismatch, an advisory statistic reports INFO (passed=None) instead
        # of FAIL, so a defensible cross-tool divergence does not break the build.
        passed = True if ok else (None if advisory else False)
        notes = []
        if use_abs:
            notes.append("magnitude only")
        if advisory and not ok:
            notes.append("advisory: severity=info, not a failure")
        note = "; ".join(notes)
        detail = (
            f"python = {fmt(a)}, r = {fmt(b)}, |delta| = {fmt(delta)} "
            f"(atol={atol:g}, rtol={rtol:g}{', abs' if use_abs else ''}"
            f"{', advisory' if advisory else ''})"
        )
        checks.append(CheckResult("5", f"triangulate:{k}", f"Python vs R: {k}", passed, detail))
        rows.append({"stat": k, "python": a, "r": b, "delta": delta, "match": ok, "note": note})
    return checks, rows