When Tests Fail: Tolerances, Silent Refactor Bugs, and the Reproducibility Record

Path Outline

Progress0%

1From Ask to Contract: What “Done” Must Mean Before You Touch Data 2Clarifying Questions That Lock Metric, Grain, and Scope 3Define the KPI So It Can’t Wiggle: Numerator, Denominator, and Inclusion Rules 4Turn an AI Draft Into a Human-Owned Analysis Plan 5Acceptance Tests, Audit Logs, Privacy Redaction, and Escalation Triggers

6Prompt Anatomy for Testable Analytics Drafts 7Plan-First Prompts That Surface Assumptions Before Code 8Prompts for Verifiable SQL: Constraints, Grain, and Test Queries 9Prompting for Robust Spreadsheet Formulas and Edge-Case Tests 10I/O Contracts, Minimal Tests, and a Verification Request Prompt for Pandas

11Give AI Just Enough Schema to Draft Safe Extraction 12Prove the Extraction Matches the Metric Grain 13Catch Join Multiplication Before It Corrupts Totals 14Lock Down Filters and Time Windows With Boundary Tests 15Build a Reconciliation Checklist You Can Reuse 16Make Transformations Deterministic and Reconcile Before vs After

17Select EDA Questions and Segments You Can Actually Prove 18Build Gold Summary Tables as Your Single Source of Truth 19Pressure-Test Statistical Interpretations Without Becoming a Stats Course 20Use an Interpretation Checklist to Catch Denominator and Seasonality Traps 21Draft Chart Specs and Reconcile Every Visual Back to a Gold Table 22Write an Evidence-Cited Executive Summary With Privacy Guardrails

23A Pipeline That Runs Is Not a Pipeline You Can Trust 24Make the Environment a First-Class Deliverable (venv + Dependencies)25Refactor the Notebook Into a Parameterized Script With an I/O Contract 26Eliminate Hidden State: Ordering, Seeds, and Time Zones 27Build Logging Checkpoints That Double as Regression Signals 28Equivalence Tests: Prove the Refactor Preserved Results

29When Tests Fail: Tolerances, Silent Refactor Bugs, and the Reproducibility Record

30Assemble Front Matter: The Analysis Contract a Reviewer Can Execute 31Package SQL and Transformations With Executed QA Evidence 32Make Claims Traceable: Map Dashboard and Narrative to Evidence 33Compile the Prompt, Assumptions, and Equivalence Evidence Into a Review Trail 34Run the Privacy Review and Publish an Audit-Pack Index and Self-Critique

Hide outline

Home
Paths
AI-Assisted Analytics Workflows: Intake to Audit Pack
When Tests Fail: Tolerances, Silent Refactor Bugs, and the Reproducibility Record