sagan-standard

name: sagan-standard description: Calibrate evidence requirements proportional to claim implausibility when evaluating extraordinary assertions

Sagan Standard

One-Liner

Extraordinary claims require extraordinary evidence—the more unusual the assertion, the stronger the proof needed.

Core Insight

The Sagan Standard, popularized by Carl Sagan ("extraordinary claims require extraordinary evidence"), establishes that the strength of evidence must be proportional to how unusual or improbable a claim is. It's not merely about requiring "more" evidence for surprising claims, but recognizing that claims contradicting massive existing evidence need proportionally stronger proof to rationally shift beliefs. The principle formalizes Bayesian reasoning: prior probability affects required evidence threshold.

Mental Model

Evidence Threshold Calibration:

Ordinary Claim (aligns with priors):
"I had coffee today"
Required Evidence: Minimal (your word suffices)

Moderately Unusual:
"I ran a 4-minute mile"
Required Evidence: Stopwatch, witnesses, video

Extraordinary (contradicts vast evidence):
"I can fly by flapping my arms"
Required Evidence: Rigorous controlled tests, replication, peer review

Prior Probability × Evidence Strength = Posterior Belief
(Very low prior) × (Modest evidence) = Still don't believe
(Very low prior) × (Overwhelming evidence) = Update belief

Bayesian Foundation: If P(claim) = 0.001%, evidence must shift likelihood ratio 100,000:1 to reach 50% confidence.

When to Use

Evaluating novel scientific claims: Proportion skepticism to claim's implausibility
Due diligence: Require stronger verification for too-good-to-be-true opportunities
Debugging: Mundane explanations need less proof than exotic ones
Hiring: Extraordinary resume claims demand extraordinary verification
Product claims: Revolutionary benefits require revolutionary evidence
Strategy validation: Paradigm-shifting plans need stronger evidence than incremental ones

Execution Steps

1. Calibrate Prior Probability

Ask: "How much existing evidence contradicts this claim?"
Quantify: Is this 1-in-10 unusual? 1-in-1000? 1-in-million?
Consider base rates: How often do claims like this turn out true?
Assess track record: Do similar extraordinary claims usually pan out?

2. Define "Extraordinary Evidence"

Not just "more evidence" but:

Independent replication: Others reach same conclusion independently
Mechanism explanation: Plausible causal story, not just correlation
Controls for bias: Pre-registration, blinding, adversarial review
Convergent evidence: Multiple independent lines pointing same direction
Ruled out alternatives: Mundane explanations actively eliminated

3. Apply Proportional Skepticism

Evidence Quality Ladder (lowest to highest):
1. Anecdote/testimonial
2. Uncontrolled observation
3. Correlation study
4. Controlled experiment
5. Replicated controlled experiments
6. Meta-analysis of quality studies
7. Converging evidence across multiple methodologies

Match required level to claim extraordinariness.

4. Avoid Common Misapplications

Not about dismissing new ideas: Novel ≠ extraordinary

Novel: "This new catalyst improves yield 10%"
Extraordinary: "This catalyst violates thermodynamics"

Not about impossible burdens: Must be achievable evidence

Bad: "Show me video of every atom behaving as quantum mechanics predicts"
Good: "Show me repeatable experimental results inconsistent with alternatives"

Not about doubling standards: Apply consistently

Apply same rigor to claims you like vs. dislike
Don't accept weak evidence for preferred conclusions

5. Implement Evidence Thresholds in Decision-Making

Hiring Example:

"5 years experience" → Check LinkedIn (ordinary)
"Increased revenue 300%" → Verify with references, see documentation (extraordinary)
"Invented algorithm 10x better than state-of-art" → Require proof-of-concept, technical deep-dive, independent review (very extraordinary)

Investment Example:

"Growing 20% YoY" → Review financials (ordinary)
"Discovered new market 10x current TAM" → Deep diligence, market evidence, customer validation (extraordinary)
"Technology defies physical limits" → Probably fraud, require independent scientific validation (extremely extraordinary)

6. Update Beliefs Proportionally

When extraordinary evidence appears: Update strongly
When ordinary evidence appears for extraordinary claim: Remain skeptical
Track claim's prior probability evolution as evidence accumulates
Don't freeze beliefs; allow extraordinary evidence to shift extraordinary claims

Real-World Examples

Cold Fusion (1989)

Claim: Room-temperature fusion in simple apparatus (contradicts massive physics evidence)
Evidence Provided: Single experiment, press conference announcement
Sagan Standard: Requires independent replication, mechanism explanation, peer review
Outcome: Couldn't be replicated; mundane explanation (experimental error) prevailed

Helicobacter Pylori Causes Ulcers (1982)

Claim: Bacteria cause ulcers (contradicted prevailing "stress and acid" theory)
Evidence Provided: Initially weak, but Marshall self-experimented, then rigorous studies
Sagan Standard: Unusual but not physics-violating; required good evidence, not impossible bar
Outcome: Replicated studies convinced community; Nobel Prize 2005

Theranos (2003-2018)

Claim: Comprehensive blood tests from finger prick (contradicted known technical limits)
Evidence Provided: Charismatic founder, impressive board, media hype, ZERO peer review
Sagan Standard Failure: Investors/partners didn't demand extraordinary evidence
Outcome: Fraud; technology didn't work

Quantum Mechanics (1900-1930)

Claim: Particles behave as waves, observer affects outcome (deeply counterintuitive)
Evidence Provided: Repeatable experiments, mathematical framework, convergent evidence
Sagan Standard: Extraordinary claim met with extraordinary evidence (replicated experiments worldwide)
Outcome: Accepted as foundation of modern physics

Common Traps

Trap 1: Impossible Standards

Demanding literally impossible evidence ("prove a negative")
Sagan Standard requires proportional evidence, not infinite evidence

Trap 2: Dismissing All Novelty

Conflating "new" with "extraordinary"
New ideas contradicting little existing evidence need good evidence, not extraordinary

Trap 3: Motivated Reasoning

Applying high bar to claims you dislike, low bar to claims you like
Must calibrate to actual prior probability, not preference

Trap 4: Static Priors

Refusing to update even when extraordinary evidence appears
Prior probability should shift as evidence accumulates

Trap 5: Confusing Extraordinary with Unlikely

"I won the lottery" is unlikely but not extraordinary (happens to someone)
"I always win the lottery" is extraordinary (contradicts probability)

Relationship to Other Principles

Occam's Razor: Prefer simpler explanations

Sagan Standard: Require stronger proof for complex explanations
Complementary: Simple explanation needs less proof

Hitchens's Razor: "What asserted without evidence can be dismissed without evidence"

More extreme: Demands some evidence for any claim
Sagan Standard: Demands evidence proportional to extraordinariness

Hume's Principle: "No testimony is sufficient to establish a miracle unless..."

Direct inspiration for Sagan Standard
Focused on miracles; Sagan generalized to all extraordinary claims

Cross-Domain Applications

Product Management: "This feature will 10x retention" → Show A/B tests, not gut feel

Sales: "Our ROI is 50x" → Require customer case studies, not vendor white paper

Debugging: "Cosmic ray bit flip" → Eliminate prosaic bugs first (memory leak, race condition)

Medicine: "This supplement cures cancer" → Phase III RCTs or it's noise

Academia: "This overturns established theory" → Replication, peer review, alternative explanations ruled out