AI ML RELIABILITY INVESTIGATOR

Every failed model is trying
to tell you something.

Sentry for software · Datadog for infra · WhyLab for machine learning

Upload your evaluation data. WhyLab investigates why your model is failing, proves the root cause with counterfactual experiments, and repairs your operating policy.

YOUR MODEL SAYS94.2% ACCURACYLooks excellent at first glance

WHYLAB SAYSHIGH-RISK FAILURE DETECTEDMalignant recall: 22.4% · Misses 3 out of 4 cancers

Evidence, not guessworkScientific provenanceFalsification engine

Launch Investigation Studio Run Flagship Melanoma Demo Explore Vision Lab

1-CLICK DETERMINISTIC BENCHMARK

Interactive Melanoma Failure Demonstration

Test the accuracy paradox deterministically. Follow the evidence, evaluate counterfactual tests, and inspect the measured threshold repair.

FLAGSHIP CASE / REPRODUCIBLE LOCAL EXPERIMENT

High accuracy. Missed malignant cases.

A synthetic melanoma classifier exposes the accuracy paradox. Follow the evidence, test competing explanations, then inspect a measured policy repair.

Synthetic teaching data: 1 = malignant, 0 = benign. No patient data or clinical validation. This deterministic local protocol runs without an API key; it is separate from the autonomous Astra investigator.

Validation Accuracy94.2%Looks healthy at a glance

Malignant Recall22.4%Misses 3 out of 4 malignant cases

MEASURED OPERATING-POINT SWEEP

Decision Threshold vs. Malignant Recall & Error Cost

100 evaluation rows · 10 malignant / 90 benign

Malignant Recall (%) Total Error Cost (Units)Hover or tap points along curve

OPERATING POINT AT T = 0.20RECOMMENDED

Malignant Recall100.0%10 of 10 detected

Missed Cancers (FN)0 casesZero false negatives

Total Error Cost4 unitsFN×10 + FP×1

Overall Accuracy96.0%96 of 100 correct

Baseline (T=0.50):92.0% Acc · 20.0% Recall · Cost: 80

Repaired (T=0.20):96.0% Acc · 100.0% Recall · Cost: 4

Measured impact: 95% reduction in error cost and +80 pp malignant recall, without sacrificing overall accuracy.

Download source CSV

Want to investigate your own evaluation dataset with autonomous tool calling?

Open Investigation Studio

REAL-WORLD INCIDENT BENCHMARKS

Proven Reliability Case Studies

Explore real-world failure patterns: probability calibration drift in clinical ICU predictions and covariate site shift in multi-hospital deployments.

ADDITIONAL INVESTIGATIONS

More investigations

Two synthetic case studies, measured locally from downloadable CSVs. No saved state or API key required.

Interactive Case

Confident predictions, unreliable probabilities

Does high prediction confidence match observed correctness?

Download case CSV

SCIENTIFIC METHOD FOR ML

How WhyLab Investigates Failure

WhyLab replaces guesswork with a rigorous 4-step causal discovery pipeline.

STEP 01

1. Observe the Signal

Ingest evaluation outputs (y_true, y_pred, y_probability). Expose accuracy paradoxes and slice performance drop-offs across production environments.

STEP 02

2. Form Competing Hypotheses

Astra generates falsifiable explanations: class imbalance, covariate shift, calibration drift, or data leakage. Every hypothesis is tied to evidence.

STEP 03

3. Falsification Testing

Autonomous diagnostic tool calls run statistical tests (Fisher's exact, Wilson intervals, stratified resampling) to actively attempt to disprove hypotheses.

STEP 04

4. Repair & Re-Test

Sweep 101 operating boundaries under your domain cost model. Re-test the optimized policy on an unchanged holdout to mathematically prove recovery.

Every failed model is trying to tell you something.