# Validation Report — Petrel Karst & Depressions Layer v0.1

## Summary

The depression band carries a real, modest signal for sinkhole occurrence:
**AUC 0.649 [0.58, 0.71]**, measured on the shipped global raster against the
only commercially usable sinkhole inventory in existence. The solubility band is
**not** validated as a discriminator and is shipped as context.

## Test set

Florida DEP Subsidence Incident Reports, `TRUE_SINK` field:

| Class | n | Meaning |
|---|---|---|
| Y | 103 | Inspector-verified sinkhole |
| N | 108 | Investigated, confirmed NOT a sinkhole |
| U | 4,232 | Reported, never verified |

The **Y vs N** comparison is the hard discrimination and the one reported: both
classes are places where somebody filed a report and an inspector attended, so
the reporting-propensity confound is largely differenced out. Sinkhole-vs-random
would score much higher and mean much less.

⚠️ **Use the GeoJSON geometry, not the `LONGDD`/`LATDD` attribute columns.**
15 attribute-column points fall outside Florida and 153 disagree with the
geometry by more than 1e-4 degrees; the geometry is clean. Reading the
attribute table silently poisons the sample.

## Results

All rows sampled with `rasterio`'s own `sample()` against the shipped global
COG, so they are reproducible by anyone holding the bundle:

| Variable | AUC(Y,N) | 95% CI | AUC(U,N) |
|---|---|---|---|
| depression_depth_m | **0.649** | [0.58, 0.71] | 0.529 |
| karst_solubility | 0.520 | [0.46, 0.58] | 0.535 |
| depth masked to solubility >= 2 | 0.520 | [0.48, 0.56] | 0.514 |
| local depth (~1.4 km window) † | 0.573 | [0.49, 0.65] | 0.513 |

† Pilot-tile figure. This variable is not a shipped band, so it cannot be
re-derived from the bundle; it is reported for the design decision it settled.

Frequency form, which is easier to act on than an AUC:
**depth > 0 at 51% of verified sinkholes, 24% of confirmed non-sinkholes, 29%
of unverified reports.**

## Three findings that changed the product

**1. The unverified reports are not weak positives.** AUC 0.529 against
confirmed non-sinkholes. Using them would have multiplied the training set 40x
and taught the model a reporting pattern.

**2. Masking depth by solubility destroys the signal** (0.649 -> 0.520, and
the two intervals do not overlap). This
was the layer's original design premise. GLiM marks only ~20% of the labelled
Florida points as soluble because the surficial geology is sand over carbonate.
The band was dropped rather than shipped.

**3. The physically-correct depth is also the more discriminative one.** A
narrower window looked better in an early sweep, but that comparison dropped
points differentially (larger windows hit coastal nodata more often, so n fell
from 103 to 47 and the classes were no longer the same population). On a single
full-tile pass with identical sampling, the true-pour-point depth beats the
~1.4 km local depth (0.573).

## Instrument checks

**A control caught a bug in our own AUC.** `karst_solubility` was included as a
variable expected to score ~0.5. It scored 0.765 while having near-identical
distributions in both classes (Y 80%/20%, N 79%/21%) — impossible. The cause
was a rank AUC using `argsort().argsort()`, which assigns tied values arbitrary
distinct ranks instead of mid-ranks; these bands are mostly zeros. Reproduced:
broken 0.781, correct 0.490 — both on the pilot tile, which is why that 0.490
differs from the 0.520 in the Results table above; the table is the global
raster. All figures here use `sklearn.roc_auc_score`, and an internal figure of
0.686 published before the fix is withdrawn.

**The product was validated, not the probe, and the first attempt at that got
it wrong.** Every number in the Results table is sampled from the COG the
pipeline wrote. The v0.1 draft of this report claimed the same thing but was
measured on the **pilot tile**, whose northern edge is 30 degrees N — it silently
dropped 21 verified and 10 confirmed-not points in north Florida and the
panhandle, and reported n=180 as though it were the whole inventory. A second
error compounded it: points were sampled through a *window* transform, whose
origin does not align to the parent grid, shifting some points a full pixel on a
band where 60% of Florida reads exactly zero. Sampling via `src.sample()` on the
global COG is origin-independent and is what every figure above now uses.

**The extreme values were checked rather than assumed.** The deepest pixel in
the pilot tile (111 m) is in the Viñales mogotes of western Cuba — genuine karst
terrain, confirmed against the source DEM (ground 177 m, pour point 288 m). Its
max/p99 ratio of 13.1x would fail a naive spike gate; it is not a spike.

## What is NOT validated

- Any region outside Florida. This is one hydrogeological setting.
- Any absolute rate. Nothing here converts depth to a probability or frequency.
- Evaporite karst specifically, which behaves differently and faster.
- The solubility band as a predictor, anywhere.

## Planned work

Independent inventories in a second hydrogeological setting are the single
highest-value addition; a commercial-use request to BGS is outstanding. A
second region would convert "indicative" into "validated across settings", and
is the prerequisite for any calibrated probability surface.