# Bulk Download Instructions — Petrel Flow Accumulation Layer v0.1

This document covers downloading the full layer bundle. Unlike the coarse Petrel
climatology Layers (a few MB), this is a **278 m global product**: each COG is on
the order of several hundred MB to ~1 GB compressed, and the full bundle is
roughly **1–2 GB**. If you only need a region, a windowed read over HTTPS is the
sensible default — see [`API_USAGE.md`](API_USAGE.md); download the whole bundle
only when you genuinely need the global grid.

## What's in a full bundle (~1–2 GB)

```
flow_distance_to_drainage_278m.tif            ~several hundred MB
flow_twi_278m.tif                             ~several hundred MB
flow_distance_to_drainage_278m_global.png     ~1 MB
README.md                                      ~7 KB
methodology.md                                 ~7 KB
model_card.md                                  ~7 KB
data_sources.md                                ~4 KB
validation_report.md                           ~6 KB
RELEASE_NOTES.md                               ~1 KB
LICENSE.md                                     ~1 KB
CITATION.cff                                   ~1 KB
API_USAGE.md                                   ~5 KB
bulk_download_instructions.md                  ~3 KB
flow_accumulation_v0.1_global_dem_stac_item.json  ~6 KB
```

(Exact raster sizes depend on final compression and are recorded in the STAC
item.)

## Two ways to bulk-download

### 1. via Petrel API (recommended)

`$PETREL_API_KEY` is your Petrel API key (`tlk.…`) from the dashboard —
the free tier works. The bundle endpoint returns a JSON manifest with a
1-hour presigned URL for every file.

```bash
# Fetch the signed-URL manifest:
curl -s -H "X-API-Key: $PETREL_API_KEY" \
  https://api.petreldata.io/v1/layer/flow_accumulation/v0.1/bundle > bundle.json

# Download everything:
mkdir -p flow_accumulation_v0.1
jq -r '.files[] | [.url, .name] | @tsv' bundle.json | \
  while IFS=$'\t' read -r url name; do
    curl -s "$url" -o "flow_accumulation_v0.1/$name"
  done
```

Or with HTTPS curl (works without the AWS CLI):

```bash
URL=$(...)  # signed base URL, as above
for f in flow_distance_to_drainage_278m.tif flow_twi_278m.tif; do
    curl -O ${URL}/${f}
done
curl -O ${URL}/flow_distance_to_drainage_278m_global.png
for doc in README methodology model_card data_sources validation_report \
           RELEASE_NOTES LICENSE API_USAGE bulk_download_instructions; do
    curl -O ${URL}/${doc}.md
done
curl -O ${URL}/CITATION.cff
curl -O ${URL}/flow_accumulation_v0.1_global_dem_stac_item.json
```

### 2. via direct S3 (for internal automation only)

If you have a Petrel staff role with direct bucket access:

```bash
aws s3 sync s3://petrel-data/layers/flow_accumulation/v0.1_global_dem/ ./flow_accumulation_v0.1/
```

This skips the API rate-limit but requires explicit IAM access. Most users
should go through the API.

## Bandwidth and rate limit notes

- The full bundle is ~1–2 GB — a large download. Free-tier accounts can download
  it up to **10 times per day**, but for regional work prefer the windowed COG
  reads in [`API_USAGE.md`](API_USAGE.md), which transfer only the pixels you
  ask for.
- Each bundle download is signed for **1 hour** from the moment the API call
  returns. If a URL expires, request a fresh one with another API call.
- For automated use (CI / production pipelines), cache the bundle locally —
  there is no reason to re-download more than once per vintage release.

## Verifying integrity

Each file has an MD5 checksum recorded in the STAC item under the
`assets.<name>.checksum` field. To verify after download:

```bash
EXPECTED=$(jq -r '.assets.flow_distance_to_drainage.checksum' flow_accumulation_v0.1_global_dem_stac_item.json)
ACTUAL=$(md5sum flow_distance_to_drainage_278m.tif | awk '{print $1}')
[ "$EXPECTED" == "$ACTUAL" ] && echo "OK" || echo "FAIL"
```

(MD5s are populated at S3 push time and recorded in the STAC item.)

## Licensing reminder

Bulk download is restricted to **non-commercial use** under CC-BY-NC-4.0.
Attribution must credit both Petrel and the underlying DEM: *"© DLR/Airbus,
provided under COPERNICUS by the European Union and ESA."* By downloading, you
accept the license terms in [`LICENSE.md`](LICENSE.md).

For commercial bundling, redistribution, or incorporation into paid
products / services / analyses, request a commercial license at **licensing via
https://petreldata.io/demo** before downloading.