Pipeline

Thirteen scripts, in run order

Every step from “which endpoints are alive on my network” to “cross-correlation across fourteen lags”. Each script states what it reads, what it writes and the exact command. Two of them run offline with no account at all, so you can verify the logic before spending a day on downloads.

GEE or command-line downloads? Settled empirically by 03_Data/source_verification_report.csv, measured from the project's own network: both daily ERDDAP chlorophyll endpoints timed out, while OC-CCI monthly, MODIS 8-day, IRI, CMEMS, CDSE and Google Earth Engine all responded. Beyond that, command-line downloading means pulling whole scenes — Sentinel-2 over three AOIs for eleven years is several terabytes — while Earth Engine does the spatial reduction server-side and returns a few thousand rows of numbers. Use GEE for the three site AOIs; use ERDDAP and CMEMS only for the coastal boxes.

Step	Script	What it does	Reads → writes	Datasets
0	`00_verify_sources.py`	Pings every endpoint from your own network and reports which are live, plus the direction each ERDDAP latitude axis runs.	`aoi_config.json` ↓ `03_Data/source_verification_report.csv`
1	`08_build_url_manifest.py`	Offline. Resolves every bbox and date window into fully expanded request URLs. No network needed.	`aoi_config.json` ↓ `03_Data/algae_api/URL_MANIFEST.csv`
2	`07_algae_api_download.py`	Ocean colour and climate via ERDDAP, IRI, CMEMS, OB.DAAC, CDS and CDSE. ERDDAP and IRI need no account.	`aoi_config.json` ↓ `03_Data/algae_api/*.csv`	AL01 AL02 AL08 AL11 AL12 AL17 AL18
3	`06_algae_gee_3sites.py`	Per-pass algae from Earth Engine, seven blocks, computed over water only.	`GEE + shapefiles` ↓ `03_Data/algae/`
3d	`10_algae_daily_3sites.py`	DAILY site-average algae, five sensors forced through one harmonised schema, 2000 to June 2026.	`GEE + shapefiles` ↓ `03_Data/algae_daily/`	AL03 AL04 AL05 AL06 AL07 AL13 AL16 AL19 AL20
3js	`13_algae_gee_codeeditor.js`	GEE CODE EDITOR script. Paste into code.earthengine.google.com. Three sites, five sensors, DAILY and WEEKLY CSV exports to Drive, 2000 to June 2026. Same harmonised schema as the Python.	`GEE assets + your 3 shapefiles` ↓ `Google Drive / CholOut_Algae/`
4	`09_algae_merge.py`	Joins the API and GEE outputs into ISO-week and monthly panels per site.	`03_Data/algae + algae_api` ↓ `03_Data/panels/algae_panel_*.csv`
4d	`11_algae_daily_fill.py`	Cross-sensor bias correction, resolution cascade, gap filling, and a per-year fill-rate report.	`03_Data/algae_daily/` ↓ `03_Data/panels/algae_daily_*.csv`
2b	`12_download_all_free.py`	Downloads EVERY free dataset in the catalogue that has a machine route, exports uniform CSV and writes the manifest this website reads.	`aoi_config.json` ↓ `03_Data/free_exports/`
5	`03_gee_extract.py`	Non-algae covariates: climate, LST, vegetation, flood.	`GEE` ↓ `03_Data/gee_raw/`
5	`04_build_panel.py`	Daily, weekly and monthly panels with lag features across the 14-step grid.	`03_Data/gee_raw/` ↓ `03_Data/panels/`
6	`05_lag_analysis.py`	Cross-correlation across every lag, heatmaps, and the three-site gradient comparison.	`panels + cholera` ↓ `03_Data/results/`
-	`01_harvest_literature.py`	Five-API scholarly harvester with an Unpaywall open-access downloader.	`scholarly APIs` ↓ `02_Downloads/`
-	`02_local_pdf_miner.py`	Triages the PDFs in Paper/ into variables, sensors, lags and results.	`Paper/.pdf` ↓ `02_Downloads/pdf_mined_.csv`

Run it, in order

cd 01_Scripts
python -m pip install requests pandas numpy earthengine-api pyshp openpyxl
earthengine authenticate

# 0 · which endpoints are alive from YOUR network
python 00_verify_sources.py --all

# 1 · offline fallback: fully expanded request URLs, no network needed
python 08_build_url_manifest.py

# 2 · coastal boxes — ERDDAP and IRI need no account at all
python 07_algae_api_download.py --sources erddap iri --sites all

# 2b · everything free that has a machine route, exported as uniform CSV
python 12_download_all_free.py --all

# 3d · DAILY site-average algae, five sensors, one schema, 2000 → June 2026
python 10_algae_daily_3sites.py --project MY_PROJ --sites chakaria --sensors s2 \
       --start 2020-01-01 --end 2020-12-31          # test one year FIRST
python 10_algae_daily_3sites.py --project MY_PROJ --sites all --sensors all \
       --start 2000-01-01 --end 2026-06-30 --to-drive

# 4d · continuous daily series + the fill-rate report
python 11_algae_daily_fill.py --selftest    # offline, sandboxed, proves the logic
python 11_algae_daily_fill.py

# 5 · non-algae covariates and the lag features
python 03_gee_extract.py --project MY_PROJ
python 04_build_panel.py

# 6 · cross-correlation and the three-site gradient test
python 05_lag_analysis.py --cadence weekly

Two safeguards worth knowing about

SELF-TEST MODE

11_algae_daily_fill.py --selftest fabricates plausible multi-sensor data in 03_Data/_selftest/ — a separate sandbox that can never contaminate real data — and runs the full merge on it. If the self-test produces panels, pandas and the merge logic are both fine, and any later failure is a data problem rather than a code problem.

FILL PROVENANCE

Every value in the daily panel carries fill_method: observed, interp or climatology. Model on chl_index, but always report the sensitivity analysis restricted to fill_method == 'observed'. If the algae effect disappears there, it was in the interpolation and not in the water. Weight by n_obs_28d — the count of real observations within ±14 days.

Read before publishing any number. Everything the Earth Engine pipeline produces is an ordinal index, not a calibrated mg/m³. Sen2Cor is land-optimised and returns negative reflectance over dark turbid water. For real concentrations: ACOLITE Dark Spectrum Fitting on Sentinel-2 L1C (AL18), local calibration against in-situ chlorophyll (AL25, AL23), or the Bangladeshi coefficients in Sarker et al. 2019 — which is missing from the corpus.