Architecture¶

LEAF DSS is a Flask application (Python 3.11) that serves both the interactive map UI and the REST API from a single process. Routes are organised into a blueprints package (13 blueprints), and the codebase is layered into clear service modules for data loading, clustering, feasibility scoring, and AI/RAG.

State that must survive restarts and be shared across worker processes lives in a Supabase Postgres database (the village clusters and their edit-protection flags). Base map geodata and the village master list are shipped as CSV and shapefiles committed to the repo, and per-block indicator values are overlaid live from Google Sheets. The app is deployed on Render behind the custom domain https://leaf-asrlm.in.

This is not a stateless, database-free app

Earlier drafts of this document described LEAF as a "monolithic Flask app with no database". That is out of date. Cluster data is persisted in Supabase Postgres, an in-process scheduler runs nightly maintenance and backups, and an AI/RAG pipeline (Chroma + OpenAI) answers recommendation queries. The sections below describe the current system.

System Overview¶

┌────────────────────────────────────────────────────────────────┐
│                            Clients                             │
│  ┌──────────┐  ┌───────────┐  ┌──────────────┐  ┌───────────┐ │
│  │ Browser  │  │ API Users │  │ Swagger /docs│  │ /update   │ │
│  │ (Map UI) │  │ (Scripts) │  │ MkDocs site  │  │ ops console│ │
│  └────┬─────┘  └─────┬─────┘  └──────┬───────┘  └─────┬─────┘ │
└───────┼──────────────┼───────────────┼────────────────┼───────┘
        │              │               │                │
┌───────▼──────────────▼───────────────▼────────────────▼───────┐
│                     Flask Application (app:app)                │
│                                                                │
│  ┌──────────────────────────────────────────────────────────┐ │
│  │        Blueprints package (13) — routes only             │ │
│  │  pages · blocks · locations · interventions · feasibility│ │
│  │  export · config · levels · villages · clusters ·        │ │
│  │  infrastructure · production_tool · api_info             │ │
│  └───────────────────────────┬──────────────────────────────┘ │
│                              │                                 │
│  ┌───────────────────────────▼──────────────────────────────┐ │
│  │                     Service layer                        │ │
│  │  data_utils   feasibility   clustering   villages        │ │
│  │  google_sheets   rag_utils   db   db_backup   scheduler  │ │
│  └──┬──────────┬──────────┬────────────┬──────────┬─────────┘ │
│     │          │          │            │          │           │
│  ┌──▼───┐  ┌───▼────┐  ┌──▼────────┐  ┌▼────────┐ ┌▼────────┐ │
│  │Shape-│  │  CSV   │  │ Supabase  │  │ Google  │ │ Chroma  │ │
│  │files │  │(villages│ │ Postgres  │  │ Sheets  │ │ + PDFs  │ │
│  │(GeoPd)│ │ + base) │ │(clusters) │  │(overlay)│ │(vectors)│ │
│  └──────┘  └────────┘  └───────────┘  └─────────┘ └─────────┘ │
└────────────────────────────────────────────────────────────────┘
                     │                        │
              ┌──────▼───────┐         ┌──────▼────────┐
              │ In-process   │         │ Supabase      │
              │ scheduler    │  ─────► │ Storage       │
              │ (daemon thd) │ nightly │ db-backups/   │
              └──────────────┘ backup  └───────────────┘

Application Entry & Blueprints¶

app.py is the gunicorn target (app:app). At import it:

Loads .env (so DATABASE_URL is available before any module needs it).
Creates the Flask app with CORS enabled.
Configures Flasgger — Swagger UI at /docs, spec at /apispec.json. A rule_filter exposes only routes under /api* plus /health.
Registers 13 blueprints from the blueprints/ package.
Calls start_scheduler() to launch the background maintenance thread.

The 13 Blueprints¶

Blueprint	File	Responsibility
`pages`	`blueprints/pages.py`	HTML views: `/`, `/clustering`, `/update`, `/about`, MkDocs + PDF serving
`blocks`	`blueprints/blocks.py`	Block GeoJSON and block lookups
`locations`	`blueprints/locations.py`	Hierarchical location data (districts → blocks)
`interventions`	`blueprints/interventions.py`	Intervention definitions and per-intervention config
`feasibility`	`blueprints/feasibility.py`	Weighted feasibility scoring endpoints
`export`	`blueprints/export.py`	CSV / data export
`config`	`blueprints/config.py`	App config, sheet validation, sheet refresh, AI-doc upload
`levels`	`blueprints/levels.py`	Level metadata + URL drill-down routes
`villages`	`blueprints/villages.py`	Village point data + cluster refresh / cadre-protection logic
`clusters`	`blueprints/clusters.py`	Cluster generation, retrieval, CSV edit cycle, coverage
`infrastructure`	`blueprints/infrastructure.py`	POI database (vet centres, pharmacies, input shops) + nearest-to-cluster
`production_tool`	`blueprints/production_tool.py`	Outbound finalised-cluster feed + inbound dashboard exchange
`api_info`	`blueprints/api_info.py`	`/api` index and meta

Blueprints hold routes, services hold logic

Blueprints are thin: they parse requests, call into the service modules (data_utils, feasibility, clustering, villages, google_sheets, rag_utils, db), and shape responses. Business logic and data access live in the service layer, not in the route handlers.

Page (non-API) Routes¶

Route	Renders	Notes
`/`	`index.html`	Main map UI
`/clustering[/<block>]`	`clustering.html`	Cluster planner / editor
`/update`	`update.html`	Ops console (coverage, refresh-all, sheet status, AI-doc upload)
`/about`	`about.html`	About page
`/<district>` · `/<district>/<block>`	app shell	URL drill-down to district and block level
`/documentation/` · `/documentation/<path>`	MkDocs site	Serves the pre-built site from `leaf_flask/site/`
`/docs`	Swagger UI	Interactive API docs
`/health`	JSON	Health probe
`/ai-docs/<path>`	PDF	Serves source policy documents

The 404 handler returns JSON for /api/* paths and otherwise falls back to the app shell so client-side routing works.

Data Storage¶

LEAF has three persistence surfaces, each with a distinct job:

Surface	What it stores	Access module	Mutable at runtime?
Supabase Postgres	Village clusters, cluster members, generation fingerprints, infrastructure POIs, maintenance state	`db.py`	Yes — written by clustering, edits, imports
CSV + shapefiles (repo)	Village master, base map layers, variable metadata, intervention config fallback	`data_utils.py`, `clustering.py`	No — read-only at runtime; changed by rebuilding files
Google Sheets (published CSV)	Per-block indicator values (the map/feasibility overlay) + intervention config	`google_sheets.py`	Yes — edited in Sheets, pulled on a TTL

Supabase Postgres (`db.py`)¶

Connection via psycopg2 with a ThreadedConnectionPool (1–10 connections). DSN comes from DATABASE_URL only (the Supabase direct connection string, IPv6). Raw SQL, no ORM.
is_configured() is simply bool(DATABASE_URL). When it is unset the app still boots — clustering features degrade rather than crash.

Tables (schema.sql):

Table	Key	Purpose
`clusters`	`cluster_id` (TEXT)	One row per cluster: commodity, block/district, member count, span, centroid, cadre assignments, `cluster_name`, and protection flags `finalized` / `locked` / `provisional` / `dashboard` (JSONB)
`cluster_villages`	row per village	Villages belonging to a cluster; FK → `clusters` `ON DELETE CASCADE`
`cluster_generation`	`block_name` + `commodity`	Stored `fingerprint` used to decide whether a scope needs regeneration
`infrastructure`	POI id	Vet centres, pharmacies, input shops
`maintenance_run`	job name	`last_run` + `last_summary` (JSONB) for scheduler bookkeeping

Edit protection is columns, not an origin field

There is no origin column. A cluster's provenance and its protection from automated regeneration are expressed by the booleans locked and finalized, the provisional flag, and whether dashboard IS NOT NULL. Any sweep that could overwrite human/cadre edits must check these. See Cadre-Edit Protection.

Migrations¶

Schema changes live in migrations/ and are applied manually and idempotently with python run_migration.py migrations/<NNN>.sql — they do not run automatically on deploy.

#	Migration	Effect
001	`smart_refresh`	Adds `clusters.locked`, adds `cluster_generation`
002	`provisional`	Adds `provisional` flag
003	`district_coordinator`	Adds cadre column
004	`cluster_name`	Adds editable `cluster_name`
005	`split_lakhipur`	`LAKHIPUR` → `LAKHIPUR (CACHAR)` / `LAKHIPUR (GOALPARA)`
006	`split_binnakandi`	`BINNAKANDI` → `(HOJAI)` / `(CACHAR)`

Migrations 005/006 resolve block-name collisions and preserve locked cadre data across the split.

Clustering Algorithm (`clustering.py`)¶

The planner groups villages into workable, per-commodity clusters.

Greedy seed-and-grow per (block, commodity). cluster_id = f"{block}-{commodity}-{uuid8}"; a cluster's district is the mode of its villages' districts.
ALGO_VERSION = 5. Bumping it invalidates fingerprints and forces all unlocked scopes to regenerate.
6 commodities: Dairy, Goatery, Piggery, Backyard_Poultry, Duckery, Fishery_Activity.
DEFAULT_PARAMS: min_members_per_village=6, min_cluster_members=30, max_cluster_members=150, max_radius_km=5.0, emit_provisional=True, provisional_min_members=1, rebalance=False. Haversine distance between village points.
regenerate_clusters() deletes the scope, inserts the new clusters + villages, and writes the new fingerprint.

Cadre-Edit Protection¶

This is the single most important invariant in the system: automated refreshes must never wipe human/cadre edits. Logic lives in blueprints/villages.py.

Smart lazy refresh (get_or_regenerate, per (block, commodity)): skips entirely if scope_is_locked() (any cluster in the scope is finalized or locked); otherwise regenerates only when the stored fingerprint differs from the current scope_fingerprint() (SHA-256 of ALGO_VERSION + params + the village blob).
Whole-state sweep (refresh_all_coverage): iterates every block × 6 commodities. A scope is skipped/protected if finalized OR locked OR dashboard IS NOT NULL. Fresh scopes (fingerprint match) are skipped; an empty-but-eligible heal guard fills genuine gaps.
Flag setters: CSV import sets locked=True on imported clusters (commodity-safe — it only deletes commodities present in the uploaded file); finalize sets finalized=True; a dashboard POST stores the JSON and sets locked=True.
Rename reconciliation: clusters under a block name that no longer appears in the village master are surfaced by unmapped_blocks(); cadre clusters (finalized / locked / dashboard) are migrated, not dropped.

There is no authentication

The only gate on destructive/admin actions is _is_admin_request — satisfied by ?admin=1 in the query string or an X-Admin: 1 header. There is no login, no session, no role check anywhere in the app. Do not expose destructive endpoints (refresh-all, import, upload) on a public URL without adding real auth in front of them.

Google Sheets Overlay (`google_sheets.py`)¶

Per-block indicator values shown on the map are read live from Google Sheets, not from a static CSV.

Three published-CSV sheets (SHEET_URLS):
- dss_input — intervention configuration.
- block_values — per-block variable values (the map / feasibility overlay).
- user_update — a friendly end-user editing sheet (LEAF-59); its URL is currently empty, so the overlay is a no-op until set.
get_sheet(key) uses a 5-minute TTL cache, falls back to the last good cache if a fetch fails, and finally falls back to the committed local CSV (DSS_input2.csv / block_values.csv). refresh() forces a re-pull.
get_block_values_overlaid() loads coded block_values and overlays the friendly user_update columns by Block_name (the user sheet wins); it is a no-op while user_update is unset.
validate_sheets() checks required ID columns, duplicate BLOCK_IDs, stray non-numeric cells, range_min > range_max, cross-sheet variable existence, and conflicting convergence-card tags. It is surfaced at GET /api/config/validate and shown on the /update console.

Village Base Data (`data/villages.csv`)¶

data/villages.csv (~21,495 rows) is the cluster planner's source of truth.

Header: district_name, block_name, gp_name, vill_name, lat, long, the 6 commodity counts, plus other-activity columns.
Built by scripts/build_village_master.py from the raw SHG workbook source_data/SHG_Assam_Consolidated_final_Jun22.xlsx (sheet Village Location Detail). District/block names are taken verbatim from the survey (uppercased, whitespace-collapsed) — they are not rewritten against the shapefile. The only rows dropped are those with missing or out-of-range coordinates (Assam bounding box). Run with --dry-run to preview.
The source workbook contains PII and is gitignored; only the derived CSV is committed.

Missing coordinates → missing villages

Villages whose survey record has Location Status = MISSING have no lat/long and are dropped by the coordinate filter, so they cannot be clustered or mapped (seen with Udalguri). They must be given coordinates at source before they can appear.

AI / RAG (`rag_utils.py`)¶

Context-aware recommendations use retrieval-augmented generation.

LangChain + OpenAI. Requires OPENAI_API_KEY (raises if unset). Chat model gpt-4o-mini; embeddings via OpenAIEmbeddings.
A Chroma vector store persisted at data/vectorstore/ is built from the *.pdf policy documents in ai-docs/ (chunk size 1000, overlap 200).
generate_recommendation() retrieves the top k=5 chunks, combines them with the selected block's metrics, and prompts the LLM.
After a new AI document is uploaded via POST /api/config/upload-ai-doc?admin=1, reset_vectorstore() rebuilds the index.

Background Jobs — Scheduler & Backup (`scheduler.py`)¶

start_scheduler() runs at app import as an in-process daemon thread — there is no separate Render worker or cron service. It is a no-op when DATABASE_URL is unset.

The loop waits 120 s after startup, then checks hourly; each job runs at most once per 24 h.

Job	Function	What it does	Guard
Coverage sweep	`villages.refresh_all_coverage()`	Rebuilds clusters for eligible scopes across the whole state (respecting cadre protection)	Postgres advisory lock + `maintenance_run` interval
Nightly backup	`db_backup.backup_to_storage()`	Dumps `clusters`, `cluster_villages`, `cluster_generation` to Supabase Storage bucket `db-backups` as `<date>/<table>.csv.gz`; retains 14 days	No-op if Supabase creds absent

Two Supabase access paths

The app talks to Postgres over the IPv6 direct connection (DATABASE_URL). The backup job instead uses Supabase REST/Storage over IPv4 (SUPABASE_URL + SUPABASE_SECRET_KEY). Both must be configured for full functionality. The advisory lock ensures the sweep runs once even though gunicorn serves 2 workers. A manual POST /api/clusters/refresh-all?admin=1 bypasses the interval.

Frontend Architecture¶

The frontend is a server-rendered Jinja shell plus vanilla JavaScript modules in static/js/. There is no build step and no framework.

Concern	Technology	Notes
Map	Leaflet.js	Choropleth of blocks + cluster overlays
Charts	Chart.js	Feasibility distributions, metrics
Modules	plain ES scripts	`app.core.js` (bootstrap/state) plus feature files: `app.map.js`, `app.blocks.js`, `app.feasibility.js`, `app.clusterview.js`, `app.airec.js`, `app.district.js`, `clusters.js`, …

Coverage & Counts¶

Fact	Value
Districts (in `villages.csv`)	35 distinct districts, all Assam
Blocks	220 distinct blocks
Commodities	6 (Dairy, Goatery, Piggery, Backyard_Poultry, Duckery, Fishery_Activity)
Live URL	https://leaf-asrlm.in
Algorithm version	`ALGO_VERSION = 5`

Stale 219 constant

A hard-coded block count of 219 still appears in clusters.py and villages.py. It is off by one after the Lakhipur/Binnakandi splits — the correct count is 220. Treat any 219 you see in code or older docs as a bug, not the source of truth.

Deployment (summary)¶

render.yaml: service iwmi-leaf, python runtime, standard plan (no spin-down). buildCommand: pip install -r requirements.txt; startCommand: gunicorn app:app --bind 0.0.0.0:$PORT --workers 2 --timeout 120 --preload. The only env var in the YAML is PYTHON_VERSION=3.11.0; all secrets are set in the Render dashboard (DATABASE_URL, SUPABASE_URL, SUPABASE_SECRET_KEY, SUPABASE_PUBLISHABLE_KEY, OPENAI_API_KEY, optional SECRET_KEY).
Auto-deploys from GitHub main. The custom domain leaf-asrlm.in is configured in the Render dashboard / DNS, not in code.
A Dockerfile (python:3.11-slim + GDAL/GEOS/PROJ) exists as an alternative, but Render uses the python-runtime path, not Docker.

See the Hosting & Handover Guide and Deployment pages for full step-by-step instructions.