Commit f0e7dc49 authored by Andrey Filippov's avatar Andrey Filippov
Browse files

CLAUDE: Add lwir_map scripts (renamed from fopen_nc_summary/fopen_overlay) 



- scripts/lwir_map/lwir_scene_summary.py  (was fopen_nc_summary.py)
- scripts/lwir_map/lwir_map_overlay.py    (was fopen_overlay.py)
- scripts/lwir_map/README.md              (pipeline doc + known limitations)
- Updated catalog.json and scripts/README.md with new names/paths

Scripts read imagej-elphel linked LWIR output and generate per-scene
TSV + GeoJSON/KML/SVG map overlays for satellite atlas navigation.
Not FOPEN-specific; usable for any airplane/drone LWIR acquisition.

Co-authored-by: default avatarClaude <claude@elphel.com>
parent 1579c9b3

scripts/README.md

+50 −1
Original line number Diff line number Diff line
@@ -8,7 +8,7 @@ This file is generated from `scripts/catalog.json`. 
2) Run:



```bash

/home/elphel/git/imagej-elphel/scripts/catalog_to_md.py

/home/elphel/wksp2/imagej-elphel/scripts/catalog_to_md.py

```



---
@@ -75,6 +75,55 @@ Outputs: 


Tags: csv, models, aggregation, plot



## lwir_scene_summary.py



Read imagej-elphel linked LWIR scene output and write a per-scene TSV with GPS corners, AGL, heading, forestation, and ground-plane columns.



Path: `scripts/lwir_map/lwir_scene_summary.py`



See also: `scripts/lwir_map/README.md` for full pipeline documentation.



Example:



```bash

/home/elphel/wksp2/imagej-elphel/scripts/lwir_map/lwir_scene_summary.py /home/elphel/lwir16-proc/NC/linked/linked_1763232117-1763234145-v88 -o /home/elphel/lwir16-proc/NC/summaries/linked_1763232117-1763234145-v88-summary.tsv

```



Inputs:

- Linked view root containing timestamp and *-index directories

- *-elevations_histogram.csv and *-egomotion.csv under each scene version directory

- *-ground_planes.csv under each following *-index version directory



Outputs:

- TSV file with ~65 columns: status, group metadata, scene IDs, timestamps, GPS corners, heading, AGL, forestation, ground-plane data



Tags: lwir, map, atlas, summary, tsv



## lwir_map_overlay.py



Generate KML, GeoJSON, SVG, and bounds TSV map overlays from a LWIR scene summary TSV.



Path: `scripts/lwir_map/lwir_map_overlay.py`



See also: `scripts/lwir_map/README.md` for full pipeline documentation.



Example:



```bash

/home/elphel/wksp2/imagej-elphel/scripts/lwir_map/lwir_map_overlay.py /home/elphel/lwir16-proc/NC/summaries/linked_1763232117-1763234145-v88-summary.tsv --out-prefix /home/elphel/lwir16-proc/NC/summaries/linked_1763232117-1763234145-v88-overlay --svg-fill none --svg-color-by agl_m --svg-label-mode first-per-group --svg-direction-cue broken-right

```



Inputs:

- Summary TSV generated by lwir_scene_summary.py



Outputs:

- Bounds TSV with min/max latitude and longitude

- GeoJSON polygon overlay

- KML polygon and label overlay

- SVG vector page with projected rectangles, configurable labels, fill, color source, and image-right direction cue



Tags: lwir, map, atlas, overlay, kml, geojson, svg



## rag_index.py



Build a local RAG index from attic/CODEX/rag_sources using fastembed + hnswlib.

scripts/catalog.json

+42 −0
Original line number Diff line number Diff line 
{

  "version": 1,

  "docs": [

    {

      "name": "lwir_map/README.md",

      "path": "scripts/lwir_map/README.md",

      "purpose": "Pipeline doc for LWIR scene atlas: purpose, 4-step workflow, script options, known limitations (tilt during turns), and future improvements."

    },

    {

      "name": "mcp-http-howto.md",

      "path": "scripts/mcp-http-howto.md",
@@ -47,6 +52,43 @@ 
      "owner": "codex",

      "created": "2026-02-03"

    },

    {

      "name": "lwir_scene_summary.py",

      "path": "scripts/lwir_map/lwir_scene_summary.py",

      "purpose": "Read imagej-elphel linked LWIR scene output and write a per-scene TSV with GPS corners, AGL, heading, forestation, and ground-plane columns.",

      "inputs": [

        "Linked view root containing timestamp and *-index directories",

        "*-elevations_histogram.csv and *-egomotion.csv under each scene version directory",

        "*-ground_planes.csv under each following *-index version directory"

      ],

      "outputs": [

        "TSV file with ~65 columns: status, group metadata, scene IDs, timestamps, GPS corners, heading, AGL, forestation, ground-plane data"

      ],

      "example": "/home/elphel/wksp2/imagej-elphel/scripts/lwir_map/lwir_scene_summary.py /home/elphel/lwir16-proc/NC/linked/linked_1763232117-1763234145-v88 -o /home/elphel/lwir16-proc/NC/summaries/linked_1763232117-1763234145-v88-summary.tsv",

      "tags": ["lwir", "map", "atlas", "summary", "tsv"],

      "dependencies": [],

      "owner": "codex",

      "created": "2026-05-14"

    },

    {

      "name": "lwir_map_overlay.py",

      "path": "scripts/lwir_map/lwir_map_overlay.py",

      "purpose": "Generate KML, GeoJSON, SVG, and bounds TSV map overlays from a LWIR scene summary TSV.",

      "inputs": [

        "Summary TSV generated by lwir_scene_summary.py"

      ],

      "outputs": [

        "Bounds TSV with min/max latitude and longitude",

        "GeoJSON polygon overlay",

        "KML polygon and label overlay",

        "SVG vector page with projected rectangles, configurable labels, fill, color source, and image-right direction cue"

      ],

      "example": "/home/elphel/wksp2/imagej-elphel/scripts/lwir_map/lwir_map_overlay.py /home/elphel/lwir16-proc/NC/summaries/linked_1763232117-1763234145-v88-summary.tsv --out-prefix /home/elphel/lwir16-proc/NC/summaries/linked_1763232117-1763234145-v88-overlay --svg-fill none --svg-color-by agl_m --svg-label-mode first-per-group --svg-direction-cue broken-right",

      "tags": ["lwir", "map", "atlas", "overlay", "kml", "geojson", "svg"],

      "dependencies": [],

      "owner": "codex",

      "created": "2026-05-14"

    },

    {

      "name": "rag_index.py",

      "path": "scripts/rag_index.py",

scripts/lwir_map/README.md

0 → 100644
+128 −0
Original line number Diff line number Diff line 
# lwir_map — LWIR Scene Map Tools



Utilities for building a navigable satellite-image atlas from imagej-elphel

LWIR scene output.  Useful after any aerial/drone LWIR acquisition to quickly

locate thousands of captured frames on a map without opening individual files.



---



## Purpose



imagej-elphel produces per-scene directories under a linked root such as:



```

/home/elphel/lwir16-proc/NC/linked/linked_1763232117-1763234145-v88/

```



These scripts read that output and produce:



1. A spreadsheet-friendly **TSV summary** — one row per scene, with GPS

   corners, AGL, heading, forestation estimate, ground-plane data, and

   status flags.

2. **Map overlay files** (GeoJSON, KML, SVG, bounds TSV) ready for the

   foliage-internal atlas rendering pipeline.



---



## Pipeline



```

imagej-elphel linked output

        │

        ▼  lwir_scene_summary.py

  summaries/<dataset>-summary.tsv

        │

        ▼  lwir_map_overlay.py

  summaries/<dataset>-overlay.geojson   (+ .kml  .svg  .bounds.tsv)

        │

        ▼  foliage-internal/scripts/download_basemap.py

  atlas/<dataset>/basemap.mbtiles  +  metadata.json

        │

        ▼  foliage-internal/scripts/render_overlay_atlas_pdf.py

  atlas/<dataset>/atlas-4x4.pdf         ← final product

```



Suggested output root: `/home/elphel/lwir16-proc/<site>/`



---



## Scripts



### `lwir_scene_summary.py`



Reads a linked scene root and writes a TSV with ~65 columns per scene.



Key columns: `status`, `scene_id`, `scene_timestamp_s`, `center_lat/long`,

`corner{1-4}_lat/long`, `rect_x_axis_heading_deg`, `agl_m`,

`forestation_percent`, `gp_*` (ground plane), `hfov_deg`, `footprint_x/y_m`.



```bash

scripts/lwir_map/lwir_scene_summary.py \

  /home/elphel/lwir16-proc/NC/linked/linked_1763232117-1763234145-v88 \

  -o /home/elphel/lwir16-proc/NC/summaries/linked_1763232117-1763234145-v88-summary.tsv

```



### `lwir_map_overlay.py`



Reads the TSV and generates map overlay files.



```bash

scripts/lwir_map/lwir_map_overlay.py \

  /home/elphel/lwir16-proc/NC/summaries/linked_1763232117-1763234145-v88-summary.tsv \

  --out-prefix /home/elphel/lwir16-proc/NC/summaries/linked_1763232117-1763234145-v88-overlay \

  --svg-fill none \

  --svg-color-by agl_m \

  --svg-label-mode first-per-group \

  --svg-direction-cue broken-right

```



Key options:



| Option | Values | Default |

|---|---|---|

| `--svg-fill` | `color`, `none` | `color` |

| `--svg-color-by` | `forestation_percent`, `agl_m`, `status` | `forestation_percent` |

| `--svg-label-mode` | `every`, `none`, `first-per-group`, `last-per-group` | `every` |

| `--svg-direction-cue` | `none`, `broken-right`, `open-right` | `broken-right` |

| `--ok-only` | flag | off |



The `broken-right` direction cue omits the middle third of the image-right

edge (corner1→corner2 side) as a flight-direction indicator.



---



## Generated outputs (NC dataset, 2026-05-14)



Summary TSV: 2193 scenes, 169 index groups, 2067 ok / 126 missing.



| File | Description |

|---|---|

| `*-summary.tsv` | Per-scene metadata table |

| `*-overlay.geojson` | GeoJSON polygon footprints |

| `*-overlay.kml` | KML for Google Earth |

| `*-overlay.svg` | Standalone SVG (equirectangular, A0) |

| `*-overlay.bounds.tsv` | Coverage bbox (~6.5 km × 4.0 km, NC site) |

| `*-overlay-agl-nofill-sparse.*` | Sparse variant: AGL color, no fill, first-per-group labels |



---



## Known limitations / future improvements



### Tilt during turns (high priority)

Rectangles are plotted centred on the aircraft GPS position with no attitude

correction.  During turns the aircraft banks, shifting the actual footprint

laterally from the nadir point.  For straight-and-level flight this error is

small; in tight turns it can be significant.



**Fix**: read roll/pitch from egomotion (or IMU) and project the four corners

through the actual camera-to-ground ray using the known AGL and sensor FoV.

`lwir_scene_summary.py` already reads egomotion CSV; roll/pitch columns would

need to be extracted and the footprint geometry updated in `_compute_corners()`.



### Other known gaps

- Forestation fraction uses a fixed elevation threshold; making it

  configurable per-dataset would improve accuracy.

- The SVG equirectangular projection is separate from the Web Mercator

  projection used by the foliage-internal rendering scripts; unifying them

  would simplify the pipeline.

- No support yet for multi-sensor rigs with per-sensor pointing offsets.