# Project Details (Draft)

## Overview
This document is a structured, living description of the data layout and processing pipeline for the current active parts of the `imagej-elphel` project. It consolidates earlier notes and the latest additions into a coherent reference.

Currently, active development is driven by a USAF contract for CUAS (Counter-Unmanned Aircraft Systems) detection, and a subcontract with the University of Utah for FOPEN (Foliage Penetration). The CUAS application uses the LWIR-16 camera mounted on a gimbal, utilizing rotational motion for FPN (Fixed-Pattern Noise) mitigation.

## Autonomous Agent Workflow (MCP & Daemon Integration)
The processing pipeline takes significant time (often hours) for large sequences. To optimize this, the program is wrapped with an MCP (Model Context Protocol) interface, an MCP server, and a lifecycle daemon.

This architecture enables autonomous, overnight operation by AI agents (Codex, Claude, Gemini). The expected agent lifecycle loop is:
1. Execute the program via MCP.
2. Configure processing parameters (e.g., adjusting FPN thresholds, tuning LMA weights).
3. Launch the processing job.
4. Wait for completion.
5. Analyze the resulting CSVs, Histograms, or TIFF profiles.
6. Re-run with refined parameters or directly edit the Java source code to fix issues.

This workflow has been successfully demonstrated for the FOPEN datasets and is currently being adapted to handle the CUAS parameter optimization sweeps to maximize the number of valid targets detected.

## Data Layout and List Files
The program starts with a list file that defines the root and source directories plus scene sequences. Example list entry:

```
1763232231_345331   1, -2, -1, 0, 0  # NC32-LY 13:43:51 #nc34a
```

The first token is a scene-sequence timestamp (directory name). The next two numbers define the scene index range to process. For example, with 499 subfolders (0..498), `1, -2` means process scenes 1..497.

Each subdirectory in the sequence is a "scene" (a set of 16 simultaneously captured LWIR images). Sensors are arranged on a 220 mm diameter circle, evenly spaced at 360/16 degrees. Sensor 0 is at the top; 1..15 are clockwise when viewed from behind the camera (same direction as the camera looks). Sensors 16..19 are RGB cameras and are not used in the current pipeline.

The raw LWIR data under:

```
/media/elphel/btrfs-data/lwir16-proc/NC/scenes/
```

is copied once into model directories (as `jp4` subfolders). Example:

```
/media/elphel/btrfs-data/lwir16-proc/NC/scenes/1763232231_345331/1763232231_362003
```

is copied to:

```
/media/elphel/btrfs-data/lwir16-proc/NC/models/1763232117-1763234145/1763232231_362003/jp4
```

IMU data for the scene is stored as:

```
/media/elphel/btrfs-data/lwir16-proc/NC/models/1763232117-1763234145/1763232231_362003/1763232231_362003-ims.corr-xml
```

The IMU source path currently comes from `CORRECTION_PARAMETERS.getImsSourceDirectory()` (edited in "Setup CLT Batch parameters") and is not yet exposed in `*.list` files.

Scene sequences are captured at ~60 Hz for ~8 seconds, followed by ~3 seconds pause due to bandwidth constraints.

## Model Directories, Versions, and Linked Views
The parent directory for processed scenes is large (the latest set contains ~85K model directories). Most contain only `jp4` and IMS data. Some contain additional results and are considered special:

- Reference scenes with computed data for their segment.
- Index scenes (latest timestamp per sequence) that include `*-ground_planes.csv`. The first column contains timestamps of reference scenes in the segment.

Processing results are versioned. The version directory is defined by `EyesisCorrectionParameters.x3dModelVersion` (configured in “Setup CLT Batch parameters”). A single model directory may contain multiple versions.

For manual browsing, the program creates a directory with symlinks to special model directories. Example:

```
/media/elphel/btrfs-data/lwir16-proc/NC/models/1763232117-1763234145
```

### FOPEN NC Summary and Map-Overlay Workflow
The current North Carolina FOPEN linked view is:

```
/home/elphel/lwir16-proc/NC/linked/linked_1763232117-1763234145-v88/
```

Entries are timestamp-named directories using `_` instead of the decimal point in seconds. The `*-index` entries are index scenes; ordinary timestamp entries before an index entry, and after the previous index entry or start of the directory, belong to that index group. The index scene version directory contains `*-ground_planes.csv`, whose first column lists the reference-scene timestamps in the group.

The spreadsheet summary output should be written under:

```
/home/elphel/lwir16-proc/NC/summaries/
```

For each ordinary timestamp directory, the useful inputs are:

- `<timestamp>-elevations_histogram.csv`: first data row, first column is AGL. The cumulative elevation histogram can be converted to a forestation estimate as the relative area above a small elevation threshold, currently 1.0 m by default.
- `<timestamp>-egomotion.csv`: after the header, each row is one 60 Hz scene. Columns 1-2 are scene index and timestamp; columns 3-8 are X/Y/Z and Euler angles relative to the reference frame. The reference-frame row has exact zeroes in all six relative pose columns. Its `lat` and `long` columns are the rectangle center for map overlays.

For map overlays, the reliable first-pass orientation should come from the local `lat`/`long` motion around the reference row, not from the absolute IMS orientation or the full first-to-last segment chord. Use scene indices `ref-1` to `ref+1` when available, with adjacent egomotion rows as fallback. Assume low wind and align the image X axis opposite the local flight direction, because new objects appear at the left margin of the images. The LWIR-16 camera HFoV is 32 degrees and VFoV is `512/640 * HFoV`; combined with AGL, this defines the footprint rectangle dimensions. Labels can use the four least significant digits of timestamp seconds plus one decimal digit.

Overlay generation should keep georeferenced and printable outputs separate: KML/GeoJSON for map/GIS import, and SVG/PDF for large-format vector inspection. The first NC v88 overlay bounds from all usable rectangle corners are approximately 34.8196196..34.8557247 latitude and -79.2299851..-79.1587591 longitude, about 6.5 km by 4.0 km at the mid-latitude.

## Core Processing Pipeline (Segment-Based)
Scene sequences are processed separately in a loop such as:

```
for (int nseq = 0; nseq < num_seq; nseq++) {
    long start_time_seq = System.nanoTime();
    System.out.println("\nSTARTED PROCESSING SCENE SEQUENCE "+nseq+" (last is "+(num_seq-1)+")\n");
    ...
}
```

Segments are defined around a reference scene, with overlap and matching constraints. Scenes are processed in reverse order (latest first) for historical reasons.

### Segment Bounds
The program finds how many preceding scenes can be included in the segment, based on overlap and the number of good matching tiles. Scenes can be "partially valid" if overlap is sufficient for translation but not for zoom or rotation fitting.

In the current application (Cessna, 25–150 m AGL, 30–40 m/s), the reference scene is near the center of the segment. Earliest and latest scenes may not overlap with each other but must overlap with the reference. The segment is continuous with no gaps. The earliest scene is determined by overlap with the reference scene (`EstimateSceneRange.getPrescanRange()`).

### Initial Depth Map (Intrascene)
Depth is computed per tile (80x64 tiles corresponding to 640x512 images) in `OpticalFlow.buildRefDSI()`. This uses intrascene correlation (parallax across sensors) and GPU/CUDA kernels under:

```
/home/elphel/git/imagej-elphel/src/main/resources/kernels
```

Java code uses JCuda (`/home/elphel/git/imagej-elphel/src/main/java/com/elphel/imagej/gpu`). Kernels are debugged via Nvidia NSight using a C++ testbench. The external CUDA repo is:

```
/home/elphel/git/tile_processor_gpu/
https://git.elphel.com/Elphel/tile_processor_gpu/tree/lwir16_2
```

Because baselines are narrow, keypoints are not used. Instead, the pipeline uses 2D phase correlation between 16x16 tiles. There are 120 sensor pairs. Pairwise results are combined by:

- Baseline-rotated/scaled correlations.
- LMA fit to all 120 pairs (more accurate, more expensive).
- DNN approach (from the arXiv paper, not yet implemented for the current camera).

Argmax accuracy is improved by an iterative approach. After each disparity estimate, a fractional shift is applied in the frequency domain and correlation is repeated. Residual disparity converges to zero without assuming correlation shape. The phase-correlation step uses a "fat zero" (small constant in the denominator) to balance noise sensitivity.

The initial disparity map is saved as:

```
.../v89/<timestamp>-DSI_MAIN.tiff
```

### Pose Estimation (Interscene)
The pipeline estimates per-scene pose relative to the reference scene. Positions are stored in `*-egomotion.csv` columns 3–5 and orientation in columns 6–8. IMS provides hints. Matching is done with interscene correlation on per-tile motion vectors.

Workflow:
- Build initial disparity map.
- Match consecutive scenes to estimate poses.
- Refine disparity map with interscene accumulation.
- Repeat with more LMA parameters enabled.

Outputs include 3D scene models, DNN training data, and auxiliary files. State is persisted in:

- `*-DSI_MAIN.tiff`
- `*-INTER-INTRA-LMA.tiff` (refined disparity, overwritten per iteration)
- `*-INTERFRAME.corr-xml` (poses, overwritten per iteration)

Iteration counters:

- `EYESIS_DCT_AUX.num_orient`
- `EYESIS_DCT_AUX.num_accum`

Used in loops like:

```
while ((master_CLT.getNumOrient() < min_num_orient) || (master_CLT.getNumAccum() < min_num_interscene))
```

### ERS and Motion Blur
Electronic Rolling Shutter (ERS) is handled during pose fitting. Each step computes per-tile motion vectors, then uses LMA to fit pose and optional velocities. Since motion blur can exceed 5 px, images are pre-aligned using IMS predictions. When IMS is unavailable, a spiral search is used. Matching starts from scenes closest to the reference, then expands outward.

#### GPU-side motion-blur handling and TD correlation flow
Current interscene matching for a segment uses a GPU-centric transform-domain (TD) pipeline:

1) The reference scene (all 16 LWIR images) is transferred to GPU memory and converted to TD once.
2) Factory-calibration aberration correction is applied in TD by convolution with the calibration kernels.
3) Motion-blur compensation is applied in TD workflow for both reference and matched scenes:
   - microbolometer blur is modeled as a first-order LPF in time;
   - correction subtracts a scaled copy of the same image shifted by motion vectors (tile-granular approximation);
   - task definitions are packed in Java (`TpTask`) and passed as float task arrays to GPU kernels.

For each matched scene, the 16 images are TD-transformed and geometrically aligned to the reference (tile by tile), using integer tile placement plus fractional subpixel shift through TD phase rotation. Pairwise products with cached reference TD are computed per sensor pair, then all 16 channels are accumulated in TD before phase-only normalization. A small "fat zero" term in normalization controls SNR vs peak sharpness tradeoff. After inverse transform, the peak argmax gives tile motion vectors (`mvX`, `mvY`), and peak magnitude is used as sample strength/weight for subsequent LMA.

This "accumulate in TD before normalization" step is important for low-contrast LWIR data because it improves robustness relative to per-channel independent normalization.

### FPN Mitigation
Uncooled LWIR sensors have fixed-pattern noise (FPN). If interscene pixel shifts are too small, correlation may lock to FPN. Scenes with expected shifts below thresholds (`IntersceneMatchParameters.fpn_min_offset`, `min_offset`) are skipped and added to `fpn_list`. Later, each skipped scene is matched to a farther scene using `Interscene.getFPNPairs`, ensuring pixel shifts exceed tile size and avoid FPN dominance.

## Differential Image Rectification (TP/CLT)
TPNET uses "differential image rectification" (US10638109B2). Instead of full rectification to a pinhole model, the pipeline computes a virtual average camera and rectifies only the small differential distortions of each sensor relative to that average. Local shifts are typically <2 pixels.

CLT (Complex Lapped Transform) is used instead of FFT. CLT has perfect reconstruction and allows lossless fractional pixel shifts via phase rotations. This enables sub-pixel disparities (~0.05–0.01 px) without upsampling or resampling. Multiple operations (aberration correction and phase correlation) are combined in the transform domain for GPU efficiency.

## Down-View Aerial Applications (SfM / SLAM)
Downward-looking aerial views reduce depth variation, which makes separating translation from rotation harder. Lateral motion and azimuth rotation both appear as X shifts; vertical motion and tilt appear as Y shifts. Traditional intrascene disparity becomes insufficient for high altitudes, so the workflow was adjusted.

### Multicopter (Mine Detection) workflow
1) Initial intrascene depth map
2) Initial segment pose estimation
3) Depth refinement via interscene accumulation
4) Pose refinement with more LMA parameters
5) SfM refinement, repeated multiple times
6) Calibration (IMS vs camera frame)
7) Outputs

### Foliage Penetration: SLAM approach
At high altitudes, pure intrascene depth fails for long baselines. The pipeline uses SfM simultaneously with pose estimation in `EstimateSceneRange.scanSfmIMS()`. Disparity is refined as scenes get farther from the reference, with weights proportional to baseline distance. Short baselines skip SfM (`min_sfm_meters`). Long baselines may filter low-confidence tiles (`sfma_filt_meters`). Background tiles are preferred when FG/BG confusion occurs.

### Current Limitations
- Disparity offset compensation is still incomplete. In navigation, far objects allowed treating minimal disparity as infinity; for down-view this is not possible. Current correction attempts use:
  - `OpticalFlow.getImgImsScale()`
  - `OpticalFlow.getImsDisparityCorrection()`
  - `QuadCLT.offsetDSI()`
  - `QuadCLT.setDispInfinityRef()`
  - `OpticalFlow.scaleImgXYZ()`

This fails in some low-altitude forest cases. The correction should be per-sequence, not per-segment, for better accuracy.

- IMS/camera orientation offset (`Interscene.getQuaternionCorrection()`) is disabled and should move to per-sequence. `QuaternionLma` should be restored to use linear motion plus rotations. We should weight angular samples (closer to reference are more accurate) and ensure time consistency. Angular data is in `*-egomotion.csv` columns 6–8.

## CUAS vs Moving-Camera Modes (Operational Difference Notes)
These notes describe the practical difference between:
- the moving-camera aerial pipeline (foliage/restoration), and
- the CUAS pipeline with gimbal rotation for target detection.

### Moving-camera aerial mode (current foliage/global-LMA development)
- Camera platform is moving.
- A scene sequence (typically ~500 scenes) is split into overlapping segments.
- Each segment is built around a captured center/reference scene.
- Scenes are transformed to be viewed from that segment reference pose.
- Matching/optimization is segment-based because far scenes may not overlap directly.

### CUAS mode (gimbal-rotation, mostly parked after 2025-10-27)
- Camera is fixed in ground frame while gimbal rotates view direction around a fixed axis.
- A virtual reference scene is used (`center_CLT = QuadCLT.restoreCenterClt(...)`) rather than a captured reference frame.
- Typical scan: about one full revolution per ~3 seconds (~0.33 Hz), with a rotation radius around ~3 angular degrees.
- Resulting inter-frame shift remains small (roughly <10% of image size), so all scenes in a long sequence (~500) can overlap a single virtual center.
- Full-sequence processing is therefore feasible without segment splitting logic used in moving-camera mode.

### Why CUAS uses rotation and deep disparity integration
The primary target class for CUAS detection is extremely difficult: targets are simultaneously low-contrast, physically sub-pixel (e.g., <0.2 pixels linear, appearing slightly larger only due to lens aberrations), and exhibit very slow apparent motion (e.g., a drone moving directly toward or away from the camera).

In this regime, the uncooled microbolometer's Fixed-Pattern Noise (FPN) becomes the dominant limiting factor. Because FPN slowly fluctuates over time, it cannot be cleanly removed by subtracting a single static "dark frame". If the camera were completely stationary, the sub-pixel target signal would be completely drowned out by this slowly drifting FPN.

By rotating the camera at ~1/3 rps on the gimbal, we force the target and the physical background to move rapidly across the sensor array in image coordinates, while the FPN remains geographically tied to the sensor pixels. This deliberate mechanical decoupling vastly improves the mathematical separability of the target from the noise floor.

Because the target footprint is incredibly small, acquiring accurate disparity (ranging) requires massive data integration. The pipeline employs a two-step process:
1. **Sensor Consolidation:** Combine data from the 16 separate LWIR cameras (each 640x512, 32° HFoV, distributed on a 220 mm diameter circle).
2. **Virtual Long-Exposure Tracking:** Implement a virtual "tracking" camera that follows the target using piece-linear motion models.

By accumulating hundreds of these 16-image scenes over several seconds, the pipeline pushes the disparity resolution deeper than a single frame allows. In favorable conditions (like the recent Eagle Mountain test), this enables sub-pixel disparity resolutions below **0.02 pixels**. Because of the 220 mm baseline across the 16 cameras, a 0.02 pixel disparity error translates to roughly a 10% ranging error at 1000 meters.

Current efforts are focused on achieving similar detection and ranging performance on recent data from Latvia (Selonia NATO range) to facilitate faster target detection. Hardware anomalies during acquisition (including a laptop failure linked to ground loops/CAT5 connections in heavy EW jamming environments) resulted in noisy sequences, but valid drone (DJI 4 Mini Pro) and bird tracks are present and being actively processed.

### FPN behavior and mitigation in CUAS
- FPN is not perfectly static over long periods.
- Boson-class sensors use periodic shutter-based correction (multiple dark frames every configured interval), which reduces but does not eliminate FPN.
- CUAS background extraction uses rotation-compensated averaging and subtraction of the averaged background.
- Two FPN components are targeted:
  - persistent sensor/lens-embedded patterns (defects, dirt, etc.),
  - row/column-correlated noise (sensor-architecture correlated stripes).
- Additional row/column normalization (1D row/column averages) is used before/with subtraction.

### Code-state context
- CUAS-focused work was parked around commit `8946631f767ddda77365ebc32d45eef1e3d21936` (October 27, 2025).
- Since then, development emphasis shifted to foliage/global-LMA paths; CUAS branches may require restoration and re-validation for bit-rot before new-data testing.

### Current CUAS target-detection flow (March 2026 LV data)
For the March 2026 LV runs, the current working input list is:

```
/home/elphel/lwir16-proc/LV/lists/lv_site_05.list
```

Per-sequence outputs are created under linked center directories such as:

```
/home/elphel/lwir16-proc/LV/linked/centeres_1773389747-1773390382/1773389818_152542-CENTER/v07
```

The first restored CUAS stage currently produces merged LWIR stacks after FPN and row/column mitigation. A typical file is:

```
1773389818_152542-CENTER-CUAS-MERGED-CUAS.tiff
```

In this stack:
- the first slice is the sequence average,
- the remaining slices are individual scenes with that average subtracted.

The corresponding unsharp-masked version, for example:

```
1773389818_152542-CENTER-CUAS-MERGED-CUAS-UM2.0_1.000_250.tiff
```

is the same merged sequence after unsharp masking (for the example above: sigma 2.0 px, amount 1.0).

The next CUAS stage is the moving-target stage:
- estimate a prevailing per-tile motion vector using 2D phase correlation on the merged/unsharp sequence,
- use those vectors as a "virtual moving camera",
- shift each contributing frame according to that motion vector,
- accumulate the shifted frames as a long exposure to improve SNR of dim moving targets,
- then locate/freeze targets on the accumulated data and render target overlays/video products.

This stage is where the sky mask matters operationally: terrain-rich areas are suppressed before accepting local maxima, so target search is effectively constrained to the allowed sky region.

#### CUAS motion-scan details: keyframes, pair geometry, and FAST/SLOW split
The per-tile motion vectors are not estimated from a single frame pair. For each keyframe index, the code builds a short temporal block and accumulates multiple pairwise correlations that all correspond to nearly the same constant-velocity target motion.

The main scan geometry is controlled by:

- `cuas_corr_offset`: separation between the two halves of each pair set
- `cuas_corr_pairs`: number of correlation pairs accumulated inside one keyframe block
- `cuas_half_step`: if `true`, adjacent keyframes advance by `cuas_corr_offset/2`; otherwise by `cuas_corr_offset`
- `cuas_precorr_ra` and `cuas_corr_step`: optional temporal smoothing/decimation before correlation

For one keyframe:

- `seq_length = cuas_corr_offset + cuas_corr_pairs`
- `corr_inc = cuas_half_step ? (cuas_corr_offset / 2) : cuas_corr_offset`
- for keyframe `n`, `frame0 = start_frame + n * corr_inc`
- `frame1 = frame0 + cuas_corr_offset`

Inside that keyframe, the code correlates multiple temporally aligned pairs:

- `(frame0 + dframe)` against `(frame1 + dframe)`
- where `dframe` runs from `corr_ra_step/2` to `< cuas_corr_pairs`, stepping by `corr_ra_step`

So with the common mental model of `offset = 8` and `pairs = 8`, the effective pairs are approximately:

- `0 vs 8`
- `1 vs 9`
- ...
- `7 vs 15`

or the decimated equivalent if `cuas_corr_step > 1`.

Each pair is correlated tile-by-tile in transform domain (`TDCorrTile`). The pair results are accumulated before conversion back to pixel-domain correlation. If `cuas_smooth` is enabled, the individual pair contributions are weighted by a sine window across the temporal block. After accumulation:

- TD correlations are normalized and converted to pixel-domain using `cuas_fat_zero`
- per-tile local maxima are extracted
- centroid recentering (`cuas_cent_radius`, `cuas_n_recenter`) estimates `vx`, `vy`, peak strength, and in-center fraction

The `*-CORR2D.tiff` files visualize these per-tile 15x15 correlation maps, one keyframe per slice.

FAST and SLOW currently share this same motion-scan geometry. The difference is only in the temporal prefiltering of the input sequence:

- FAST: `temporalUnsharpMask()` using `cuas_temporal_um`
- SLOW: `runningGaussian()` using `cuas_slow_ra`

Both then call the same `prepareMotionBasedSequence()` path and therefore use the same `cuas_corr_offset`, `cuas_corr_pairs`, `cuas_half_step`, `cuas_precorr_ra`, and `cuas_corr_step`.

Practical consequence: if targets are too fast and the peaks in `*-FAST-CORR2D.tiff` hit the `+-7` tile borders, reducing `cuas_corr_offset` will reduce the apparent motion in both FAST and SLOW modes. There is currently no separate fast-only correlation offset parameter; adding one would require a code change.

Velocity units in the current implementation are intentionally split:

- `RSLT_VX` / `RSLT_VY` store the raw displacement measured by the correlation peak over one pair separation, i.e. over `cuas_corr_offset` scenes.
- To convert these stored values to pixels per frame, multiply by `1.0 / cuas_corr_offset`.
- To convert them to displacement between neighboring keyframe centers, multiply by `getCorrIncExact() / cuas_corr_offset`.

Odd `cuas_corr_offset` with `cuas_half_step=true` needs special care:

- the scan index increment is still integer (`getCorrInc()` truncates `cuas_corr_offset / 2`),
- but the intended keyframe-center geometry remains half the full offset (`getCorrIncExact()` uses `0.5 * cuas_corr_offset`),
- so midpoint / before-after chaining logic must use the exact geometric half-step, not the truncated scan increment.

This distinction is what caused the temporary `1.25` compensation when `cuas_corr_offset=5`: the scan step became `2`, but the intended half-step geometry was still `2.5`.

### Where the current code does this
- `CuasRanging.processMovingTargetsMulti()` renders the per-sequence `...-CUAS...` stack and applies the unsharp mask before creating `fpixels`.
- `CuasMotion.processMovingTargetsMulti()` runs the fast/slow motion-target preparation and resolves non-conflicting motion candidates.
- `CuasMotion.generateExtractFilterMovingTargets()` expands the target field, performs the motion-compensated accumulation, and renders target/background outputs.
- `CuasMotion.shiftAndRenderAccumulate()` is the "virtual moving camera / long exposure" step.
- `CuasMotion.getAccumulatedCoordinates()` searches the accumulated tiles for isolated local maxima, applies the sky mask, and refines target coordinates with the CUAS LMA fit.

### Top-level trigger path
The March 2026 CUAS workflow order is:

1. Normal per-sequence CUAS scene processing (`cuas_proc_mode = 0`) produces the merged/unsharp stacks, motion vectors, accumulated target frames, and per-sequence CUAS outputs.
2. `CUAS Combine` runs later on the already generated linked centers (`CuasMultiSeries.processGlobals()`).
3. `CUAS Video` runs after that to combine/package already produced CUAS video outputs.

So the motion-vector and long-exposure stage belongs to the first, per-sequence CUAS processing pass, while `CUAS Combine` and `CUAS Video` are later-stage steps.

The direct gate for producing the accumulated target outputs is:

```
IntersceneMatchParameters.cuas_generate
```

shown in the UI as:

```
Generate and save detected targets
```

If that checkbox is off, the code will still do earlier target/ranging work but will skip the accumulated target TIFF/video generation stage.

### Operational note: sequence continuity is required for CUAS target detection
There is now an explicit distinction between:

- the early per-sequence preparation stages (initial orientation, center/reference generation, FPN extraction), and
- the later target-detection stages that depend on temporal continuity.

The local switch:

```
clt_parameters.imp.cuas_targets_en
```

is used to disable target detection while still allowing the earlier orientation/FPN stages to run. This is useful when debugging sequence geometry, IMS consistency, or FPN extraction without generating target products.

For CUAS processing, a real camera movement or a sharp IMU orientation discontinuity inside one source-list sequence is considered invalid input, not a condition to be automatically "recovered". The later CUAS stages use running temporal references, including reuse of the previous `~8 s` sequence when building the reference image for target detection. If the camera moved in between, that reference image no longer matches the later data.

Operational rule:

- if the camera was moved, or the IMS quaternion has a discontinuity large enough to break overlap/continuity, split the sequence;
- do not reuse the previous running-average reference across that split;
- process the "good" part of the sequence separately from the later part;
- if needed, use `.list` `first/last` trimming to isolate the valid sub-range;
- for the later sub-range, start a new sequence rather than continuing through the bad transition.

As of April 2026, the CUAS code is intentionally conservative here: if initial orientations fail in CUAS mode because IMS/orientation input is inconsistent, processing aborts with a log message instead of retrying the same bad tail indefinitely.

### Next calibration step: target-assisted LY for LV data
The next planned work after restoring/stabilizing CUAS detection is to improve LWIR field calibration ("Lazy Eye", `LY`) for the Latvia (`LV`) data.

Calibration is currently understood as two layers:

1. Factory calibration:
   - uses the special `3.0 m x 7.0 m` curved checkerboard target,
   - measures lens distortion / aberration, including non-radial terms,
   - is assumed fixed for the current camera unless the optics themselves change.
2. Field calibration / `LY`:
   - keeps the lens model fixed,
   - adjusts residual per-sensor attitude errors only,
   - is split into:
     - convergence/divergence (`disparity at infinity`),
     - all remaining relative attitude corrections.

The existing `LY` implementation already supports multiscene adjustment. The planned new functionality is not a new solver; it is a new way to prepare the calibration data for that existing multiscene `LY` code.

Why this is needed for LV:

- the 2025 data often had textured clouds across much of the sky, so ordinary scene content was sufficient for `LY`,
- the Latvia data has mostly clear blue sky,
- usable static texture is concentrated near the bottom of the FOV,
- current calibration is good enough for target detection,
- but accurate ranging needs much better residual disparity accuracy.

Planned approach:

- first restore and validate moving-target detection/tracking,
- then use detected targets themselves as additional `LY` features,
- combine multiple scenes so the selected calibration data covers most of the FOV, especially the corners,
- reuse the simultaneity of each LWIR scene: each accepted target contributes measurements from all 16 sensors at the same timestamp.

One likely implementation path is to build a composite/synthetic calibration scene:

- instead of a few targets (`~5`) per timestamp, gather targets from many timestamps,
- keep each accepted target as a simultaneous 16-view measurement from one scene,
- merge many such measurements into a synthetic scene or multiscene bundle,
- effectively present the `LY` stage with a much denser set of valid points (`~500` targets instead of `~5`), while still respecting the per-scene simultaneity needed by the geometry.

Operational goal:

- use textured terrain where available,
- augment it with detected airborne targets (mostly birds in the current LV data, possibly drones later),
- solve `LY` on a selected subset of scenes that together span the full FOV,
- then reuse the improved `LY` correction for final disparity/ranging runs.

This work should stay separate from factory calibration logic. The intended change is in scene/feature preparation and selection, not in the underlying multiscene `LY` optimizer.

## OpenMVS 3D Reconstruction (foliage subproject)

Public repo: `git@git.elphel.com:Elphel/foliage.git` (cloned at `attic/foliage/`).
Internal context: `attic/foliage-internal/docs/FOPEN_MVS_3D_Workflow.md`.

### Input files per segment reference scene

| File | Content |
|------|---------|
| `*-TERRAIN-DISP-MERGED.tiff` | Multi-frame 32-bit float TIFF. Slices labelled `"average"` (first two) are skipped. Remaining slices are labelled with timestamps (`<ts>-<n>`) and correspond to the fused 640×512 LWIR image for each scene. |
| `*-INTERFRAME.corr-xml` | Per-scene poses under keys `EYESIS_DCT_AUX.scenes_<ts>` = `X, Y, Z, azimuth, tilt, roll` (metres, radians) relative to the reference scene. `_dt` / `_d2t` variants are velocity/acceleration and are ignored. Reference scene timestamp in `EYESIS_DCT_AUX.timestamp_reference`. |

### Coordinate conventions

**Elphel camera frame** (per `GeometryCorrection`):
- X = image-plane right, Y = image-plane up, Z < 0 along optical axis (OpenGL-style)
- `focalLength` in mm (field in `GeometryCorrection`), `pixelSize` in μm
- Focal length in pixels: `fl_px = focalLength_mm / (pixelSize_μm × 0.001)`
- For 32° HFoV, 640 px width: `fl_px ≈ 1116`
- Principal point: (319.5, 255.5) for 640×512

**Rotation matrix** — mirrors `GeometryCorrection.getCommonRotMatrix()`:
```
R_w2c = R_roll(roll) @ R_tilt(tilt) @ R_az(azimuth)
```
Each factor is the transposed standard rotation (R_Y^T, R_X^T, R_Z^T respectively).
Combined R_w2c transforms world → Elphel-camera coords.

**COLMAP/OpenMVS flip** (Elphel Y-up/−Z-fwd → COLMAP Y-dn/+Z-fwd):
```
R_colmap = diag(1, −1, −1) @ R_w2c
t_colmap = R_colmap @ (−C)     where C = [X, Y, Z] is the camera centre
```

Convention is validated in Phase 3 Blender smoke test (see workflow doc).

### Pipeline (all CLI tools, no library linking)

```
Phase 0  scripts/elphel_to_colmap.py
         → <out>/colmap/  (cameras.txt, images.txt, 16-bit PNGs)

Phase 2  InterfaceCOLMAP → DensifyPointCloud → ReconstructMesh
         → RefineMesh → TextureMesh
         (one-command: scripts/run_openmvs_pipeline.sh)
```

OpenMVS is used entirely as CLI tools. No C++ library linking required.
Set `OPENMVS_BIN=` env var if tools are not on PATH; see script header for
build instructions.

### Disparity-to-depth formula (not used by OpenMVS directly)

```
Z_m = SCENE_UNITS_SCALE × focalLength_mm × disparityRadius_mm
      / (disparity_px × 0.001 × pixelSize_μm)
```

where `SCENE_UNITS_SCALE = 0.001` (mm → m) and `disparityRadius = 150 mm`
(= √2 × cameraRadius, set in `GeometryCorrection`).
The 80×64 `*_SZXY.tiff` contains pre-computed (S, Z, X, Y) per tile and
can be used as a Phase 1 sparse seed once a suitable importer is written.

## Latest Additions
### Segment freezing with `keep_segments`
Index scenes (`*-index`) contain `*-INTERFRAME.corr-xml` with keys like:

```
<entry key="EYESIS_DCT_AUX.refscenes_<timestamp>"></entry>
```

The new parameter:

```
IntersceneMatchParameters.keep_segments = true
```

If true, and refscenes exist in the index scene’s `*-INTERFRAME.corr-xml`, those reference scenes are reused instead of recomputing segment splits. This preserves segment definitions when re-running with new parameters or calibrations.

### Elevation histograms for quick classification
Files like:

```
.../v88/<timestamp>-elevations_histogram.csv
```

contain cumulative histograms of tile elevations relative to a fitted ground plane. Columns:

- `AGL`: single value in row 2, altitude above ground.
- `ELEVATION`: elevation value for histogram bin.
- `FRACTION_LOWER`: fraction of tiles below that elevation.

Parameters in `IntersceneMatchParameters`:

```
fgnd_hist_gen = true
fgnd_hist_suffix = "-elevations_histogram.csv"
fgnd_hist_frac = 0.03
fgnd_hist_bins = 100
```

`fgnd_hist_frac` defines outlier trimming; min/max elevation are computed from the remaining tiles. These histograms enable fast, low-bandwidth MCP classification by reading small CSVs instead of large images. Example classification features:

- AGL per segment
- "Forestation ratio" (fraction of tiles above a height threshold, e.g. 1 m)
- Maximum elevation at a given FRACTION_LOWER (e.g. 0.9)

Suggested aggregation: for each scene sequence, compute average/min/max AGL and forestation ratio across reference scenes.

### Global LMA objective and correlation modes (implementation notes)
The global LMA stage is intended to solve one joint optimization problem, not separate ones:

- Correlation residuals for all selected scene/reference pairs (currently scene-to-center, later also 1/4 and 3/4 references).
- Inter-scene LPF/curvature penalties.
- Optional pull-to-target penalties.

The practical requirement is to keep these terms explicit and inspectable in one place in the code:

- `prepareLMA()` should build one combined `y_vector` and one combined `w_vector`.
- Correlation rows should be concatenated first, then LPF rows, then pull rows.
- `getFxDerivs()` should contain visible blocks for each term group in that same order.

#### Correlation residual representation
The single-scene pair LMA supports two measurement modes:

1. Direct XY mode:
   - Two residual components per tile are used as-is (`dx`, `dy`).
2. Eigenvector mode (preferred in current data):
   - Correlation peak shape (ellipse) is used to weight directional certainty.
   - Residuals and derivatives are transformed into eigen directions.
   - Direction perpendicular to linear texture carries most information; along-line direction is de-emphasized.

Current development/testing mostly uses mode 2 (eigenvector mode), but mode 1 should remain supported and periodically checked to avoid bit-rot.

#### Why this structure matters
Keeping all residual segments in one explicit Jacobian path makes it easier to:

- Verify convergence term-by-term.
- Diagnose whether LPF/pull are dominating correlation, or vice versa.
- Add new residual segments incrementally without losing traceability of `y - f(x)` composition.

## Parked State Snapshot (2026-02-25)
This section captures the latest validated state before pausing Global LMA work for C-UAS field-test preparation.

### What is validated now
- Quarter references are selected and used in Global LMA flow.
- Quarter `-INTER-INTRA-LMA.tiff` prerequisites are generated before Global LMA when missing.
- Force regeneration path is working (`force quarter regeneration = true`).
- SfM-only quarter generation path is running and bypassing pose/LMA adjustment in the quarter build stage.
- Regenerated quarter files looked reasonable on manual visual inspection.
- Full run continues into Global LMA and finishes normally.

### Last known-good config for quarter-force + sfm-only
- `/media/elphel/btrfs-data/lwir16-proc/NC/config/MI250-v89-nc_site_37A-FORCE-SFM-XYAT-lpf_XYZ100-ATR3-freeze-RMS1e-4-ratio100-quat4-tw0.0005-cw1.0-noperseries-regularization0.8-transl-rot-nocombo-no_out-preseries_only.corr-xml`

### Key files from the validated run
- Quarter 1:
  `/media/elphel/btrfs-data/lwir16-proc/NC/models/1763232117-1763234145/1763233239_481130/v89/1763233239_481130-INTER-INTRA-LMA.tiff`
- Quarter 3:
  `/media/elphel/btrfs-data/lwir16-proc/NC/models/1763232117-1763234145/1763233241_531813/v89/1763233241_531813-INTER-INTRA-LMA.tiff`
- Baseline copy for comparison:
  `/media/elphel/btrfs-data/lwir16-proc/NC/models/1763232117-1763234145/1763233240_531480/debug01/46`

### Current run pattern (stable for automation)
1. Restore config.
2. Optional parameter changes in `Setup CLT` (`Inter-Global-LMA` section).
3. Save config revision.
4. Run `Aux Build Series`.
5. Inspect console + CSV/TIFF outputs; archive debug artifacts as needed.

## `refineMotionVectors()` — Two-Stage Motion Vector Refinement

### Background: MCLT, Phase Correlation, and Noise Integration

The pipeline detects sub-pixel UAS targets using the GPU Tile Processor (TP), which operates
on fixed 16×16 50%-overlapped tiles using a 2D Modified Complex Lapped Transform (MCLT). MCLT
is structurally similar to FFT: convolution and correlation are multiplication in the Transform
Domain (TD). Critically, fractional-pixel image shifts in the pixel domain (PD) correspond to
phase rotations (multiplication by complex unit vectors) in the TD. Integer shifts select which
16×16 window is loaded; the residual fractional shift (−0.5 to +0.5 px) is applied as TD phase
rotation. Both are handled by the GPU kernel.

Pairwise tile correlation uses Phase Correlation: each TD component is normalized by its
amplitude (retaining only phase), so the inverse-transformed result is a near-delta-function
for noiseless data. Noise mitigation uses two mechanisms:

1. **Fat zero:** a positive constant added to the normalization denominator, trading between
   pure phase correlation (sharp but noise-fragile) and classical correlation (noise-robust but
   blunt). Controlled by `cuas_fat_zero` / `cuas_rng_fz`.
2. **TD pre-averaging:** for a linearly moving target, pairwise TD products with the same
   temporal offset carry the same target phase shift. Accumulating N such pairs before
   normalization averages down noise by √N while the signal remains coherent. This is done in
   TD, before normalization — pairs like (0,8),(1,9),(2,10),…,(7,15) are summed, then
   normalized and IDCT'd. `getTargetsFromCorr2d()` implements this.

The current motion-vector scan integrates the **full 16×16 tile** for each pairwise product.
For a sub-pixel target occupying only ~2–4 px, this wastes most of the integration area on
background noise, limiting achievable SNR.

### Purpose of `refineMotionVectors()`

After the first (non-centered) pass gives approximate motion vectors (MV) and the
virtual-tracking-camera pass (`shiftAndRenderAccumulate()`) locates the target centroid
within each tile, we know where in each tile the target sits. The new method exploits this
knowledge to re-run the pairwise correlation with a spatial mask that zeros out pixels far
from the known target center. This:

- Reduces the effective noise integration area from 16×16 to a small window of radius ~r1
  (default 6 tiles = ~6 px),
- Optionally increases the number of accumulated pairs (`recalc_mv_boost`, default 4×),
- Produces a refined MV estimate, which feeds back into the next centered-accumulation pass
  for improved ranging and target localization.

The method is called (when `recalc_mv=true`) **before**
`CuasMotion.shiftAndRenderAccumulate()` at line 8203, replacing the motion vectors that
`extended_scan` / `targets_nonoverlap` carry.

`recalc_mv` is disabled for `slow_mode` (background suppression path) because the slow
path uses a different temporal pre-filter and the refinement is only meaningful for the fast
(target-motion) path.

### Parameters

```java
boolean recalc_mv       = clt_parameters.imp.cuas_recalc_mv && !slow_mode;
//   When true, re-correlate with spatial mask to refine MVs before centered accumulation.

double  recalc_mv_boost = clt_parameters.imp.cuas_recalc_mv_boost; // default ~4.0
//   Multiply the default number of correlation pairs used in the refinement step.
//   More pairs → better SNR; practical ceiling is limited by sequence length.

double  recalc_mv_r0    = clt_parameters.imp.cuas_recalc_mv_r0;    // default ~2.0 (tiles)
//   Raised-cosine mask inner radius. For r ≤ r0: weight = 1.0

double  recalc_mv_r1    = clt_parameters.imp.cuas_recalc_mv_r1;    // default ~6.0 (tiles)
//   Raised-cosine mask outer radius. For r ≥ r1: weight = 0.0
//   For r0 < r < r1: weight = 0.5 * (cos(π*(r−r0)/(r1−r0)) + 1)
```

The mask is centered on the known target fractional position within the tile (from the
prior centroid/LMA fit). Units are pixels (not tile indices).

### Pipeline position

```
[non-centered pass]  → approximate MV in target_sequence_multi / extended_scan
         ↓
[refineMotionVectors()]    ← NEW (optional, recalc_mv=true, fast mode only)
   apply spatial mask around target center
   re-run TD pairwise accumulation with boosted pairs
   return refined MV → overwrite extended_scan (or produce new vector_field)
         ↓
[shiftAndRenderAccumulate()]   ← existing, line 8203
   virtual tracking camera, long exposure
         ↓
[getAccumulatedCoordinates() / LMA]
```

### Open questions (to resolve before implementation)

1. **Where is the target center at this point in the code?** The centroid/LMA result lives
   in `target_sequence_multi`. Which field indices (`RSLT_X`, `RSLT_Y`, etc.) hold the
   fractional pixel offset within the tile for each keyframe?

2. **What does the mask multiply?** The source `fpixels_tum` float array, or the TD
   tiles already computed by the GPU? If the GPU has already done the TD transform by this
   point, we need to apply the mask in PD before re-loading, or apply it in TD (which is a
   convolution — more complex). Is the mask applied in Java before calling the GPU, or does
   it need a new GPU kernel path?

3. **Pair geometry for the boosted refinement:** `recalc_mv_boost` scales the number of
   pairs. Does this mean reusing the same `cuas_corr_offset` with more `(frame0+d,
   frame1+d)` pairs, or changing the offset too? Are there enough frames in the sequence
   to support 4× more pairs without running past the sequence boundary?

4. **Return value:** Should `refineMotionVectors()` return a new `double[][][]` vector
   field (same structure as `extended_scan`) with refined `RSLT_VX/VY`, or modify the
   existing `target_sequence_multi` in-place?

5. **Tiles with no known target:** For tiles where the first pass found no valid target
   (`target_sequence_multi[nseq][ntile] == null` or score below threshold), should
   refinement be skipped (keep original MV) or skipped entirely (null)?

6. **New `IntersceneMatchParameters` fields:** Are `cuas_recalc_mv`,
   `cuas_recalc_mv_boost`, `cuas_recalc_mv_r0`, `cuas_recalc_mv_r1` already added to
   `IntersceneMatchParameters.java`, or do they need to be added as part of this work?

### Next TODO (priority order)
1. Performance pass: identify current bottlenecks and low-hanging optimizations.
   - **Two-stage motion vector calculation:** Repeat correlation with a tight mask (R=4) around located targets to significantly decrease the noise integration area and eliminate motion blur.
   - **Fix ranging:** Correct ranging logic for weak targets.
   - **Radar Annotation Cleanup:** Implement the GUI checkbox logic to hide/ignore false-positive targets at infinity (atmospheric turbulence) in the video output, *except* for the ground-truth flight log target.
   - **Flight Log Robustness:** Test Latvia datasets that lack flight logs to ensure the pipeline degrades gracefully without crashing.
2. **Agent-Driven Algorithm Evolution (Future Roadmap):**
   - **Adaptive Integration Windows:** Implement a competitive selection mechanism (e.g., 1x vs 4x integration) where the pipeline automatically selects the motion vector that yields the highest Contrast-to-Noise Ratio (CNR), rather than relying on hardcoded thresholds.
   - **Fat Zero Auto-Scaling:** Automatically scale `cuas_rng_fz` (fat zero) down proportionally to the length of the temporal integration (longer integrations lower the stochastic noise floor, allowing closer-to-pure phase correlation).
   - **Acceleration Compensation:** Refine the "Virtual Moving Camera" model to handle non-linear motion (like U-turns) that currently "squash" correlation peaks during long integrations.
   - **Hybrid Classical/DNN Tracking (U of U FOPEN Collaboration):** Export unnormalized MCLT correlation hyperstacks to train an Attention/SSM-based neural network. The goal is to replace the LMA fitter with a DNN regression head `(X, Y, vX, vY)` that applies learned "soft masks" to 16x16 macroblocks, running inference via TorchScript/DJL inside the Java pipeline. See `03_UU_RongRong_Hybrid_DNN_Architecture.md` for details. **Meeting scheduled with Rong-Rong team for Thursday, May 7th @ 2 PM at Elphel.**
   - *Note: These deeper algorithmic optimizations are intentionally deferred. The strategy is to establish a working baseline first, expose the necessary low-bandwidth tile metrics via the MCP server, and then allow AI agents (Codex, Claude, Gemini) to autonomously sweep, analyze, and optimize these specific sub-problems.*
3. **FPGA / Hardware Teaming Roadmap (U of U Collaboration):**
   - **MCP for GTKWave:** Develop a Model Context Protocol (MCP) bridge to allow LLMs to natively analyze `.vcd` files. This will enable natural language querying of simulation waveform data (e.g., "Find the memory arbiter hang").
   - **Cocotb Integration:** Revive the Python-based simulation-to-hardware workflow. The goal is to ensure that testbenches used in Icarus Verilog remain perfectly valid through physical hardware testing and eventual C-code kernel driver development.
   - **GPU Top-Level Dispatcher (Human Latency Reduction):** Investigate moving the GPU pipeline orchestration from Java (JCuda sequential calls) into a single C++ "Master Dispatcher" kernel. By hollowing out the Java loops and decision logic and placing them into a single `.cu` file that calls mathematical modules as `__device__` functions, we eliminate the need to duplicate scheduling code across C++ and Java. This ensures that the production ImageJ environment uses the exact same orchestration logic as the development/Nsight environment, reducing human effort and convergence-translation errors.
   - **Agent-Assisted Onboarding:** Leverage agents to bridge the gap for "occasional" users (like graduate students) by guiding them through the specialized hardware/Verilog knowledge base. **Collaboration with U of U FPGA students starts after exams end in May.**
4. Batch replay of this quarter+global stage on previously processed data; classify failures and choose representative/challenging short test sequences.
5. Algorithm improvement for occlusion handling:
   - Predict likely-occluded tiles from depth/disparity behavior.
   - Conditionally zero tile weights even when correlation strength is high, if non-occluded data is sufficient.
   - Keep enough constraints for small-overlap scenes where joint XYZATR fitting can become underconstrained.
4. Add global-LMA-style disparity/depth refinement stage:
   - Start from center + quarter maps.
   - Reuse pairwise correlations and include occlusion-aware weighting.
   - Revisit FG/BG disparity slices in `-INTER-INTRA-LMA.tiff` (currently mostly pass-through/copy).
5. Deeper threading/performance tuning after correctness is stable.
