Commit 1ea74186 authored by Andrey Filippov's avatar Andrey Filippov

Initialize Docker-first elphel393 migration workspace (stage1 CLI)

parents
Pipeline #3690 canceled with stages
# Local working trees and build artifacts
workspace/
cache/
logs/
tmp/
# Python
__pycache__/
*.pyc
# Editor/system
.DS_Store
*~
# elphel393-docker
Docker-first migration workspace for running and maintaining `elphel393` (Yocto warrior) on a newer host while keeping compatibility with existing workflow.
## Current goal
Stage 1 only:
1. Build from command line in Docker on this machine.
2. Keep behavior close to the current known-good setup on `elphel@192.168.0.107:/home/elphel/git/elphel393`.
3. Defer Eclipse/VDT integration details to later, but keep architecture decisions compatible with that future work.
## Mode policy
- This repository targets **developer mode only**.
- For git access, use **SSH only** (`git@git.elphel.com:...`).
- HTTPS/user mode is intentionally out of scope here.
## Scope
In scope now:
- Docker image/runtime for Yocto warrior builds.
- Repeatable host-to-container workflow.
- Documentation and checklists for migration.
- Utility scripts used during hardware work (including Konsole multi-SSH launcher).
Out of scope now:
- Full Eclipse integration.
- VDT plugin integration.
- Runtime deployment redesign on camera hosts.
## Repository layout
- `docker/` container build/runtime files.
- `scripts/` helper scripts.
- `docs/` migration notes and plans.
## Quick start
1. Build and enter the container:
```bash
./scripts/run_docker.sh
```
2. Inside container, use normal `elphel393` flow (after workspace is prepared):
```bash
./setup.py
cd poky
. ./oe-init-build-env
bitbake u-boot device-tree linux-xlnx core-image-elphel393
```
Bootstrap workspace in developer mode:
```bash
./scripts/bootstrap_workspace.sh
./scripts/verify_ssh_remotes.sh
```
## References
- Main upstream repository: `git@git.elphel.com:Elphel/elphel393.git` (`warrior` branch)
- Development wiki: https://wiki.elphel.com/wiki/Development_for_10393
- Poky manual page used by project: https://wiki.elphel.com/wiki/Poky_manual
## Notes
- The canonical setup/update logic for subprojects remains `setup.py` from project root.
- Migration should preserve sparse-kernel workflow and future Eclipse/VDT compatibility.
FROM ubuntu:20.04
ENV DEBIAN_FRONTEND=noninteractive
ENV LANG=en_US.UTF-8
ENV LC_ALL=en_US.UTF-8
ARG USER_NAME=builder
ARG USER_ID=1000
ARG GROUP_ID=1000
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
locales \
tzdata \
sudo \
git \
rsync \
gawk \
wget \
diffstat \
unzip \
texinfo \
chrpath \
socat \
cpio \
build-essential \
gcc-multilib \
g++-multilib \
python \
python-numpy \
python3 \
python3-pip \
python3-pexpect \
python3-git \
python3-jinja2 \
python3-numpy \
xz-utils \
debianutils \
iputils-ping \
file \
libssl-dev \
libsdl1.2-dev \
xterm \
bc \
bison \
flex \
libncurses5-dev \
libncursesw5-dev \
ccache \
less \
vim \
&& locale-gen en_US.UTF-8 \
&& update-locale LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 \
&& rm -rf /var/lib/apt/lists/*
RUN groupadd -g "${GROUP_ID}" "${USER_NAME}" \
&& useradd -m -u "${USER_ID}" -g "${GROUP_ID}" -s /bin/bash "${USER_NAME}" \
&& echo "${USER_NAME} ALL=(ALL) NOPASSWD:ALL" >/etc/sudoers.d/"${USER_NAME}" \
&& chmod 0440 /etc/sudoers.d/"${USER_NAME}"
COPY docker/entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh
USER ${USER_NAME}
WORKDIR /work/elphel393
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
CMD ["bash"]
services:
warrior:
image: elphel393-warrior:local
build:
context: ..
dockerfile: docker/Dockerfile
args:
USER_NAME: ${USER_NAME:-builder}
USER_ID: ${USER_ID:-1000}
GROUP_ID: ${GROUP_ID:-1000}
working_dir: /work/elphel393
environment:
HOME: /home/${USER_NAME:-builder}
CCACHE_DIR: /home/${USER_NAME:-builder}/.ccache
LANG: en_US.UTF-8
LC_ALL: en_US.UTF-8
tty: true
stdin_open: true
volumes:
- ../workspace:/work/elphel393
- ../cache/downloads:/work/elphel393/poky/build/downloads
- ../cache/sstate-cache:/work/elphel393/poky/build/sstate-cache
- ../cache/ccache:/home/${USER_NAME:-builder}/.ccache
#!/usr/bin/env bash
set -euo pipefail
mkdir -p "${HOME}/.ccache"
mkdir -p /work/elphel393 || true
# Avoid git safe-directory issues when host and container users differ.
git config --global --add safe.directory /work/elphel393 || true
exec "$@"
# Project details (working context)
This file captures hardware/system context that affects migration decisions.
## System role
`elphel393` runs on Elphel camera hardware (Zynq ARM+FPGA) with custom Linux/Yocto stack and many custom drivers and FPGA-linked interfaces.
## Sensor and host topology
Active camera subsystem hosts:
- `192.168.0.41` controls LWIR channels `0..3` (channel 0 at top, then clockwise viewed in camera direction)
- `192.168.0.42` controls LWIR channels `4..7`
- `192.168.0.43` controls LWIR channels `8..11`
- `192.168.0.44` controls LWIR channels `12..15`
- `192.168.0.45` controls RGB channels:
- `16` right/top
- `17` right/bottom
- `18` left/bottom
- `19` left/top
- `192.168.0.46` controls IMU/GNSS subsystem (InertialSense-based path)
Each host has local SATA SSD storage. Filenames are synchronized via shared master timestamp.
## Synchronization
In addition to Ethernet communication, subsystem hosts are tied by a dedicated coax sync link:
- Trigger edge + 64-bit master timestamp
- Decoded by FPGA
- Timing accuracy around 1 microsecond
## Software architecture constraints
- Kernel is heavily customized (not only `drivers/elphel`).
- Sparse-tree method is used for modified kernel files.
- Eclipse workflow follows symlinks into git-controlled sparse sources.
- Header handling for CDT is augmented by custom scripts that inspect Yocto build include usage.
- Build/deploy workflow is already script-integrated with Eclipse and SCP.
## Migration intent
- Primary target: command-line reproducible Docker build on this workstation.
- Secondary target (later): preserve compatibility for Eclipse + VDT workflow.
- Keep old workstation (`192.168.0.107`) as reference while migrating.
# Stage 1 plan: command-line build in Docker
Goal: produce a reliable CLI build path on this machine, inside Docker, matching current behavior as closely as practical.
## Success criteria
1. `setup.py` runs successfully in containerized workspace.
2. `bitbake u-boot device-tree linux-xlnx core-image-elphel393` completes.
3. Output artifacts are produced in expected deploy path.
4. No host reboot required for normal build cycles.
## Baseline source of truth
- Reference environment:
- Host: `elphel@192.168.0.107`
- Path: `/home/elphel/git/elphel393`
- Branch: `warrior`
- Setup/update entrypoint:
- `./setup.py`
- Mode:
- developer mode only (SSH remotes)
## Work steps
1. Inventory reference environment (`scripts/audit_107.sh`).
2. Build Docker image (`docker/Dockerfile`).
3. Create workspace volume and clone `elphel393` (`warrior`) into `workspace/`.
- force SSH: `git@git.elphel.com:Elphel/elphel393.git`
4. Run `setup.py` inside container.
5. Build incremental targets first (`u-boot`, `device-tree`, `linux-xlnx`), then full image.
6. Capture logs and deltas vs `.107`.
## Known constraints
- Yocto warrior and dependencies are sensitive to host toolchain drift.
- Some components still expect Python 2 era compatibility.
- Kernel/FPGA workflows and Eclipse-specific conventions must remain possible later.
## Deferred to later stages
- Eclipse runtime integration and launch config parity.
- VDT plugin/toolchain specifics.
- Deployment UX and automation hardening.
# Migration trouble assessment (initial)
This is an initial risk view before running migration actions.
## Estimated trouble by area
1. Yocto warrior dependency compatibility on modern host: **High**
2. Docker user/permission/ownership behavior with existing git trees: **High**
3. Build cache performance (`downloads`, `sstate-cache`) and disk usage: **Medium**
4. Reproducibility of `setup.py` multi-repo state inside container: **Medium**
5. Eclipse/VDT future compatibility (not implemented in Stage 1): **Medium-High**
## Why this is manageable
- Existing `.107` installation is available as a known-good reference.
- `setup.py` already centralizes multi-repo orchestration.
- Stage 1 is limited to CLI build parity, reducing variables.
- User/HTTPS mode is excluded; this repo standardizes on developer/SSH mode only.
## Main failure modes to watch
1. Missing/changed package versions in container.
2. Broken SSH credentials inside container for private repo access.
3. UID/GID mismatch causing file ownership or git safety issues.
4. Non-deterministic branch/hash selection in project dependencies.
5. Build scripts assuming non-container host paths.
## Mitigation strategy
1. Keep container user mapped to host UID/GID.
2. Persist `downloads` and `sstate-cache` across runs.
3. Keep migration logs and exact command history in this repo.
4. Validate each step against `.107` before moving forward.
5. Defer IDE concerns until CLI build is stable.
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
LOG_DIR="${REPO_ROOT}/logs"
mkdir -p "${LOG_DIR}"
TARGET="${TARGET:-elphel@192.168.0.107}"
REMOTE_REPO="${REMOTE_REPO:-/home/elphel/git/elphel393}"
TS="$(date +%Y%m%d_%H%M%S)"
OUT="${LOG_DIR}/audit_107_${TS}.log"
{
echo "=== AUDIT START ${TS} ==="
echo "TARGET=${TARGET}"
echo "REMOTE_REPO=${REMOTE_REPO}"
echo
ssh "${TARGET}" "set -euo pipefail; \
echo '=== HOST ==='; hostname; uname -a; \
echo; \
echo '=== REPO ==='; cd '${REMOTE_REPO}'; pwd; \
git rev-parse --abbrev-ref HEAD; \
git rev-parse HEAD; \
git remote -v; \
echo; \
echo '=== ROOT CONTENTS ==='; ls -la | sed -n '1,160p'; \
echo; \
echo '=== README (top) ==='; sed -n '1,220p' README.md; \
echo; \
echo '=== setup.py (top) ==='; sed -n '1,320p' setup.py; \
echo; \
echo '=== projects.json (top) ==='; sed -n '1,320p' projects.json; \
echo; \
echo '=== TOOLCHAIN BITS ==='; \
command -v bitbake || true; \
command -v python || true; python --version 2>&1 || true; \
command -v python3 || true; python3 --version 2>&1 || true; \
command -v git || true; git --version || true; \
command -v docker || true; docker --version 2>&1 || true"
} | tee "${OUT}"
echo "Saved audit log: ${OUT}"
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
WORKSPACE="${REPO_ROOT}/workspace"
REMOTE_URL_DEFAULT="git@git.elphel.com:Elphel/elphel393.git"
REMOTE_URL="${REMOTE_URL:-$REMOTE_URL_DEFAULT}"
BRANCH="${BRANCH:-warrior}"
RUN_SETUP=0
usage() {
cat <<'EOF'
Prepare local workspace/elphel393 clone using SSH remote.
Usage:
bootstrap_workspace.sh [--run-setup]
Environment overrides:
REMOTE_URL default: git@git.elphel.com:Elphel/elphel393.git
BRANCH default: warrior
Notes:
- This script enforces developer mode (SSH remotes only).
- --run-setup runs ./setup.py after clone/update.
EOF
}
while [[ $# -gt 0 ]]; do
case "$1" in
--run-setup)
RUN_SETUP=1
shift
;;
-h|--help)
usage
exit 0
;;
*)
echo "ERROR: unknown argument: $1" >&2
usage
exit 1
;;
esac
done
if [[ "${REMOTE_URL}" != git@git.elphel.com:* ]]; then
echo "ERROR: developer mode only. REMOTE_URL must be ssh (git@git.elphel.com:...)" >&2
exit 2
fi
mkdir -p "${WORKSPACE}"
if [[ ! -d "${WORKSPACE}/.git" ]]; then
git clone -b "${BRANCH}" "${REMOTE_URL}" "${WORKSPACE}"
else
(
cd "${WORKSPACE}"
git remote set-url origin "${REMOTE_URL}"
git fetch origin
git checkout "${BRANCH}"
git pull --ff-only origin "${BRANCH}"
)
fi
(
cd "${WORKSPACE}"
git remote set-url origin "${REMOTE_URL}"
)
if [[ "${RUN_SETUP}" -eq 1 ]]; then
(
cd "${WORKSPACE}"
./setup.py
)
fi
echo "Workspace ready: ${WORKSPACE}"
echo "Origin: $(cd "${WORKSPACE}" && git remote get-url origin)"
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
WORKSPACE="${REPO_ROOT}/workspace"
if [[ ! -x "${WORKSPACE}/setup.py" ]]; then
echo "ERROR: ${WORKSPACE}/setup.py not found or not executable." >&2
echo "Run ./scripts/bootstrap_workspace.sh first." >&2
exit 1
fi
if [[ $# -eq 0 ]]; then
TARGETS=(u-boot device-tree linux-xlnx core-image-elphel393)
else
TARGETS=("$@")
fi
TARGETS_STR="${TARGETS[*]}"
"${SCRIPT_DIR}/run_docker.sh" bash -lc "
set -e
cd /work/elphel393
./setup.py
cd poky
set +u
. ./oe-init-build-env build
set -u
bitbake ${TARGETS_STR}
"
#!/usr/bin/env bash
set -euo pipefail
# Open a new Konsole window in a 2x3 split layout and run SSH sessions.
# Default targets:
# root@192.168.0.41 ... root@192.168.0.46
usage() {
cat <<'EOF'
Usage:
konsole_ssh_2x3.sh [user@host ...]
konsole_ssh_2x3.sh [--user USER] [--prefix A.B.C] [--from N]
Options:
--user USER SSH user for generated host list (default: root)
--prefix A.B.C First 3 octets for generated host list (default: 192.168.0)
--from N Starting host number, creates 6 hosts N..N+5 (default: 41)
-h, --help Show this help
Examples:
konsole_ssh_2x3.sh
konsole_ssh_2x3.sh --user root --prefix 192.168.0 --from 41
konsole_ssh_2x3.sh root@192.168.0.41 root@192.168.0.42 root@192.168.0.43 \
root@192.168.0.44 root@192.168.0.45 root@192.168.0.46
EOF
}
log() {
printf '[%s] %s\n' "$(date '+%F %T')" "$*"
}
need_cmd() {
command -v "$1" >/dev/null 2>&1 || {
echo "ERROR: required command not found: $1" >&2
exit 1
}
}
need_cmd konsole
need_cmd qdbus
need_cmd rg
ssh_user="root"
host_prefix="192.168.0"
start_octet=41
declare -a targets=()
while [[ $# -gt 0 ]]; do
case "$1" in
--user)
ssh_user="$2"
shift 2
;;
--prefix)
host_prefix="$2"
shift 2
;;
--from)
start_octet="$2"
shift 2
;;
-h|--help)
usage
exit 0
;;
--)
shift
while [[ $# -gt 0 ]]; do
targets+=("$1")
shift
done
;;
-*)
echo "ERROR: unknown option: $1" >&2
usage
exit 1
;;
*)
targets+=("$1")
shift
;;
esac
done
if [[ "${#targets[@]}" -eq 0 ]]; then
for i in 0 1 2 3 4 5; do
n=$((start_octet + i))
targets+=("${ssh_user}@${host_prefix}.${n}")
done
fi
if [[ "${#targets[@]}" -ne 6 ]]; then
echo "ERROR: expected exactly 6 SSH targets, got ${#targets[@]}" >&2
exit 1
fi
layout_file="$(mktemp --tmpdir konsole_2x3_layout_XXXXXX.json)"
trap 'rm -f "$layout_file"' EXIT
cat >"$layout_file" <<'EOF'
{
"Orientation": "Vertical",
"Widgets": [
{
"Orientation": "Horizontal",
"Widgets": [ { "SessionRestoreId": 0 }, { "SessionRestoreId": 0 } ]
},
{
"Orientation": "Horizontal",
"Widgets": [ { "SessionRestoreId": 0 }, { "SessionRestoreId": 0 } ]
},
{
"Orientation": "Horizontal",
"Widgets": [ { "SessionRestoreId": 0 }, { "SessionRestoreId": 0 } ]
}
]
}
EOF
log "Opening Konsole window with 2x3 layout"
konsole --separate --layout "$layout_file" >/dev/null 2>&1 &
konsole_pid="$!"
service="org.kde.konsole-${konsole_pid}"
# Wait for Konsole DBus service.
for _ in $(seq 1 100); do
if qdbus "$service" >/dev/null 2>&1; then
break
fi
sleep 0.05
done
if ! qdbus "$service" >/dev/null 2>&1; then
echo "ERROR: failed to connect to $service" >&2
exit 1
fi
window_path=""
for _ in $(seq 1 100); do
window_path="$(qdbus "$service" 2>/dev/null | rg '^/Windows/[0-9]+$' | head -n1 || true)"
if [[ -n "$window_path" ]]; then
break
fi
sleep 0.05
done
if [[ -z "$window_path" ]]; then
echo "ERROR: could not find Konsole window DBus path for $service" >&2
exit 1
fi
# Wait until all split sessions are created.
for _ in $(seq 1 100); do
count="$(qdbus "$service" "$window_path" org.kde.konsole.Window.sessionCount 2>/dev/null || echo 0)"
if [[ "$count" =~ ^[0-9]+$ ]] && [[ "$count" -ge 6 ]]; then
break
fi
sleep 0.05
done
if ! [[ "$count" =~ ^[0-9]+$ ]] || [[ "$count" -lt 6 ]]; then
echo "ERROR: expected >=6 sessions, got: ${count:-unknown}" >&2
exit 1
fi
mapfile -t sessions < <(
qdbus "$service" "$window_path" org.kde.konsole.Window.sessionList \
| tr -d '"' \
| sed '/^$/d' \
| head -n 6 \
| awk '{a[NR]=$0} END{for(i=NR;i>=1;i--) print a[i]}'
)
if [[ "${#sessions[@]}" -ne 6 ]]; then
echo "ERROR: failed to collect 6 session IDs (got ${#sessions[@]})" >&2
exit 1
fi
for i in 0 1 2 3 4 5; do
session_path="${sessions[$i]}"
[[ "$session_path" == /Sessions/* ]] || session_path="/Sessions/$session_path"
cmd="ssh -o ConnectTimeout=5 ${targets[$i]}"
log "Session $((i + 1)): $cmd"
qdbus "$service" "$session_path" org.kde.konsole.Session.runCommand "$cmd" >/dev/null
done
log "Done."
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
if command -v docker >/dev/null 2>&1; then
:
else
echo "ERROR: docker is not installed" >&2
exit 1
fi
if docker compose version >/dev/null 2>&1; then
COMPOSE=(docker compose)
elif command -v docker-compose >/dev/null 2>&1; then
COMPOSE=(docker-compose)
else
echo "ERROR: docker compose is not available" >&2
exit 1
fi
mkdir -p "${REPO_ROOT}/workspace" "${REPO_ROOT}/cache/downloads" "${REPO_ROOT}/cache/sstate-cache" "${REPO_ROOT}/cache/ccache"
export USER_ID="${USER_ID:-$(id -u)}"
export GROUP_ID="${GROUP_ID:-$(id -g)}"
export USER_NAME="${USER_NAME:-$(id -un)}"
if [[ $# -eq 0 ]]; then
CMD=(bash)
else
CMD=("$@")
fi
"${COMPOSE[@]}" -f "${REPO_ROOT}/docker/docker-compose.yml" run --rm warrior "${CMD[@]}"
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
WORKSPACE="${WORKSPACE:-${REPO_ROOT}/workspace}"
if [[ ! -d "${WORKSPACE}" ]]; then
echo "ERROR: workspace not found: ${WORKSPACE}" >&2
exit 1
fi
fail=0
while IFS= read -r gitdir; do
repo="${gitdir%/.git}"
url="$(git -C "${repo}" remote get-url origin 2>/dev/null || true)"
if [[ -z "${url}" ]]; then
continue
fi
if [[ "${url}" == *"git.elphel.com"* ]]; then
if [[ "${url}" != git@git.elphel.com:* ]]; then
echo "NON-SSH remote: ${repo} -> ${url}"
fail=1
fi
fi
done < <(find "${WORKSPACE}" -type d -name .git)
if [[ "${fail}" -ne 0 ]]; then
echo "FAIL: found non-SSH remotes for git.elphel.com repos" >&2
exit 2
fi
echo "OK: all git.elphel.com remotes under ${WORKSPACE} use SSH."
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment