diff --git a/.gitignore b/.gitignore index b80ce30..06e795a 100644 --- a/.gitignore +++ b/.gitignore @@ -26,3 +26,13 @@ Thumbs.db *~ .vscode/ .idea/ + +# Node +node_modules/ + +# Test artifacts +test-results/ +e2e-report/ + +# Harness +.omp/ diff --git a/.omp/plans/zerotier-gateway-switching.md b/.omp/plans/zerotier-gateway-switching.md deleted file mode 100644 index d96d941..0000000 --- a/.omp/plans/zerotier-gateway-switching.md +++ /dev/null @@ -1,1258 +0,0 @@ -# Plan: luci-app-zt-gateway — ZeroTier Exit Gateway Switcher - -## Goal - -A LuCI app for OpenWRT that lets users switch which remote ZeroTier node acts as the internet exit gateway for WIBLAN clients (`10.11.13.0/24`). Users select a region/gateway, choose a switch mode, and the app reconfigures routing with zero or minimal disruption. - -## Existing Architecture (constraints) - -- **Router**: OpenWRT, `192.168.13.1` -- **ZT network**: `e3918db48378cb6a`, interface `ztk4jpk77j`, bridge `br-zt` -- **Current gateway**: `10.11.12.3` (ocirosea641, amsterdam) -- **WIBLAN clients**: `10.11.13.0/24` routed via `ip rule` priority 100 → table 100 -- **Table 100**: `default via dev br-zt` -- **Host route**: ` dev br-zt` (ARP fix — gateway IP not directly reachable via bridge) -- **mwan3**: uses table 1; requires `10.11.13.0/24 dev br-zt table 1` for return traffic -- **Hotplug**: `/etc/hotplug.d/net/99-zerotier-bridge` adds `ztk4jpk77j` to `br-zt`, removes conflicting routes, adds table 1 route -- **Conntrack**: `conntrack` package available; `conntrack -D -s 10.11.13.0/24` flushes WIBLAN entries - -## Two Switch Modes - -### 1. Force Switch (instant, ~1-2s disruption) - -All existing connections break immediately. Clients auto-reconnect through the new gateway. - -**Steps:** -1. Pre-flight: ping new gateway over ZT, abort if unreachable -2. Atomically replace host route + table 100 default route -3. Flush conntrack for `10.11.13.0/24` -4. Update hotplug script + rc.local with new gateway IP -5. Update UCI network routes for persistence -6. Verify: `ip route get 8.8.8.8 from 10.11.13.1 table 100` - -**User impact:** Active downloads/video calls drop and reconnect. Web pages refresh. SSH sessions die. - -### 2. Graceful Drain (zero disruption, complex) - -Existing connections continue on the old gateway until they naturally close. New connections go through the new gateway immediately. Uses fwmark + dual routing table. - -**Steps:** -1. Pre-flight: ping new gateway over ZT, abort if unreachable -2. Create drain table (101) with `default via dev br-zt` -3. Add iptables mangle rules to mark all ESTABLISHED connections from WIBLAN with fwmark `0x100` -4. Add `ip rule add fwmark 0x100 table 101 priority 99` (before the existing priority-100 src rule) -5. Update table 100 default to new gateway -6. Update host route to new gateway -7. Start drain monitor: count conntrack entries with mark; when zero → clean up mangle rules, drain table, fwmark rule -8. Update hotplug script + rc.local + UCI network routes for persistence -9. If drain doesn't complete within configurable timeout (default 10 min), auto-fallback to force switch - -**Key insight:** fwmark 0x100 at priority 99 matches BEFORE the src `10.11.13.0/24` → table 100 rule. So marked (existing) connections go to table 101 (old gateway), while unmarked (new) connections go to table 100 (new gateway). - -**User impact:** Zero disruption. Active connections keep working. New connections get new gateway latency/exit point. - -**Conntrack marking detail:** -```sh -# Restore mark from conntrack for returning packets -iptables -t mangle -A PREROUTING -i br-zt -s 10.11.13.0/24 -m conntrack --ctstate ESTABLISHED,RELATED -j CONNMARK --restore-mark - -# Mark ESTABLISHED packets that don't have a mark yet -iptables -t mangle -A PREROUTING -i br-zt -s 10.11.13.0/24 -m conntrack --ctstate ESTABLISHED,RELATED -m mark --mark 0 -j MARK --set-mark 0x100 - -# Save packet mark back to conntrack -iptables -t mangle -A PREROUTING -i br-zt -s 10.11.13.0/24 -j CONNMARK --save-mark -``` - -**Drain monitor:** Periodic check via `conntrack -L -m 0x100 2>/dev/null | wc -l`. When count is 0, clean up. - -## File Structure - -``` -luci-app-zt-gateway/ -├── Makefile -├── htdocs/ -│ └── luci-static/ -│ └── resources/ -│ └── view/ -│ └── zt-gateway/ -│ └── overview.js # Main UI view -├── root/ -│ ├── etc/ -│ │ └── config/ -│ │ └── zt-gateway # UCI config: gateway registry + active state -│ ├── usr/ -│ │ ├── share/ -│ │ │ ├── acl.d/ -│ │ │ │ └── luci-app-zt-gateway.json # ACL permissions -│ │ │ ├── luci/ -│ │ │ │ └── menu.d/ -│ │ │ │ └── luci-app-zt-gateway.json # Menu entry -│ │ │ └── rpcd/ -│ │ │ └── ucode/ -│ │ │ └── zt-gateway.uc # rpcd backend: switch, status, health, drain -│ │ └── sbin/ -│ │ └── zt-gateway-switch # Shell script: actual switch logic (called by rpcd) -``` - -## UCI Config Schema (`/etc/config/zt-gateway`) - -``` -config global 'global' - option active_gateway '' # Region key of active gateway - option switch_mode 'force' # 'force' or 'graceful' - option drain_timeout '600' # Graceful drain timeout in seconds (default 10 min) - option health_interval '60' # Health check interval in seconds - option auto_failback '0' # Auto-failback to default on gateway down - -config gateway - option region 'amsterdam' - option label 'Amsterdam (ocirosea641)' - option ip '10.11.12.3' - option default '1' # Default gateway on boot - option health_check 'ping' # 'ping' | 'tcp' | 'none' - -config gateway - option region 'tirunelveli' - option label 'Tirunelveli (rpi1000)' - option ip '10.11.12.5' - option default '0' - option health_check 'ping' - -config gateway - option region 'bangalore' - option label 'Bangalore (sensecap-m4)' - option ip '10.11.12.4' - option default '0' - option health_check 'ping' # Not yet joined to ZT network; assign IP after joining -``` - -## rpcd Backend (`/usr/share/rpcd/ucode/zt-gateway.uc`) - -Exposes these ubus methods: - -| Method | Args | Returns | Description | -|---|---|---|---| -| `status` | — | `{ active_gateway, gateways: [...], drain_status? }` | Current state + all configured gateways | -| `switch` | `{ region, mode }` | `{ success, message }` | Execute gateway switch | -| `health` | `{ region }` | `{ reachable, latency_ms }` | Ping a gateway via ZT | -| `drain_status` | — | `{ active, remaining_connections }` | Graceful drain progress | -| `cancel_drain` | — | `{ success }` | Cancel graceful drain, force switch to new gateway | - -## Shell Script (`/usr/sbin/zt-gateway-switch`) - -The rpcd backend calls this script for the actual system changes. Separating it from ucode makes it testable from SSH. - -``` -Usage: zt-gateway-switch [drain_timeout] - -Modes: - force - Instant switch + conntrack flush - graceful - fwmark drain with timeout fallback to force -``` - -**Force switch logic:** -1. Validate new gateway IP reachable: `ping -c 2 -W 3 ` -2. `ip route replace dev br-zt` (host route) -3. `ip route replace default via dev br-zt table 100` -4. `conntrack -D -s 10.11.13.0/24 2>/dev/null` -5. Update `/etc/hotplug.d/net/99-zerotier-bridge` with new gateway IP -6. Update `/etc/rc.local` with new gateway IP -7. Update UCI network routes: `zt_gateway_host`, `zt_gateway_default` -8. `uci commit network` -9. Verify new route - -**Graceful switch logic:** -1. Validate new gateway IP reachable -2. Read current gateway from UCI -3. Create table 101: `ip route add default via dev br-zt table 101` -4. Add mangle rules (fwmark 0x100 for ESTABLISHED) -5. Add `ip rule add fwmark 0x100 table 101 priority 99` -6. Update table 100 default + host route to new gateway -7. Start drain monitor loop: - - Every 5s: `conntrack -L -m 0x100 2>/dev/null | wc -l` - - If count == 0: clean up mangle rules, fwmark rule, table 101 → done - - If timeout exceeded: force flush + clean up → done -8. Update hotplug, rc.local, UCI - -**Cleanup function** (called after drain completes or on cancel): -1. `iptables -t mangle -D ` -2. `ip rule del fwmark 0x100 table 101 priority 99` -3. `ip route del default dev br-zt table 101` - -## LuCI Frontend (`overview.js`) - -### Layout - -``` -┌─────────────────────────────────────────────────┐ -│ ZeroTier Exit Gateway │ -├─────────────────────────────────────────────────┤ -│ │ -│ Active Gateway: amsterdam (10.11.12.3) ● UP │ -│ Switch Mode: [Force v] │ -│ Drain Timeout: 600s (shown if graceful) │ -│ │ -│ ┌─────────────────────────────────────────────┐│ -│ │ Region │ Label │ IP ││ -│ ├─────────────┼──────────────────┼─────────────┤│ -│ │ ● amsterdam │ Amsterdam │ 10.11.12.3 ││ -│ │ ○ tirunelveli│ Tirunelveli │ 10.11.12.5 ││ -│ │ ○ bangalore │ Bangalore │ 10.11.12.4 ││ -│ └─────────────┴──────────────────┴─────────────┘│ -│ │ -│ [Switch to Selected] │ -│ │ -│ ── Status ── │ -│ Drain progress: 14 connections remaining │ -│ [Cancel Drain → Force Switch] │ -│ │ -└─────────────────────────────────────────────────┘ -``` - -### JavaScript architecture - -- Form.Map tied to `zt-gateway` UCI config for settings section (switch mode, drain timeout) -- Radio-button list for gateway selection (reads `config gateway` sections) -- Custom button handler for "Switch to Selected" — calls `zt-gateway.switch` RPC -- Polling for drain status when graceful switch is in progress -- Health status indicator (green/red dot) per gateway, refreshed every `health_interval` - -### Key imports -```js -'use strict'; -import { view } from 'luci.view'; -import { form } from 'luci.form'; -import { rpc } from 'luci.rpc'; -import { ui } from 'luci.ui'; -import { dom } from 'luci.dom'; -import { Poll } from 'luci.poll'; -``` - -## Persistence Model - -The gateway IP appears in **five places** on the router. The switch script updates them all atomically: - -| Location | What | How updated | -|---|---|---| -| Kernel: host route | ` dev br-zt` | `ip route replace` | -| Kernel: table 100 default | `default via dev br-zt table 100` | `ip route replace` | -| UCI: `/etc/config/network` | `zt_gateway_host` + `zt_gateway_default` routes | `uci set` + `uci commit` | -| Hotplug: `99-zerotier-bridge` | host route + table 100 route + table 1 return | `sed` replace | -| Boot: `/etc/rc.local` | all three routes | `sed` replace | - -The hotplug and rc.local currently hardcode `10.11.12.3`. The script replaces these with the new gateway IP. A cleaner future approach would be to have the hotplug/rc.local read from UCI, but that's a refactoring step. - -## Health Check - -Default method: `ping -c 2 -W 3 ` over the ZT network. - -- Runs every `health_interval` seconds via cron or the LuCI polling mechanism -- If active gateway becomes unreachable AND `auto_failback='1'`: switch to the `default='1'` gateway -- Health status displayed in UI with latency - -## Rollback / Safety - -- Pre-flight: refuse to switch if new gateway is unreachable -- Graceful drain timeout: if drain doesn't complete, auto-fallback to force -- If switch script fails partway: the host route + table 100 are updated atomically; worst case the old conntrack entries persist and time out naturally -- Manual recovery: `uci set zt-gateway.global.active_gateway=''` + run `/usr/sbin/zt-gateway-switch force` - -## Files to Create - -| File | Purpose | -|---|---| -| `luci-app-zt-gateway/Makefile` | OpenWRT build system | -| `luci-app-zt-gateway/root/etc/config/zt-gateway` | UCI config skeleton | -| `luci-app-zt-gateway/root/usr/share/acl.d/luci-app-zt-gateway.json` | RPC + UCI ACL | -| `luci-app-zt-gateway/root/usr/share/luci/menu.d/luci-app-zt-gateway.json` | Menu entry | -| `luci-app-zt-gateway/root/usr/share/rpcd/ucode/zt-gateway.uc` | rpcd backend | -| `luci-app-zt-gateway/root/usr/sbin/zt-gateway-switch` | Switch script | -| `luci-app-zt-gateway/htdocs/luci-static/resources/view/zt-gateway/overview.js` | Frontend UI | - -## Verification - -1. Install package on router -2. Configure 2+ gateways in `/etc/config/zt-gateway` -3. **Force switch test**: select new region → Force → verify `ip route show table 100` shows new gateway, `curl ifconfig.me` from WIBLAN client shows new exit IP -4. **Graceful switch test**: select new region → Graceful → verify existing SSH on WIBLAN client stays alive, new connections go through new gateway, drain counter decrements -5. **Fail test**: switch to unreachable gateway → verify pre-flight blocks the switch with error message -6. **Drain timeout test**: graceful switch with 30s timeout → verify it falls back to force after timeout -7. **Reboot persistence**: switch gateway → reboot router → verify table 100 still has the new gateway - -## Future: ZeroTier Central API Integration - -Skipping for now. When added, the switch script should: - -1. Call `DELETE /api/network/{networkId}/route/{routeId}` for the old `0.0.0.0/0` managed route -2. Call `POST /api/network/{networkId}/route` with `{ "target": "0.0.0.0/0", "via": "" }` -3. Toggle "Allow Default Route Override" on new gateway node, disable on old - -Requires storing a ZeroTier Central API token in UCI: -``` -config global 'global' - option zt_central_token '' - option zt_network_id 'e3918db48378cb6a' -``` - -This is only needed so other ZeroTier peers also get the correct default route — WIBLAN clients are already handled by the local `ip rule`. - -## Future: Gateway Auto-Provisioning - -Currently, each gateway node must be manually set up with: -- ZeroTier membership + static IP -- IP forwarding + NAT -- iptables FORWARD rules - -A future enhancement could SSH into candidate nodes from the controller and set up NAT automatically, or use an agent on each gateway that phones home. Out of scope for v1. - ---- - -## Building and Installing luci-app-zt-gateway - -This section covers how to build the `luci-app-zt-gateway` package from source into an `.ipk` using the OpenWRT SDK, then install it on the target router. - -### Prerequisites - -- A **Linux** development machine (Ubuntu 22.04+ / Debian 12+ recommended) -- **~2 GB** free disk space for the SDK -- Basic build tools: `gcc`, `make`, `perl`, `python3`, `zstd` -- SSH/SCP access to the target router (`192.168.13.1`) -- The target router's **OpenWRT version** and **architecture** (determine these before downloading the SDK) - -#### Determine router architecture - -SSH into the router and run: - -```sh -cat /etc/openwrt_release # shows OPENWRT_RELEASE, OPENWRT_ARCH, OPENWRT_BOARD -opkg print-architecture # lists available architectures -``` - -The `OPENWRT_ARCH` line tells you the target (e.g. `mipsel_24kc`, `aarch64_cortex-a53`, `x86_64`). - -### Step 1: Download the OpenWRT SDK - -Download the SDK matching your router's **exact OpenWRT version and architecture** from the OpenWRT downloads server: - -- **OpenWRT 24.10**: https://downloads.openwrt.org/releases/24.10.7/targets/ -- **OpenWRT 23.05**: https://downloads.openwrt.org/releases/23.05.5/targets/ - -Navigate into the appropriate target subdirectory (e.g. `x86/64/`, `ramips/mt7621/`, `mediatek/filogic/`) and download the `openwrt-sdk-*.tar.zst` (or `.tar.xz`) file. - -```sh -# Example for x86_64 / OpenWRT 24.10 -wget https://downloads.openwrt.org/releases/24.10.7/targets/x86/64/openwrt-sdk-24.10.7-x86-64_gcc-13.3.0_musl.Linux-x86_64.tar.zst - -# Extract -tar --zstd -xf openwrt-sdk-24.10.7-*.tar.zst -cd openwrt-sdk-24.10.7-* -``` - -### Step 2: Prepare the SDK - -```sh -# Update and install standard feed definitions -./scripts/feeds update -a -./scripts/feeds install -a -``` - -### Step 3: Add the package source - -Place the `luci-app-zt-gateway` directory into the SDK's package tree. There are two approaches: - -#### Option A: Direct placement (simplest) - -```sh -# Copy or symlink the package directory into the SDK -# If your source is at ~/projects/luci-app-zt-gateway/: -ln -s ~/projects/luci-app-zt-gateway package/luci-app-zt-gateway -``` - -#### Option B: Custom feed (recommended for multiple packages) - -```sh -# 1. Create a feed directory -mkdir -p ~/my-openwrt-feed -ln -s ~/projects/luci-app-zt-gateway ~/my-openwrt-feed/luci-app-zt-gateway - -# 2. Register the feed in the SDK -echo 'src-link customfeed /home/YOURUSER/my-openwrt-feed' >> feeds.conf.default - -# 3. Update and install the custom feed -./scripts/feeds update customfeed -./scripts/feeds install -p customfeed luci-app-zt-gateway -``` - -### Step 4: The Makefile - -The `luci-app-zt-gateway/Makefile` uses the LuCI build system (`luci.mk`) which automates directory mapping, installation, and packaging. The Makefile must be in the package root alongside the `htdocs/` and `root/` directories. - -```makefile -# luci-app-zt-gateway/Makefile -include $(TOPDIR)/rules.mk - -PKG_NAME:=luci-app-zt-gateway -LUCI_TITLE:=LuCI support for ZeroTier Exit Gateway Switching -LUCI_DEPENDS:=+luci-base +ucode +conntrack -LUCI_PKGARCH:=all -PKG_VERSION:=1.0.0 -PKG_RELEASE:=1 -PKG_LICENSE:=Apache-2.0 -PKG_MAINTAINER:=Your Name - -include $(TOPDIR)/feeds/luci/luci.mk - -# call BuildPackage - OpenWrt buildroot signature -``` - -Key points: - -- `luci.mk` (not `package.mk`) is used for LuCI applications. It scans `htdocs/` and `root/` and automatically creates install rules that map files to their target paths: - - `htdocs/luci-static/resources/view/` → `/www/luci-static/resources/view/` - - `root/` → mirrors directly onto the target filesystem root (`/`) -- `LUCI_PKGARCH:=all` because LuCI apps are architecture-independent (JS, ucode, shell only). -- `include $(TOPDIR)/feeds/luci/luci.mk` uses the absolute path variant. If building inside the main `openwrt/luci` feed, use `include ../../luci.mk` instead. -- The trailing comment `# call BuildPackage - OpenWrt buildroot signature` is **required** by the OpenWRT feed scanner to discover this package. -- **Dependencies**: `+luci-base` is mandatory. `+ucode` is needed for the rpcd backend. `+conntrack` is needed because the switch script uses `conntrack` CLI. - -#### Directory-to-path mapping (luci.mk automatics) - -When `luci.mk` processes this package, it automatically installs: - -| Source path in package | Target path on router | -|---|---| -| `htdocs/luci-static/resources/view/zt-gateway/overview.js` | `/www/luci-static/resources/view/zt-gateway/overview.js` | -| `root/etc/config/zt-gateway` | `/etc/config/zt-gateway` | -| `root/usr/share/rpcd/ucode/zt-gateway.uc` | `/usr/share/rpcd/ucode/zt-gateway.uc` | -| `root/usr/share/acl.d/luci-app-zt-gateway.json` | `/usr/share/rpcd/acl.d/luci-app-zt-gateway.json` | -| `root/usr/share/luci/menu.d/luci-app-zt-gateway.json` | `/usr/share/luci/menu.d/luci-app-zt-gateway.json` | -| `root/usr/sbin/zt-gateway-switch` | `/usr/sbin/zt-gateway-switch` | - -### Step 5: Configure and build - -```sh -# Select the package in menuconfig -make menuconfig -# Navigate: LuCI → 3. Applications → luci-app-zt-gateway -# Press 'M' to build as module (do NOT press 'Y' — we want an .ipk, not a firmware image) -# Save and exit - -# Build the package with verbose output -make package/luci-app-zt-gateway/compile V=s -``` - -If `make menuconfig` doesn't show the package, ensure the feeds are updated: - -```sh -./scripts/feeds update -a -./scripts/feeds install luci-app-zt-gateway -``` - -### Step 6: Locate the built .ipk - -```sh -find bin/ -name 'luci-app-zt-gateway*.ipk' -type f -``` - -Typical output path: `bin/packages//customfeed/luci-app-zt-gateway_1.0.0-1_all.ipk` - -Because `LUCI_PKGARCH:=all`, the `.ipk` filename ends with `_all.ipk` — it's architecture-independent and works on any OpenWRT target. - -### Step 7: Install on the router - -#### Transfer the package - -```sh -scp bin/packages/*/customfeed/luci-app-zt-gateway_1.0.0-1_all.ipk root@192.168.13.1:/tmp/ -``` - -#### Install via opkg (OpenWRT ≤ 24.10) - -```sh -ssh root@192.168.13.1 -opkg update -opkg install /tmp/luci-app-zt-gateway_1.0.0-1_all.ipk -``` - -If you get dependency errors (e.g. missing `luci-base`), install them first: - -```sh -opkg install luci-base ucode conntrack -opkg install /tmp/luci-app-zt-gateway_1.0.0-1_all.ipk -``` - -#### Install via apk (OpenWRT 25.12+) - -Starting with OpenWRT 25.12, the package manager is `apk` instead of `opkg`: - -```sh -ssh root@192.168.13.1 -apk add --allow-untrusted /tmp/luci-app-zt-gateway_1.0.0-1_all.ipk -``` - -#### Post-install: refresh LuCI cache - -```sh -rm -rf /tmp/luci-indexcache /tmp/luci-modulecache -``` - -Then navigate to the LuCI web UI — the "ZeroTier Exit Gateway" page should appear under **Services**. - -### Step 8: Configure gateways - -After installation, edit the UCI config to add your gateway nodes: - -```sh -vi /etc/config/zt-gateway -``` - -Or use `uci` commands: - -```sh -uci set zt-gateway.global.active_gateway='amsterdam' -uci set zt-gateway.global.switch_mode='force' - -uci add zt-gateway gateway -uci set zt-gateway.@gateway[-1].region='amsterdam' -uci set zt-gateway.@gateway[-1].label='Amsterdam (ocirosea641)' -uci set zt-gateway.@gateway[-1].ip='10.11.12.3' -uci set zt-gateway.@gateway[-1].default='1' -uci set zt-gateway.@gateway[-1].health_check='ping' - -uci add zt-gateway gateway -uci set zt-gateway.@gateway[-1].region='tirunelveli' -uci set zt-gateway.@gateway[-1].label='Tirunelveli (rpi1000)' -uci set zt-gateway.@gateway[-1].ip='10.11.12.5' -uci set zt-gateway.@gateway[-1].default='0' -uci set zt-gateway.@gateway[-1].health_check='ping' - -uci add zt-gateway gateway -uci set zt-gateway.@gateway[-1].region='bangalore' -uci set zt-gateway.@gateway[-1].label='Bangalore (sensecap-m4)' -uci set zt-gateway.@gateway[-1].ip='10.11.12.4' -uci set zt-gateway.@gateway[-1].default='0' -uci set zt-gateway.@gateway[-1].health_check='ping' - -uci commit zt-gateway -``` - -### Step 9: Verify the rpcd backend is registered - -After install, verify the ubus backend is available: - -```sh -# Restart rpcd to pick up the new backend -/etc/init.d/rpcd restart - -# List available methods -ubus list zt-gateway.* -# Expected output: -# zt-gateway.status -# zt-gateway.switch -# zt-gateway.health -# zt-gateway.drain_status -# zt-gateway.cancel_drain - -# Test the status method -ubus call zt-gateway status -``` - -If `ubus list` doesn't show `zt-gateway.*`, check: - -1. The ucode script is executable and at the correct path: `ls -la /usr/share/rpcd/ucode/zt-gateway.uc` -2. The ACL file exists: `cat /usr/share/rpcd/acl.d/luci-app-zt-gateway.json` -3. rpcd log output: `logread -e rpcd` - -### Quick-build cheat sheet (no SDK, manual .ipk) - -If you don't want to bother with the full SDK and just want to produce an `.ipk` manually, you can use the `opkg-utils` tools on any Linux machine: - -```sh -# Install opkg-utils -sudo apt install opkg-utils # Debian/Ubuntu - -# Create the package staging directory -mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/CONTROL -mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/www/luci-static/resources/view/zt-gateway -mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/etc/config -mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/rpcd/ucode -mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/rpcd/acl.d -mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/luci/menu.d -mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/sbin - -# Copy source files into staging (adjust source paths as needed) -cp htdocs/luci-static/resources/view/zt-gateway/overview.js \ - /tmp/luci-app-zt-gateway_1.0.0-1_all/www/luci-static/resources/view/zt-gateway/ -cp root/etc/config/zt-gateway /tmp/luci-app-zt-gateway_1.0.0-1_all/etc/config/ -cp root/usr/share/rpcd/ucode/zt-gateway.uc /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/rpcd/ucode/ -cp root/usr/share/rpcd/acl.d/luci-app-zt-gateway.json /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/rpcd/acl.d/ -cp root/usr/share/luci/menu.d/luci-app-zt-gateway.json /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/luci/menu.d/ -cp root/usr/sbin/zt-gateway-switch /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/sbin/ -chmod +x /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/sbin/zt-gateway-switch - -# Write the control file -cat > /tmp/luci-app-zt-gateway_1.0.0-1_all/CONTROL/control << 'EOF' -Package: luci-app-zt-gateway -Version: 1.0.0-1 -Depends: libc, luci-base, ucode, conntrack -Source: luci-app-zt-gateway -Section: luci -Architecture: all -Installed-Size: 20480 -Description: LuCI support for ZeroTier Exit Gateway Switching -EOF - -# Build the .ipk -cd /tmp -opkg-build -O -o root -g root luci-app-zt-gateway_1.0.0-1_all - -# Result: /tmp/luci-app-zt-gateway_1.0.0-1_all.ipk -``` - -This manual approach skips the SDK entirely. It's suitable for quick iteration during development. The tradeoff: no automatic dependency resolution, no translation compilation, no JS minification. - -### Development workflow: iterate fast without rebuilding - -During development, you can skip the build step entirely by editing files directly on the router: - -```sh -# Edit the JS view directly -scp htdocs/luci-static/resources/view/zt-gateway/overview.js \ - root@192.168.13.1:/www/luci-static/resources/view/zt-gateway/overview.js - -# Edit the ucode backend -scp root/usr/share/rpcd/ucode/zt-gateway.uc \ - root@192.168.13.1:/usr/share/rpcd/ucode/zt-gateway.uc -/etc/init.d/rpcd restart - -# Edit the switch script -scp root/usr/sbin/zt-gateway-switch \ - root@192.168.13.1:/usr/sbin/zt-gateway-switch - -# Edit the UCI config -scp root/etc/config/zt-gateway \ - root@192.168.13.1:/etc/config/zt-gateway - -# Clear LuCI browser cache after JS changes -ssh root@192.168.13.1 'rm -rf /tmp/luci-indexcache /tmp/luci-modulecache' -``` - -Build the `.ipk` only when you need a clean install or are sharing the package. - -### Troubleshooting - -| Problem | Cause | Fix | -|---|---|---| -| `make menuconfig` doesn't show the package | Feed not registered | Run `./scripts/feeds update -a && ./scripts/feeds install -a` | -| `opkg install` fails with "cannot install package" | Missing dependencies | Run `opkg update` then install dependencies first | -| LuCI page shows 404 | JS view not found at expected path | Verify `/www/luci-static/resources/view/zt-gateway/overview.js` exists on router | -| ubus methods missing | rpcd didn't pick up the ucode backend | `/etc/init.d/rpcd restart`; check `logread -e rpcd` for errors | -| ACL denied when calling ubus from LuCI | ACL JSON file missing or wrong format | Verify `/usr/share/rpcd/acl.d/luci-app-zt-gateway.json` and restart rpcd | -| `luci.mk` not found during build | Building outside the LuCI feed | Use `include $(TOPDIR)/feeds/luci/luci.mk` (absolute) instead of `include ../../luci.mk` (relative) | -| JS view loads but is blank | Import path errors in JS | Open browser dev console; LuCI JS modules use `luci.view`, `luci.form`, etc. | - - -## Local Testing with Docker (macvlan) - -This section covers how to test `luci-app-zt-gateway` locally using the official `openwrt/rootfs` Docker image with macvlan networking. This gives you a full OpenWRT environment with real `ip rule`, `iptables`, `conntrack`, and rpcd — no physical router required. - -### Test architecture - -``` -┌──────────────────────────────────────────────────────────────────┐ -│ Docker host (your dev machine) │ -│ │ -│ ┌──────────────────────┐ ┌──────────────────────┐ │ -│ │ openwrt-router │ │ wiblan-client │ │ -│ │ (macvlan 10.11.13.1)│ │ (macvlan 10.11.13.10)│ │ -│ │ │ │ │ │ -│ │ - LuCI on :80 │ │ - curl / ping │ │ -│ │ - rpcd + ucode │ │ - traffic exits via │ │ -│ │ - zt-gateway-switch │ │ router's table 100 │ │ -│ │ - br-zt + ip rules │ │ │ │ -│ └──────────┬───────────┘ └───────────┬───────────┘ │ -│ │ │ │ -│ └──────────┬─────────────────┘ │ -│ │ │ -│ zt-gateway-macvlan │ -│ 10.11.13.0/24 subnet │ -│ (macvlan bridge mode) │ -│ │ -│ ┌──────────────────────┐ │ -│ │ zt-gw-amsterdam │ ← Simulates a remote ZeroTier exit │ -│ │ (macvlan 10.11.12.3) │ node with NAT + ip_forward │ -│ │ │ │ -│ │ - ip_forward=1 │ │ -│ │ - iptables MASQ │ │ -│ └──────────────────────┘ │ -│ │ -│ zt-exit-macvlan │ -│ 10.11.12.0/24 subnet │ -└──────────────────────────────────────────────────────────────────┘ -``` - -The key insight: we use **two macvlan networks** to simulate the two network segments the real router has: -1. **zt-gateway-macvlan** (`10.11.13.0/24`) — represents the WIBLAN subnet. The router sits at `.1`, client at `.10`. -2. **zt-exit-macvlan** (`10.11.12.0/24`) — represents the ZeroTier network where exit gateway nodes live. - -### Step 1: Create the Dockerfile - -Create a Dockerfile that builds a pre-configured OpenWRT image with all dependencies and the zt-gateway package: - -```dockerfile -# Dockerfile.openwrt-dev -FROM openwrt/rootfs:x86-64-24.10.7 - -# Install required packages -RUN opkg update && opkg install \ - luci \ - luci-base \ - ucode \ - conntrack \ - ip-bridge \ - iptables \ - kmod-ipt-conntrack \ - kmod-ipt-connmark \ - kmod-ipt-mark \ - kmod-ipt-extra \ - && rm -rf /var/opkg-lists/* - -# Enable uhttpd -RUN /etc/init.d/uhttpd enable -RUN /etc/init.d/uhttpd start - -# Enable rpcd -RUN /etc/init.d/rpcd enable -RUN /etc/init.d/rpcd start - -# Set root password (required for LuCI login) -RUN echo "root:root" | chpasswd - -EXPOSE 80 - -CMD ["/sbin/init"] -``` - -Build it: - -```sh -docker build -t openwrt-zt-gateway-dev -f Dockerfile.openwrt-dev . -``` - -**Note:** The exact `kmod-*` packages available depend on the OpenWRT version. If `opkg install` fails on kernel modules, skip them — the Docker container's kernel is the host kernel, so the modules are already loaded. What matters is that `iptables -t mangle` and `conntrack` work. Verify with: - -```sh -# Run a quick check after building -docker run --rm --privileged openwrt-zt-gateway-dev sh -c \ - "iptables -t mangle -L && echo 'mangle OK' && conntrack -L 2>/dev/null && echo 'conntrack OK'" -``` - -### Step 2: Create the Docker Compose file - -```yaml -# docker-compose.yml -version: "3.8" - -networks: - # Simulates the WIBLAN client network (10.11.13.0/24) - zt-gateway-lan: - driver: macvlan - driver_opts: - mode: bridge - ipam: - config: - - subnet: "10.11.13.0/24" - gateway: "10.11.13.1" - ip_range: "10.11.13.100/28" - - # Simulates the ZeroTier exit node network (10.11.12.0/24) - zt-exit-net: - driver: macvlan - driver_opts: - mode: bridge - ipam: - config: - - subnet: "10.11.12.0/24" - ip_range: "10.11.12.100/28" - -services: - # The OpenWRT router running luci-app-zt-gateway - openwrt-router: - image: openwrt-zt-gateway-dev - container_name: openwrt-router - privileged: true - networks: - zt-gateway-lan: - ipv4_address: "10.11.13.1" - zt-exit-net: - ipv4_address: "10.11.12.1" - ports: - - "8080:80" - volumes: - # Hot-mount source files for live iteration (key advantage) - - ./luci-app-zt-gateway/htdocs/luci-static/resources/view/zt-gateway/overview.js:/www/luci-static/resources/view/zt-gateway/overview.js - - ./luci-app-zt-gateway/root/usr/share/rpcd/ucode/zt-gateway.uc:/usr/share/rpcd/ucode/zt-gateway.uc - - ./luci-app-zt-gateway/root/usr/sbin/zt-gateway-switch:/usr/sbin/zt-gateway-switch - - ./luci-app-zt-gateway/root/etc/config/zt-gateway:/etc/config/zt-gateway - - ./luci-app-zt-gateway/root/usr/share/rpcd/acl.d/luci-app-zt-gateway.json:/usr/share/rpcd/acl.d/luci-app-zt-gateway.json - - ./luci-app-zt-gateway/root/usr/share/luci/menu.d/luci-app-zt-gateway.json:/usr/share/luci/menu.d/luci-app-zt-gateway.json - cap_add: - - NET_ADMIN - - SYS_ADMIN - sysctls: - - net.ipv4.ip_forward=1 - - net.ipv4.conf.all.send_redirects=0 - command: ["/sbin/init"] - - # A WIBLAN client to test routing through the gateway - wiblan-client: - image: alpine:latest - container_name: wiblan-client - networks: - zt-gateway-lan: - ipv4_address: "10.11.13.10" - cap_add: - - NET_ADMIN - command: ["sleep", "infinity"] - - # Simulates the amsterdam ZeroTier exit gateway (10.11.12.3) - zt-gw-amsterdam: - image: alpine:latest - container_name: zt-gw-amsterdam - networks: - zt-exit-net: - ipv4_address: "10.11.12.3" - cap_add: - - NET_ADMIN - sysctls: - - net.ipv4.ip_forward=1 - command: ["sleep", "infinity"] - - # Simulates the tirunelveli ZeroTier exit gateway (10.11.12.5) - zt-gw-tirunelveli: - image: alpine:latest - container_name: zt-gw-tirunelveli - networks: - zt-exit-net: - ipv4_address: "10.11.12.5" - cap_add: - - NET_ADMIN - sysctls: - - net.ipv4.ip_forward=1 - command: ["sleep", "infinity"] -``` - -### Step 3: Launch and configure the environment - -```sh -# Start all containers -docker compose up -d - -# Wait a few seconds for init, then configure the simulated exit gateways -# These simulate real ZeroTier exit nodes with NAT -``` - -#### Configure the amsterdam exit gateway (10.11.12.3) - -```sh -docker exec -it zt-gw-amsterdam sh -c ' - # Enable forwarding - echo 1 > /proc/sys/net/ipv4/ip_forward - - # Add a default route back to the router so return traffic works - ip route add 10.11.13.0/24 via 10.11.12.1 - - # Add NAT so traffic from WIBLAN clients can reach the internet - # (or just other container networks for testing) - apk add --no-cache iptables - iptables -t nat -A POSTROUTING -s 10.11.13.0/24 -j MASQUERADE - iptables -A FORWARD -s 10.11.13.0/24 -j ACCEPT - iptables -A FORWARD -d 10.11.13.0/24 -m state --state ESTABLISHED,RELATED -j ACCEPT - - # Add a fake "internet" endpoint (just the Docker host gateway) - # In real use, this would be the actual internet - ping -c 1 10.11.12.1 2>/dev/null -' -``` - -#### Configure the tirunelveli exit gateway (10.11.12.5) - -```sh -docker exec -it zt-gw-tirunelveli sh -c ' - echo 1 > /proc/sys/net/ipv4/ip_forward - ip route add 10.11.13.0/24 via 10.11.12.1 - apk add --no-cache iptables - iptables -t nat -A POSTROUTING -s 10.11.13.0/24 -j MASQUERADE - iptables -A FORWARD -s 10.11.13.0/24 -j ACCEPT - iptables -A FORWARD -d 10.11.13.0/24 -m state --state ESTABLISHED,RELATED -j ACCEPT -' -``` - -#### Configure the OpenWRT router - -```sh -docker exec -it openwrt-router sh -``` - -Inside the router container: - -```sh -# 1. Create the br-zt bridge (simulates the ZeroTier bridge) -ip link add name br-zt type bridge -ip link set br-zt up - -# Assign the ZT-exit IP to the bridge so the router can reach 10.11.12.0/24 -ip addr add 10.11.12.1/24 dev br-zt - -# 2. Set up policy routing (same as the real router) - -# Host route to the current gateway (ARP fix) -ip route replace 10.11.12.3 dev br-zt - -# Default route via the current gateway in table 100 -ip route replace default via 10.11.12.3 dev br-zt table 100 - -# Route WIBLAN subnet via br-zt in mwan3 table (for return traffic) -ip route replace 10.11.13.0/24 dev br-zt table 1 - -# Source-based routing rule: WIBLAN traffic → table 100 -ip rule add from 10.11.13.0/24 table 100 priority 100 - -# 3. Verify routing -ip route show table 100 -# Expected: default via 10.11.12.3 dev br-zt - -ip rule show -# Expected: 100: from 10.11.13.0/24 lookup 100 - -# 4. Test from the router itself -ping -c 2 10.11.12.3 # Should reach amsterdam gateway -``` - -#### Configure the WIBLAN client - -```sh -docker exec -it wiblan-client sh -c ' - # Add a route to the router for the ZT subnet - ip route add 10.11.12.0/24 via 10.11.13.1 - - # Set the OpenWRT router as default gateway - ip route add default via 10.11.13.1 - - # Test connectivity - ping -c 2 10.11.12.3 -' -``` - -### Step 4: Install and test the luci-app-zt-gateway package - -#### Option A: The package is already volume-mounted - -If you used the `volumes:` section in `docker-compose.yml`, the files are already in place. Just restart services: - -```sh -docker exec -it openwrt-router sh -c ' - chmod +x /usr/sbin/zt-gateway-switch - /etc/init.d/rpcd restart - rm -rf /tmp/luci-indexcache /tmp/luci-modulecache -' -``` - -#### Option B: Install the .ipk - -If you built an `.ipk`, copy it into the container: - -```sh -docker cp luci-app-zt-gateway_1.0.0-1_all.ipk openwrt-router:/tmp/ -docker exec -it openwrt-router opkg install /tmp/luci-app-zt-gateway_1.0.0-1_all.ipk -``` - -#### Verify the rpcd backend - -```sh -docker exec -it openwrt-router ubus list zt-gateway.* -# Expected: -# zt-gateway.status -# zt-gateway.switch -# zt-gateway.health -# zt-gateway.drain_status -# zt-gateway.cancel_drain - -docker exec -it openwrt-router ubus call zt-gateway status -``` - -#### Access the LuCI UI - -Open http://localhost:8080 in your browser. Log in with `root` / `root`. Navigate to **Services → ZeroTier Exit Gateway**. - -### Step 5: Run the test scenarios - -#### Force switch test - -```sh -# 1. Verify current gateway is amsterdam (10.11.12.3) -docker exec -it openwrt-router ip route show table 100 -# Expected: default via 10.11.12.3 dev br-zt - -# 2. Trigger a force switch to tirunelveli via the CLI script -docker exec -it openwrt-router zt-gateway-switch 10.11.12.5 force - -# 3. Verify the route changed -docker exec -it openwrt-router ip route show table 100 -# Expected: default via 10.11.12.5 dev br-zt - -# 4. Verify conntrack was flushed -docker exec -it openwrt-router conntrack -L -s 10.11.13.0/24 2>/dev/null -# Expected: empty (all entries flushed) - -# 5. Verify from the client side -docker exec -it wiblan-client ping -c 2 10.11.12.5 -``` - -#### Graceful drain test - -```sh -# 1. Start a long-running connection from the client -docker exec -d wiblan-client ping 10.11.12.3 # background ping - -# 2. Trigger a graceful switch (this will be done from the LuCI UI -# or via ubus call) -docker exec -it openwrt-router zt-gateway-switch 10.11.12.5 graceful 300 - -# 3. Verify both routing tables exist during drain -docker exec -it openwrt-router ip route show table 100 -# Expected: default via 10.11.12.5 dev br-zt (new gateway) - -docker exec -it openwrt-router ip route show table 101 -# Expected: default via 10.11.12.3 dev br-zt (old gateway, drain) - -# 4. Verify the fwmark rule exists -docker exec -it openwrt-router ip rule show -# Expected: 99: from all fwmark 0x100 lookup 101 - -# 5. Verify the mangle rules -docker exec -it openwrt-router iptables -t mangle -L -v - -# 6. Watch drain progress -docker exec -it openwrt-router conntrack -L -m 0x100 2>/dev/null | wc -l - -# 7. Wait for drain to complete (or kill the background ping to speed it up) -docker exec -it wiblan-client pkill ping - -# 8. After drain completes, verify cleanup -docker exec -it openwrt-router ip route show table 101 -# Expected: empty (drain table removed) -docker exec -it openwrt-router ip rule show -# Expected: no fwmark 0x100 rule -``` - -#### Switch to unreachable gateway test - -```sh -# 1. Stop the tirunelveli gateway to simulate it being down -docker stop zt-gw-tirunelveli - -# 2. Try to switch to it (should fail pre-flight) -docker exec -it openwrt-router zt-gateway-switch 10.11.12.5 force -# Expected: "Gateway unreachable" error, no route changes - -# 3. Verify routes unchanged -docker exec -it openwrt-router ip route show table 100 - -# 4. Restart for subsequent tests -docker start zt-gw-tirunelveli -``` - -#### Drain timeout test - -```sh -# 1. Start a persistent connection -docker exec -d wiblan-client ping 10.11.12.3 - -# 2. Graceful switch with short timeout (30s) -docker exec -it openwrt-router zt-gateway-switch 10.11.12.5 graceful 30 - -# 3. Watch the drain monitor — it will hit the 30s timeout -# because the ping keeps conntrack entries alive -docker exec -it openwrt-router conntrack -L -m 0x100 2>/dev/null | wc -l - -# 4. After 30s, should auto-fallback to force switch -docker exec -it openwrt-router ip route show table 100 -# Expected: default via 10.11.12.5 dev br-zt (force applied) -docker exec -it openwrt-router ip route show table 101 -# Expected: empty (drain cleaned up by force) - -# 5. Clean up -docker exec -it wiblan-client pkill ping -``` - -### Step 6: Iterate on fixes - -The key advantage of the Docker setup is the **volume mounts** in the compose file. When you edit a source file on the host, it's immediately reflected inside the container. - -#### Live-editing workflow - -```sh -# 1. Edit a source file on the host -vim luci-app-zt-gateway/root/usr/sbin/zt-gateway-switch - -# 2. Re-test inside the container (file is already updated via volume mount) -docker exec -it openwrt-router zt-gateway-switch 10.11.12.5 force - -# 3. Check results -docker exec -it openwrt-router ip route show table 100 -``` - -For LuCI JS changes, you need to clear the cache: - -```sh -# Edit JS on host -vim luci-app-zt-gateway/htdocs/luci-static/resources/view/zt-gateway/overview.js - -# Clear cache inside container -docker exec -it openwrt-router rm -rf /tmp/luci-indexcache /tmp/luci-modulecache - -# Refresh browser -``` - -For ucode backend changes: - -```sh -# Edit ucode on host -vim luci-app-zt-gateway/root/usr/share/rpcd/ucode/zt-gateway.uc - -# Restart rpcd to reload the backend -docker exec -it openwrt-router /etc/init.d/rpcd restart -``` - -#### Quick-snapshot the exact container state - -If you want to save a known-good state before making a risky change: - -```sh -# Save container state as a Docker image -docker commit openwrt-router openwrt-zt-gateway-snapshot:good - -# ... make changes, break things ... - -# Restore from snapshot -docker compose down -docker run -d --name openwrt-router \ - --privileged \ - -p 8080:80 \ - openwrt-zt-gateway-snapshot:good -``` - -#### Watch logs in real-time - -```sh -# OpenWRT system log -docker exec -it openwrt-router logread -f - -# rpcd errors specifically -docker exec -it openwrt-router logread -f -e rpcd - -# conntrack events in real-time -docker exec -it openwrt-router conntrack -E -``` - -#### Run the full test suite in one shot - -```sh -#!/bin/bash -# test-zt-gateway.sh — Run all test scenarios in Docker -set -e - -ROUTER="openwrt-router" -CLIENT="wiblan-client" -GW_AMS="10.11.12.3" -GW_TIR="10.11.12.5" - -echo "=== Test 1: Force switch amsterdam → tirunelveli ===" -docker exec $ROUTER zt-gateway-switch $GW_TIR force -ROUTE=$(docker exec $ROUTER ip route show table 100 | grep default) -echo " Route after force: $ROUTE" -echo "$ROUTE" | grep -q "$GW_TIR" && echo " PASS" || echo " FAIL" - -echo "=== Test 2: Force switch tirunelveli → amsterdam ===" -docker exec $ROUTER zt-gateway-switch $GW_AMS force -ROUTE=$(docker exec $ROUTER ip route show table 100 | grep default) -echo " Route after force: $ROUTE" -echo "$ROUTE" | grep -q "$GW_AMS" && echo " PASS" || echo " FAIL" - -echo "=== Test 3: Switch to unreachable gateway ===" -docker stop zt-gw-tirunelveli -if docker exec $ROUTER zt-gateway-switch $GW_TIR force 2>&1 | grep -qi "unreachable\|fail\|error"; then - echo " PASS (blocked as expected)" -else - echo " FAIL (should have been blocked)" -fi -docker start zt-gw-tirunelveli -sleep 3 - -echo "=== Test 4: Verify client can reach gateway ===" -docker exec $CLIENT ping -c 2 $GW_AMS && echo " PASS" || echo " FAIL" - -echo "=== Test 5: Verify ubus backend ===" -docker exec $ROUTER ubus call zt-gateway status | grep -q "amsterdam" && echo " PASS" || echo " FAIL" - -echo "=== All tests complete ===" -``` - -### Important limitations of the Docker test environment - -| Limitation | Impact | Mitigation | -|---|---|---| -| Docker shares the host kernel | `kmod-*` packages from opkg won't load (wrong kernel version) | Use host kernel modules; verify with `lsmod`/`iptables -t mangle -L` | -| `br-zt` is simulated, not a real ZeroTier bridge | No actual ZT membership, no `ztk4jpk77j` interface | Manually create `br-zt` bridge and assign IPs; skip hotplug scripts | -| `conntrack -D -s` may behave differently | Docker's conntrack table includes host traffic too | Use `-s` and `-d` filters; or run `conntrack -F` for a clean slate | -| No real `rc.local` / hotplug execution on boot | Persistence scripts can't be tested end-to-end | Test the switch script logic; verify `sed` replacements on the files separately | -| `iptables -t mangle` requires `NET_ADMIN` | Container must run `--privileged` or `--cap-add=NET_ADMIN` | Already configured in compose file | -| Macvlan host isolation | Host can't ping macvlan containers by default | Use a macvlan shim interface (see below) or just `docker exec` | -| No real ZeroTier Central API | Can't test the future ZT API integration | Mock the API calls or skip this test | - -### Fixing the macvlan host-isolation issue - -By default, the Docker host **cannot** communicate directly with containers on a macvlan network. If you need to ping/curl the router or client from your host machine (not just via `docker exec`), create a macvlan shim interface: - -```sh -# Create a macvlan interface on the host that bridges to the same parent -# Replace eth0 with your host's primary interface -sudo ip link add link eth0 name mv-shim type macvlan mode bridge -sudo ip addr add 10.11.13.254/32 dev mv-shim -sudo ip link set mv-shim up - -# Add a route to the WIBLAN subnet via the shim -sudo ip route add 10.11.13.0/24 dev mv-shim - -# Now you can reach the containers from host -ping 10.11.13.1 # the router -curl http://10.11.13.1 # LuCI directly (skip the port mapping) - -# For the ZT exit subnet -sudo ip link add link eth0 name mv-exit-shim type macvlan mode bridge -sudo ip addr add 10.11.12.254/32 dev mv-exit-shim -sudo ip link set mv-exit-shim up -sudo ip route add 10.11.12.0/24 dev mv-exit-shim -``` - -Clean up when done: - -```sh -sudo ip link del mv-shim -sudo ip link del mv-exit-shim -``` - -### Teardown - -```sh -# Stop and remove all containers -docker compose down - -# Remove the shim interfaces (if created) -sudo ip link del mv-shim 2>/dev/null -sudo ip link del mv-exit-shim 2>/dev/null - -# Remove the custom images (optional) -docker rmi openwrt-zt-gateway-dev openwrt-zt-gateway-snapshot:good -``` \ No newline at end of file diff --git a/Dockerfile.openwrt b/Dockerfile.openwrt new file mode 100644 index 0000000..b155626 --- /dev/null +++ b/Dockerfile.openwrt @@ -0,0 +1,29 @@ +FROM openwrt/rootfs:latest + +# Install LuCI, bootstrap theme, and dependencies. +# Build this image with podman (NOT docker buildx) because Podman-backed Docker +# seccomp blocks uclient-fetch syscalls at a level BuildKit cannot override. +RUN mkdir -p /etc/apk/repositories.d /var/lock /var/run /www && \ + printf \ +'https://downloads.openwrt.org/snapshots/targets/x86/64/packages/packages.adb\n\ +https://downloads.openwrt.org/snapshots/packages/x86_64/base/packages.adb\n\ +https://downloads.openwrt.org/snapshots/packages/x86_64/luci/packages.adb\n\ +https://downloads.openwrt.org/snapshots/packages/x86_64/packages/packages.adb\n\ +https://downloads.openwrt.org/snapshots/packages/x86_64/routing/packages.adb\n' \ + > /etc/apk/repositories.d/distfeeds.list && \ + apk add --no-cache --allow-untrusted \ + luci-base luci-compat luci-mod-admin-full luci-proto-ppp \ + luci-theme-bootstrap \ + uhttpd uhttpd-mod-ubus ucode-mod-lua \ + ca-certificates curl + +# Application files (self-contained image; compose may override with volumes) +COPY root/usr/sbin/zt-gateway-switch /usr/sbin/zt-gateway-switch +RUN chmod +x /usr/sbin/zt-gateway-switch && \ + rm -f /var/run/zt-gateway-drain.* /tmp/zt-gw-stderr /tmp/zt-gw-out-* + +COPY docker/openwrt-luci-entrypoint.sh /entrypoint.sh +RUN chmod +x /entrypoint.sh + +EXPOSE 80 +ENTRYPOINT ["/entrypoint.sh"] diff --git a/docker-compose.yml b/docker-compose.yml index 525d382..4772f4d 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,4 +1,37 @@ services: + # OpenWRT LuCI container — web UI for Playwright E2E tests + # Uses the pre-imported zt-gateway-luci:dev image (OpenWrt 25.12.4 rootfs + # with LuCI + zt-gateway application overlay). No docker build needed. + openwrt-luci: + image: zt-gateway-luci:dev + container_name: openwrt-luci + privileged: true + cap_add: + - NET_ADMIN + - SYS_ADMIN + ports: + - "8080:80" + networks: + zt-exit-net: + ipv4_address: "10.99.12.2" + environment: + ZTG_SKIP_PERSIST: "1" + ZTG_PING_IFACE: "br-zt" + ZTG_WIBLAN_CIDR: "10.99.13.0/24" + entrypoint: ["/entrypoint.sh"] + volumes: + # Overlay fixed application files (rpcd backend, LuCI view, config, menu, acl) + - ./root/usr/share/rpcd/ucode/zt-gateway.uc:/usr/share/rpcd/ucode/zt-gateway.uc:ro + - ./root/usr/share/rpcd/ucode/system.uc:/usr/share/rpcd/ucode/system.uc:ro + - ./root/usr/share/ucode/luci/runtime.uc:/usr/share/ucode/luci/runtime.uc:ro + - ./root/usr/share/luci/menu.d/luci-app-zt-gateway.json:/usr/share/luci/menu.d/luci-app-zt-gateway.json:ro + - ./root/usr/share/rpcd/acl.d/luci-app-zt-gateway.json:/usr/share/rpcd/acl.d/luci-app-zt-gateway.json:ro + # Stage host config so the entrypoint can copy it into the writable overlay. + - ./root/etc/config:/host-config:ro + - ./htdocs/luci-static/resources/view/zt-gateway/overview.js:/www/luci-static/resources/view/zt-gateway/overview.js:ro + - ./docker/openwrt-luci-entrypoint.sh:/entrypoint.sh:ro + + # Arch-based router — CLI routing tests openwrt-router: build: context: . diff --git a/docker/build-base-image.sh b/docker/build-base-image.sh new file mode 100755 index 0000000..6ac2474 --- /dev/null +++ b/docker/build-base-image.sh @@ -0,0 +1,31 @@ +#!/bin/sh +# Build the OpenWrt base image with LuCI packages installed. +# Docker buildkit cannot build openwrt/rootfs directly because buildkit's +# seccomp profile blocks uclient-fetch (wget) syscalls used by apk 3.0. +# We use 'docker run --security-opt seccomp=unconfined' instead. +set -eu + +CONTAINER_NAME="zt-gateway-luci-build-tmp" +BASE_IMAGE="zt-gateway-luci:base" + +docker rm -f "$CONTAINER_NAME" 2>/dev/null || true + +docker run -d --name "$CONTAINER_NAME" \ + --security-opt seccomp=unconfined \ + openwrt/rootfs:latest sleep infinity + +# Setup repos and install packages +docker exec "$CONTAINER_NAME" mkdir -p /etc/apk/repositories.d /var/lock /var/run + +docker exec "$CONTAINER_NAME" sh -c ' +printf "https://downloads.openwrt.org/releases/25.12.4/targets/x86/64/packages/packages.adb\nhttps://downloads.openwrt.org/releases/25.12.4/packages/x86_64/base/packages.adb\nhttps://downloads.openwrt.org/releases/25.12.4/packages/x86_64/luci/packages.adb\nhttps://downloads.openwrt.org/releases/25.12.4/packages/x86_64/packages/packages.adb\nhttps://downloads.openwrt.org/releases/25.12.4/packages/x86_64/routing/packages.adb\n" > /etc/apk/repositories.d/distfeeds.list +apk add --no-cache --allow-untrusted \ + luci-base luci-mod-admin-full luci-proto-ppp \ + uhttpd uhttpd-mod-ubus ucode-mod-lua \ + ca-certificates curl +' + +docker commit "$CONTAINER_NAME" "$BASE_IMAGE" +docker rm -f "$CONTAINER_NAME" + +echo "Built $BASE_IMAGE" diff --git a/docker/openwrt-luci-entrypoint.sh b/docker/openwrt-luci-entrypoint.sh new file mode 100755 index 0000000..d79b3ca --- /dev/null +++ b/docker/openwrt-luci-entrypoint.sh @@ -0,0 +1,95 @@ +#!/bin/sh +# openwrt-luci-entrypoint.sh +# +# Starts ubusd, rpcd, configures the br-zt bridge, seeds UCI state, +# and runs uhttpd in the foreground (the only foreground process). +set -eu + +WIBLAN_CIDR="${ZTG_WIBLAN_CIDR:-10.99.13.0/24}" + +# ── 1. Start ubusd first (required for rpcd + uci) ────────────────────────── +rm -f /var/run/ubus/ubus.sock /tmp/run/ubus/ubus.sock +rm -f /tmp/lock/procd_*.lock +mkdir -p /var/run/ubus +/sbin/ubusd & +UBUSD_PID=$! + +# Wait for ubus socket (up to 10s) +for _i in 1 2 3 4 5 6 7 8 9 10; do + if [ -S /var/run/ubus/ubus.sock ]; then break; fi + sleep 1 +done + +if ! ubus list >/dev/null 2>&1; then + echo "ERROR: ubusd did not start in time" >&2 + kill $UBUSD_PID 2>/dev/null || true + exit 1 +fi + +# ── 2. Start rpcd ──────────────────────────────────────────────────────────── +/sbin/rpcd & + +for _i in 1 2 3 4 5; do + if ubus call system board >/dev/null 2>&1; then break; fi + sleep 1 +done + +# ── 3. Seed root password (for LuCI login) ─────────────────────────────────── +if [ -f /etc/shadow ]; then + sed -i 's|^root:.*|root::0:0:99999:7:::|' /etc/shadow +fi + +# ── 4a. Seed zt-gateway config from host staging if present ───────────────── +if [ -f /host-config/zt-gateway ] && [ ! -f /etc/config/zt-gateway ]; then + cp /host-config/zt-gateway /etc/config/zt-gateway +fi + +# ── 4. Configure gateway switch mode to initial state ──────────────────────── +if command -v uci >/dev/null 2>&1; then + uci set zt-gateway.global.active_gateway='amsterdam' 2>/dev/null || true + uci set zt-gateway.global.switch_mode='force' 2>/dev/null || true + uci set zt-gateway.global.drain_timeout='600' 2>/dev/null || true + uci commit zt-gateway 2>/dev/null || true +fi + +echo "[entrypoint] ubusd + rpcd + config done" + +# ── 5. Network setup (bridge + routing) ────────────────────────────────────── +# The target bridge device (used by zt-gateway-switch and health checks). +BRIDGE="${ZTG_BRIDGE:-br-zt}" +# Detect the interface that currently holds the 10.99.12.x address. +SOURCE_IFACE=$(ip -o -4 addr show 2>/dev/null \ + | awk '$4 ~ /^10\.99\.12\./ {print $2; exit}') + +if [ -z "$SOURCE_IFACE" ]; then + echo "WARNING: no 10.99.12.x interface found (expected zt-exit-net); using Docker bridge" + echo 1 > /proc/sys/net/ipv4/ip_forward + echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects 2>/dev/null || true +else + ip link add name "$BRIDGE" type bridge 2>/dev/null || true + ip link set "$BRIDGE" up + + ADDR=$(ip -o -4 addr show dev "$SOURCE_IFACE" \ + | awk '$4 ~ /^10\.99\.12\./ {print $4; exit}') + if [ -n "$ADDR" ]; then + ip addr del "$ADDR" dev "$SOURCE_IFACE" 2>/dev/null || true + ip addr add "$ADDR" dev "$BRIDGE" + ip link set "$SOURCE_IFACE" master "$BRIDGE" + fi + + echo 1 > /proc/sys/net/ipv4/ip_forward + echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects 2>/dev/null || true + echo 0 > /proc/sys/net/ipv4/conf/"$BRIDGE"/send_redirects 2>/dev/null || true + + ip rule del from "$WIBLAN_CIDR" table 100 2>/dev/null || true + ip rule add from "$WIBLAN_CIDR" table 100 priority 100 + ip route replace default via 10.99.12.3 dev "$BRIDGE" 2>/dev/null || true + ip route replace "$WIBLAN_CIDR" dev "$BRIDGE" table 1 2>/dev/null || true + + echo "[entrypoint] networking: SOURCE=$SOURCE_IFACE BRIDGE=$BRIDGE ADDR=$ADDR" +fi + +echo "[entrypoint] starting uhttpd..." + +# ── 6. Run uhttpd in foreground (replaces shell) ───────────────────────────── +exec uhttpd -f -p 80 -h /www -u /ubus -a -r "OpenWRT-Test-Router" diff --git a/docs/PROGRESS.md b/docs/PROGRESS.md new file mode 100644 index 0000000..bf944d6 --- /dev/null +++ b/docs/PROGRESS.md @@ -0,0 +1,97 @@ +# Progress: Playwright E2E Tests for luci-app-zt-gateway + +## Date: 2026-06-26/2026-06-27 + +## Completed Fixes + +### 1. `login()` Helper (`e2e/utils/auth.ts`) +- **Root cause**: LuCI login form lives inside `