From f6bf6e7cd4259daa2e805a696cd7a60039aa8477 Mon Sep 17 00:00:00 2001 From: Malar Invention Date: Fri, 19 Jun 2026 00:03:09 +0530 Subject: [PATCH] docs: initial plan for ZeroTier exit gateway switcher MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Design: .omp/plans/zerotier-gateway-switching.md — LuCI app + rpcd/ucode backend + zt-gateway-switch shell script for force / graceful-drain gateway switching on OpenWRT. --- .gitignore | 28 + .omp/plans/zerotier-gateway-switching.md | 1258 ++++++++++++++++++++++ README.md | 38 + 3 files changed, 1324 insertions(+) create mode 100644 .gitignore create mode 100644 .omp/plans/zerotier-gateway-switching.md create mode 100644 README.md diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..b80ce30 --- /dev/null +++ b/.gitignore @@ -0,0 +1,28 @@ +# OpenWRT / LuCI build artifacts +bin/ +tmp/ +.build/ +.config +.config.old +feeds.conf.default +feeds.conf +*.ipk +*.ipk.opk + +# SDK / build cache +dl/ +staging_dir*/ +build_dir*/ + +# Translations (compiled) +*.po~ +*.pot + +# Common OS / editor files +.DS_Store +Thumbs.db +*.swp +*.swo +*~ +.vscode/ +.idea/ diff --git a/.omp/plans/zerotier-gateway-switching.md b/.omp/plans/zerotier-gateway-switching.md new file mode 100644 index 0000000..d96d941 --- /dev/null +++ b/.omp/plans/zerotier-gateway-switching.md @@ -0,0 +1,1258 @@ +# Plan: luci-app-zt-gateway — ZeroTier Exit Gateway Switcher + +## Goal + +A LuCI app for OpenWRT that lets users switch which remote ZeroTier node acts as the internet exit gateway for WIBLAN clients (`10.11.13.0/24`). Users select a region/gateway, choose a switch mode, and the app reconfigures routing with zero or minimal disruption. + +## Existing Architecture (constraints) + +- **Router**: OpenWRT, `192.168.13.1` +- **ZT network**: `e3918db48378cb6a`, interface `ztk4jpk77j`, bridge `br-zt` +- **Current gateway**: `10.11.12.3` (ocirosea641, amsterdam) +- **WIBLAN clients**: `10.11.13.0/24` routed via `ip rule` priority 100 → table 100 +- **Table 100**: `default via dev br-zt` +- **Host route**: ` dev br-zt` (ARP fix — gateway IP not directly reachable via bridge) +- **mwan3**: uses table 1; requires `10.11.13.0/24 dev br-zt table 1` for return traffic +- **Hotplug**: `/etc/hotplug.d/net/99-zerotier-bridge` adds `ztk4jpk77j` to `br-zt`, removes conflicting routes, adds table 1 route +- **Conntrack**: `conntrack` package available; `conntrack -D -s 10.11.13.0/24` flushes WIBLAN entries + +## Two Switch Modes + +### 1. Force Switch (instant, ~1-2s disruption) + +All existing connections break immediately. Clients auto-reconnect through the new gateway. + +**Steps:** +1. Pre-flight: ping new gateway over ZT, abort if unreachable +2. Atomically replace host route + table 100 default route +3. Flush conntrack for `10.11.13.0/24` +4. Update hotplug script + rc.local with new gateway IP +5. Update UCI network routes for persistence +6. Verify: `ip route get 8.8.8.8 from 10.11.13.1 table 100` + +**User impact:** Active downloads/video calls drop and reconnect. Web pages refresh. SSH sessions die. + +### 2. Graceful Drain (zero disruption, complex) + +Existing connections continue on the old gateway until they naturally close. New connections go through the new gateway immediately. Uses fwmark + dual routing table. + +**Steps:** +1. Pre-flight: ping new gateway over ZT, abort if unreachable +2. Create drain table (101) with `default via dev br-zt` +3. Add iptables mangle rules to mark all ESTABLISHED connections from WIBLAN with fwmark `0x100` +4. Add `ip rule add fwmark 0x100 table 101 priority 99` (before the existing priority-100 src rule) +5. Update table 100 default to new gateway +6. Update host route to new gateway +7. Start drain monitor: count conntrack entries with mark; when zero → clean up mangle rules, drain table, fwmark rule +8. Update hotplug script + rc.local + UCI network routes for persistence +9. If drain doesn't complete within configurable timeout (default 10 min), auto-fallback to force switch + +**Key insight:** fwmark 0x100 at priority 99 matches BEFORE the src `10.11.13.0/24` → table 100 rule. So marked (existing) connections go to table 101 (old gateway), while unmarked (new) connections go to table 100 (new gateway). + +**User impact:** Zero disruption. Active connections keep working. New connections get new gateway latency/exit point. + +**Conntrack marking detail:** +```sh +# Restore mark from conntrack for returning packets +iptables -t mangle -A PREROUTING -i br-zt -s 10.11.13.0/24 -m conntrack --ctstate ESTABLISHED,RELATED -j CONNMARK --restore-mark + +# Mark ESTABLISHED packets that don't have a mark yet +iptables -t mangle -A PREROUTING -i br-zt -s 10.11.13.0/24 -m conntrack --ctstate ESTABLISHED,RELATED -m mark --mark 0 -j MARK --set-mark 0x100 + +# Save packet mark back to conntrack +iptables -t mangle -A PREROUTING -i br-zt -s 10.11.13.0/24 -j CONNMARK --save-mark +``` + +**Drain monitor:** Periodic check via `conntrack -L -m 0x100 2>/dev/null | wc -l`. When count is 0, clean up. + +## File Structure + +``` +luci-app-zt-gateway/ +├── Makefile +├── htdocs/ +│ └── luci-static/ +│ └── resources/ +│ └── view/ +│ └── zt-gateway/ +│ └── overview.js # Main UI view +├── root/ +│ ├── etc/ +│ │ └── config/ +│ │ └── zt-gateway # UCI config: gateway registry + active state +│ ├── usr/ +│ │ ├── share/ +│ │ │ ├── acl.d/ +│ │ │ │ └── luci-app-zt-gateway.json # ACL permissions +│ │ │ ├── luci/ +│ │ │ │ └── menu.d/ +│ │ │ │ └── luci-app-zt-gateway.json # Menu entry +│ │ │ └── rpcd/ +│ │ │ └── ucode/ +│ │ │ └── zt-gateway.uc # rpcd backend: switch, status, health, drain +│ │ └── sbin/ +│ │ └── zt-gateway-switch # Shell script: actual switch logic (called by rpcd) +``` + +## UCI Config Schema (`/etc/config/zt-gateway`) + +``` +config global 'global' + option active_gateway '' # Region key of active gateway + option switch_mode 'force' # 'force' or 'graceful' + option drain_timeout '600' # Graceful drain timeout in seconds (default 10 min) + option health_interval '60' # Health check interval in seconds + option auto_failback '0' # Auto-failback to default on gateway down + +config gateway + option region 'amsterdam' + option label 'Amsterdam (ocirosea641)' + option ip '10.11.12.3' + option default '1' # Default gateway on boot + option health_check 'ping' # 'ping' | 'tcp' | 'none' + +config gateway + option region 'tirunelveli' + option label 'Tirunelveli (rpi1000)' + option ip '10.11.12.5' + option default '0' + option health_check 'ping' + +config gateway + option region 'bangalore' + option label 'Bangalore (sensecap-m4)' + option ip '10.11.12.4' + option default '0' + option health_check 'ping' # Not yet joined to ZT network; assign IP after joining +``` + +## rpcd Backend (`/usr/share/rpcd/ucode/zt-gateway.uc`) + +Exposes these ubus methods: + +| Method | Args | Returns | Description | +|---|---|---|---| +| `status` | — | `{ active_gateway, gateways: [...], drain_status? }` | Current state + all configured gateways | +| `switch` | `{ region, mode }` | `{ success, message }` | Execute gateway switch | +| `health` | `{ region }` | `{ reachable, latency_ms }` | Ping a gateway via ZT | +| `drain_status` | — | `{ active, remaining_connections }` | Graceful drain progress | +| `cancel_drain` | — | `{ success }` | Cancel graceful drain, force switch to new gateway | + +## Shell Script (`/usr/sbin/zt-gateway-switch`) + +The rpcd backend calls this script for the actual system changes. Separating it from ucode makes it testable from SSH. + +``` +Usage: zt-gateway-switch [drain_timeout] + +Modes: + force - Instant switch + conntrack flush + graceful - fwmark drain with timeout fallback to force +``` + +**Force switch logic:** +1. Validate new gateway IP reachable: `ping -c 2 -W 3 ` +2. `ip route replace dev br-zt` (host route) +3. `ip route replace default via dev br-zt table 100` +4. `conntrack -D -s 10.11.13.0/24 2>/dev/null` +5. Update `/etc/hotplug.d/net/99-zerotier-bridge` with new gateway IP +6. Update `/etc/rc.local` with new gateway IP +7. Update UCI network routes: `zt_gateway_host`, `zt_gateway_default` +8. `uci commit network` +9. Verify new route + +**Graceful switch logic:** +1. Validate new gateway IP reachable +2. Read current gateway from UCI +3. Create table 101: `ip route add default via dev br-zt table 101` +4. Add mangle rules (fwmark 0x100 for ESTABLISHED) +5. Add `ip rule add fwmark 0x100 table 101 priority 99` +6. Update table 100 default + host route to new gateway +7. Start drain monitor loop: + - Every 5s: `conntrack -L -m 0x100 2>/dev/null | wc -l` + - If count == 0: clean up mangle rules, fwmark rule, table 101 → done + - If timeout exceeded: force flush + clean up → done +8. Update hotplug, rc.local, UCI + +**Cleanup function** (called after drain completes or on cancel): +1. `iptables -t mangle -D ` +2. `ip rule del fwmark 0x100 table 101 priority 99` +3. `ip route del default dev br-zt table 101` + +## LuCI Frontend (`overview.js`) + +### Layout + +``` +┌─────────────────────────────────────────────────┐ +│ ZeroTier Exit Gateway │ +├─────────────────────────────────────────────────┤ +│ │ +│ Active Gateway: amsterdam (10.11.12.3) ● UP │ +│ Switch Mode: [Force v] │ +│ Drain Timeout: 600s (shown if graceful) │ +│ │ +│ ┌─────────────────────────────────────────────┐│ +│ │ Region │ Label │ IP ││ +│ ├─────────────┼──────────────────┼─────────────┤│ +│ │ ● amsterdam │ Amsterdam │ 10.11.12.3 ││ +│ │ ○ tirunelveli│ Tirunelveli │ 10.11.12.5 ││ +│ │ ○ bangalore │ Bangalore │ 10.11.12.4 ││ +│ └─────────────┴──────────────────┴─────────────┘│ +│ │ +│ [Switch to Selected] │ +│ │ +│ ── Status ── │ +│ Drain progress: 14 connections remaining │ +│ [Cancel Drain → Force Switch] │ +│ │ +└─────────────────────────────────────────────────┘ +``` + +### JavaScript architecture + +- Form.Map tied to `zt-gateway` UCI config for settings section (switch mode, drain timeout) +- Radio-button list for gateway selection (reads `config gateway` sections) +- Custom button handler for "Switch to Selected" — calls `zt-gateway.switch` RPC +- Polling for drain status when graceful switch is in progress +- Health status indicator (green/red dot) per gateway, refreshed every `health_interval` + +### Key imports +```js +'use strict'; +import { view } from 'luci.view'; +import { form } from 'luci.form'; +import { rpc } from 'luci.rpc'; +import { ui } from 'luci.ui'; +import { dom } from 'luci.dom'; +import { Poll } from 'luci.poll'; +``` + +## Persistence Model + +The gateway IP appears in **five places** on the router. The switch script updates them all atomically: + +| Location | What | How updated | +|---|---|---| +| Kernel: host route | ` dev br-zt` | `ip route replace` | +| Kernel: table 100 default | `default via dev br-zt table 100` | `ip route replace` | +| UCI: `/etc/config/network` | `zt_gateway_host` + `zt_gateway_default` routes | `uci set` + `uci commit` | +| Hotplug: `99-zerotier-bridge` | host route + table 100 route + table 1 return | `sed` replace | +| Boot: `/etc/rc.local` | all three routes | `sed` replace | + +The hotplug and rc.local currently hardcode `10.11.12.3`. The script replaces these with the new gateway IP. A cleaner future approach would be to have the hotplug/rc.local read from UCI, but that's a refactoring step. + +## Health Check + +Default method: `ping -c 2 -W 3 ` over the ZT network. + +- Runs every `health_interval` seconds via cron or the LuCI polling mechanism +- If active gateway becomes unreachable AND `auto_failback='1'`: switch to the `default='1'` gateway +- Health status displayed in UI with latency + +## Rollback / Safety + +- Pre-flight: refuse to switch if new gateway is unreachable +- Graceful drain timeout: if drain doesn't complete, auto-fallback to force +- If switch script fails partway: the host route + table 100 are updated atomically; worst case the old conntrack entries persist and time out naturally +- Manual recovery: `uci set zt-gateway.global.active_gateway=''` + run `/usr/sbin/zt-gateway-switch force` + +## Files to Create + +| File | Purpose | +|---|---| +| `luci-app-zt-gateway/Makefile` | OpenWRT build system | +| `luci-app-zt-gateway/root/etc/config/zt-gateway` | UCI config skeleton | +| `luci-app-zt-gateway/root/usr/share/acl.d/luci-app-zt-gateway.json` | RPC + UCI ACL | +| `luci-app-zt-gateway/root/usr/share/luci/menu.d/luci-app-zt-gateway.json` | Menu entry | +| `luci-app-zt-gateway/root/usr/share/rpcd/ucode/zt-gateway.uc` | rpcd backend | +| `luci-app-zt-gateway/root/usr/sbin/zt-gateway-switch` | Switch script | +| `luci-app-zt-gateway/htdocs/luci-static/resources/view/zt-gateway/overview.js` | Frontend UI | + +## Verification + +1. Install package on router +2. Configure 2+ gateways in `/etc/config/zt-gateway` +3. **Force switch test**: select new region → Force → verify `ip route show table 100` shows new gateway, `curl ifconfig.me` from WIBLAN client shows new exit IP +4. **Graceful switch test**: select new region → Graceful → verify existing SSH on WIBLAN client stays alive, new connections go through new gateway, drain counter decrements +5. **Fail test**: switch to unreachable gateway → verify pre-flight blocks the switch with error message +6. **Drain timeout test**: graceful switch with 30s timeout → verify it falls back to force after timeout +7. **Reboot persistence**: switch gateway → reboot router → verify table 100 still has the new gateway + +## Future: ZeroTier Central API Integration + +Skipping for now. When added, the switch script should: + +1. Call `DELETE /api/network/{networkId}/route/{routeId}` for the old `0.0.0.0/0` managed route +2. Call `POST /api/network/{networkId}/route` with `{ "target": "0.0.0.0/0", "via": "" }` +3. Toggle "Allow Default Route Override" on new gateway node, disable on old + +Requires storing a ZeroTier Central API token in UCI: +``` +config global 'global' + option zt_central_token '' + option zt_network_id 'e3918db48378cb6a' +``` + +This is only needed so other ZeroTier peers also get the correct default route — WIBLAN clients are already handled by the local `ip rule`. + +## Future: Gateway Auto-Provisioning + +Currently, each gateway node must be manually set up with: +- ZeroTier membership + static IP +- IP forwarding + NAT +- iptables FORWARD rules + +A future enhancement could SSH into candidate nodes from the controller and set up NAT automatically, or use an agent on each gateway that phones home. Out of scope for v1. + +--- + +## Building and Installing luci-app-zt-gateway + +This section covers how to build the `luci-app-zt-gateway` package from source into an `.ipk` using the OpenWRT SDK, then install it on the target router. + +### Prerequisites + +- A **Linux** development machine (Ubuntu 22.04+ / Debian 12+ recommended) +- **~2 GB** free disk space for the SDK +- Basic build tools: `gcc`, `make`, `perl`, `python3`, `zstd` +- SSH/SCP access to the target router (`192.168.13.1`) +- The target router's **OpenWRT version** and **architecture** (determine these before downloading the SDK) + +#### Determine router architecture + +SSH into the router and run: + +```sh +cat /etc/openwrt_release # shows OPENWRT_RELEASE, OPENWRT_ARCH, OPENWRT_BOARD +opkg print-architecture # lists available architectures +``` + +The `OPENWRT_ARCH` line tells you the target (e.g. `mipsel_24kc`, `aarch64_cortex-a53`, `x86_64`). + +### Step 1: Download the OpenWRT SDK + +Download the SDK matching your router's **exact OpenWRT version and architecture** from the OpenWRT downloads server: + +- **OpenWRT 24.10**: https://downloads.openwrt.org/releases/24.10.7/targets/ +- **OpenWRT 23.05**: https://downloads.openwrt.org/releases/23.05.5/targets/ + +Navigate into the appropriate target subdirectory (e.g. `x86/64/`, `ramips/mt7621/`, `mediatek/filogic/`) and download the `openwrt-sdk-*.tar.zst` (or `.tar.xz`) file. + +```sh +# Example for x86_64 / OpenWRT 24.10 +wget https://downloads.openwrt.org/releases/24.10.7/targets/x86/64/openwrt-sdk-24.10.7-x86-64_gcc-13.3.0_musl.Linux-x86_64.tar.zst + +# Extract +tar --zstd -xf openwrt-sdk-24.10.7-*.tar.zst +cd openwrt-sdk-24.10.7-* +``` + +### Step 2: Prepare the SDK + +```sh +# Update and install standard feed definitions +./scripts/feeds update -a +./scripts/feeds install -a +``` + +### Step 3: Add the package source + +Place the `luci-app-zt-gateway` directory into the SDK's package tree. There are two approaches: + +#### Option A: Direct placement (simplest) + +```sh +# Copy or symlink the package directory into the SDK +# If your source is at ~/projects/luci-app-zt-gateway/: +ln -s ~/projects/luci-app-zt-gateway package/luci-app-zt-gateway +``` + +#### Option B: Custom feed (recommended for multiple packages) + +```sh +# 1. Create a feed directory +mkdir -p ~/my-openwrt-feed +ln -s ~/projects/luci-app-zt-gateway ~/my-openwrt-feed/luci-app-zt-gateway + +# 2. Register the feed in the SDK +echo 'src-link customfeed /home/YOURUSER/my-openwrt-feed' >> feeds.conf.default + +# 3. Update and install the custom feed +./scripts/feeds update customfeed +./scripts/feeds install -p customfeed luci-app-zt-gateway +``` + +### Step 4: The Makefile + +The `luci-app-zt-gateway/Makefile` uses the LuCI build system (`luci.mk`) which automates directory mapping, installation, and packaging. The Makefile must be in the package root alongside the `htdocs/` and `root/` directories. + +```makefile +# luci-app-zt-gateway/Makefile +include $(TOPDIR)/rules.mk + +PKG_NAME:=luci-app-zt-gateway +LUCI_TITLE:=LuCI support for ZeroTier Exit Gateway Switching +LUCI_DEPENDS:=+luci-base +ucode +conntrack +LUCI_PKGARCH:=all +PKG_VERSION:=1.0.0 +PKG_RELEASE:=1 +PKG_LICENSE:=Apache-2.0 +PKG_MAINTAINER:=Your Name + +include $(TOPDIR)/feeds/luci/luci.mk + +# call BuildPackage - OpenWrt buildroot signature +``` + +Key points: + +- `luci.mk` (not `package.mk`) is used for LuCI applications. It scans `htdocs/` and `root/` and automatically creates install rules that map files to their target paths: + - `htdocs/luci-static/resources/view/` → `/www/luci-static/resources/view/` + - `root/` → mirrors directly onto the target filesystem root (`/`) +- `LUCI_PKGARCH:=all` because LuCI apps are architecture-independent (JS, ucode, shell only). +- `include $(TOPDIR)/feeds/luci/luci.mk` uses the absolute path variant. If building inside the main `openwrt/luci` feed, use `include ../../luci.mk` instead. +- The trailing comment `# call BuildPackage - OpenWrt buildroot signature` is **required** by the OpenWRT feed scanner to discover this package. +- **Dependencies**: `+luci-base` is mandatory. `+ucode` is needed for the rpcd backend. `+conntrack` is needed because the switch script uses `conntrack` CLI. + +#### Directory-to-path mapping (luci.mk automatics) + +When `luci.mk` processes this package, it automatically installs: + +| Source path in package | Target path on router | +|---|---| +| `htdocs/luci-static/resources/view/zt-gateway/overview.js` | `/www/luci-static/resources/view/zt-gateway/overview.js` | +| `root/etc/config/zt-gateway` | `/etc/config/zt-gateway` | +| `root/usr/share/rpcd/ucode/zt-gateway.uc` | `/usr/share/rpcd/ucode/zt-gateway.uc` | +| `root/usr/share/acl.d/luci-app-zt-gateway.json` | `/usr/share/rpcd/acl.d/luci-app-zt-gateway.json` | +| `root/usr/share/luci/menu.d/luci-app-zt-gateway.json` | `/usr/share/luci/menu.d/luci-app-zt-gateway.json` | +| `root/usr/sbin/zt-gateway-switch` | `/usr/sbin/zt-gateway-switch` | + +### Step 5: Configure and build + +```sh +# Select the package in menuconfig +make menuconfig +# Navigate: LuCI → 3. Applications → luci-app-zt-gateway +# Press 'M' to build as module (do NOT press 'Y' — we want an .ipk, not a firmware image) +# Save and exit + +# Build the package with verbose output +make package/luci-app-zt-gateway/compile V=s +``` + +If `make menuconfig` doesn't show the package, ensure the feeds are updated: + +```sh +./scripts/feeds update -a +./scripts/feeds install luci-app-zt-gateway +``` + +### Step 6: Locate the built .ipk + +```sh +find bin/ -name 'luci-app-zt-gateway*.ipk' -type f +``` + +Typical output path: `bin/packages//customfeed/luci-app-zt-gateway_1.0.0-1_all.ipk` + +Because `LUCI_PKGARCH:=all`, the `.ipk` filename ends with `_all.ipk` — it's architecture-independent and works on any OpenWRT target. + +### Step 7: Install on the router + +#### Transfer the package + +```sh +scp bin/packages/*/customfeed/luci-app-zt-gateway_1.0.0-1_all.ipk root@192.168.13.1:/tmp/ +``` + +#### Install via opkg (OpenWRT ≤ 24.10) + +```sh +ssh root@192.168.13.1 +opkg update +opkg install /tmp/luci-app-zt-gateway_1.0.0-1_all.ipk +``` + +If you get dependency errors (e.g. missing `luci-base`), install them first: + +```sh +opkg install luci-base ucode conntrack +opkg install /tmp/luci-app-zt-gateway_1.0.0-1_all.ipk +``` + +#### Install via apk (OpenWRT 25.12+) + +Starting with OpenWRT 25.12, the package manager is `apk` instead of `opkg`: + +```sh +ssh root@192.168.13.1 +apk add --allow-untrusted /tmp/luci-app-zt-gateway_1.0.0-1_all.ipk +``` + +#### Post-install: refresh LuCI cache + +```sh +rm -rf /tmp/luci-indexcache /tmp/luci-modulecache +``` + +Then navigate to the LuCI web UI — the "ZeroTier Exit Gateway" page should appear under **Services**. + +### Step 8: Configure gateways + +After installation, edit the UCI config to add your gateway nodes: + +```sh +vi /etc/config/zt-gateway +``` + +Or use `uci` commands: + +```sh +uci set zt-gateway.global.active_gateway='amsterdam' +uci set zt-gateway.global.switch_mode='force' + +uci add zt-gateway gateway +uci set zt-gateway.@gateway[-1].region='amsterdam' +uci set zt-gateway.@gateway[-1].label='Amsterdam (ocirosea641)' +uci set zt-gateway.@gateway[-1].ip='10.11.12.3' +uci set zt-gateway.@gateway[-1].default='1' +uci set zt-gateway.@gateway[-1].health_check='ping' + +uci add zt-gateway gateway +uci set zt-gateway.@gateway[-1].region='tirunelveli' +uci set zt-gateway.@gateway[-1].label='Tirunelveli (rpi1000)' +uci set zt-gateway.@gateway[-1].ip='10.11.12.5' +uci set zt-gateway.@gateway[-1].default='0' +uci set zt-gateway.@gateway[-1].health_check='ping' + +uci add zt-gateway gateway +uci set zt-gateway.@gateway[-1].region='bangalore' +uci set zt-gateway.@gateway[-1].label='Bangalore (sensecap-m4)' +uci set zt-gateway.@gateway[-1].ip='10.11.12.4' +uci set zt-gateway.@gateway[-1].default='0' +uci set zt-gateway.@gateway[-1].health_check='ping' + +uci commit zt-gateway +``` + +### Step 9: Verify the rpcd backend is registered + +After install, verify the ubus backend is available: + +```sh +# Restart rpcd to pick up the new backend +/etc/init.d/rpcd restart + +# List available methods +ubus list zt-gateway.* +# Expected output: +# zt-gateway.status +# zt-gateway.switch +# zt-gateway.health +# zt-gateway.drain_status +# zt-gateway.cancel_drain + +# Test the status method +ubus call zt-gateway status +``` + +If `ubus list` doesn't show `zt-gateway.*`, check: + +1. The ucode script is executable and at the correct path: `ls -la /usr/share/rpcd/ucode/zt-gateway.uc` +2. The ACL file exists: `cat /usr/share/rpcd/acl.d/luci-app-zt-gateway.json` +3. rpcd log output: `logread -e rpcd` + +### Quick-build cheat sheet (no SDK, manual .ipk) + +If you don't want to bother with the full SDK and just want to produce an `.ipk` manually, you can use the `opkg-utils` tools on any Linux machine: + +```sh +# Install opkg-utils +sudo apt install opkg-utils # Debian/Ubuntu + +# Create the package staging directory +mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/CONTROL +mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/www/luci-static/resources/view/zt-gateway +mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/etc/config +mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/rpcd/ucode +mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/rpcd/acl.d +mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/luci/menu.d +mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/sbin + +# Copy source files into staging (adjust source paths as needed) +cp htdocs/luci-static/resources/view/zt-gateway/overview.js \ + /tmp/luci-app-zt-gateway_1.0.0-1_all/www/luci-static/resources/view/zt-gateway/ +cp root/etc/config/zt-gateway /tmp/luci-app-zt-gateway_1.0.0-1_all/etc/config/ +cp root/usr/share/rpcd/ucode/zt-gateway.uc /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/rpcd/ucode/ +cp root/usr/share/rpcd/acl.d/luci-app-zt-gateway.json /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/rpcd/acl.d/ +cp root/usr/share/luci/menu.d/luci-app-zt-gateway.json /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/luci/menu.d/ +cp root/usr/sbin/zt-gateway-switch /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/sbin/ +chmod +x /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/sbin/zt-gateway-switch + +# Write the control file +cat > /tmp/luci-app-zt-gateway_1.0.0-1_all/CONTROL/control << 'EOF' +Package: luci-app-zt-gateway +Version: 1.0.0-1 +Depends: libc, luci-base, ucode, conntrack +Source: luci-app-zt-gateway +Section: luci +Architecture: all +Installed-Size: 20480 +Description: LuCI support for ZeroTier Exit Gateway Switching +EOF + +# Build the .ipk +cd /tmp +opkg-build -O -o root -g root luci-app-zt-gateway_1.0.0-1_all + +# Result: /tmp/luci-app-zt-gateway_1.0.0-1_all.ipk +``` + +This manual approach skips the SDK entirely. It's suitable for quick iteration during development. The tradeoff: no automatic dependency resolution, no translation compilation, no JS minification. + +### Development workflow: iterate fast without rebuilding + +During development, you can skip the build step entirely by editing files directly on the router: + +```sh +# Edit the JS view directly +scp htdocs/luci-static/resources/view/zt-gateway/overview.js \ + root@192.168.13.1:/www/luci-static/resources/view/zt-gateway/overview.js + +# Edit the ucode backend +scp root/usr/share/rpcd/ucode/zt-gateway.uc \ + root@192.168.13.1:/usr/share/rpcd/ucode/zt-gateway.uc +/etc/init.d/rpcd restart + +# Edit the switch script +scp root/usr/sbin/zt-gateway-switch \ + root@192.168.13.1:/usr/sbin/zt-gateway-switch + +# Edit the UCI config +scp root/etc/config/zt-gateway \ + root@192.168.13.1:/etc/config/zt-gateway + +# Clear LuCI browser cache after JS changes +ssh root@192.168.13.1 'rm -rf /tmp/luci-indexcache /tmp/luci-modulecache' +``` + +Build the `.ipk` only when you need a clean install or are sharing the package. + +### Troubleshooting + +| Problem | Cause | Fix | +|---|---|---| +| `make menuconfig` doesn't show the package | Feed not registered | Run `./scripts/feeds update -a && ./scripts/feeds install -a` | +| `opkg install` fails with "cannot install package" | Missing dependencies | Run `opkg update` then install dependencies first | +| LuCI page shows 404 | JS view not found at expected path | Verify `/www/luci-static/resources/view/zt-gateway/overview.js` exists on router | +| ubus methods missing | rpcd didn't pick up the ucode backend | `/etc/init.d/rpcd restart`; check `logread -e rpcd` for errors | +| ACL denied when calling ubus from LuCI | ACL JSON file missing or wrong format | Verify `/usr/share/rpcd/acl.d/luci-app-zt-gateway.json` and restart rpcd | +| `luci.mk` not found during build | Building outside the LuCI feed | Use `include $(TOPDIR)/feeds/luci/luci.mk` (absolute) instead of `include ../../luci.mk` (relative) | +| JS view loads but is blank | Import path errors in JS | Open browser dev console; LuCI JS modules use `luci.view`, `luci.form`, etc. | + + +## Local Testing with Docker (macvlan) + +This section covers how to test `luci-app-zt-gateway` locally using the official `openwrt/rootfs` Docker image with macvlan networking. This gives you a full OpenWRT environment with real `ip rule`, `iptables`, `conntrack`, and rpcd — no physical router required. + +### Test architecture + +``` +┌──────────────────────────────────────────────────────────────────┐ +│ Docker host (your dev machine) │ +│ │ +│ ┌──────────────────────┐ ┌──────────────────────┐ │ +│ │ openwrt-router │ │ wiblan-client │ │ +│ │ (macvlan 10.11.13.1)│ │ (macvlan 10.11.13.10)│ │ +│ │ │ │ │ │ +│ │ - LuCI on :80 │ │ - curl / ping │ │ +│ │ - rpcd + ucode │ │ - traffic exits via │ │ +│ │ - zt-gateway-switch │ │ router's table 100 │ │ +│ │ - br-zt + ip rules │ │ │ │ +│ └──────────┬───────────┘ └───────────┬───────────┘ │ +│ │ │ │ +│ └──────────┬─────────────────┘ │ +│ │ │ +│ zt-gateway-macvlan │ +│ 10.11.13.0/24 subnet │ +│ (macvlan bridge mode) │ +│ │ +│ ┌──────────────────────┐ │ +│ │ zt-gw-amsterdam │ ← Simulates a remote ZeroTier exit │ +│ │ (macvlan 10.11.12.3) │ node with NAT + ip_forward │ +│ │ │ │ +│ │ - ip_forward=1 │ │ +│ │ - iptables MASQ │ │ +│ └──────────────────────┘ │ +│ │ +│ zt-exit-macvlan │ +│ 10.11.12.0/24 subnet │ +└──────────────────────────────────────────────────────────────────┘ +``` + +The key insight: we use **two macvlan networks** to simulate the two network segments the real router has: +1. **zt-gateway-macvlan** (`10.11.13.0/24`) — represents the WIBLAN subnet. The router sits at `.1`, client at `.10`. +2. **zt-exit-macvlan** (`10.11.12.0/24`) — represents the ZeroTier network where exit gateway nodes live. + +### Step 1: Create the Dockerfile + +Create a Dockerfile that builds a pre-configured OpenWRT image with all dependencies and the zt-gateway package: + +```dockerfile +# Dockerfile.openwrt-dev +FROM openwrt/rootfs:x86-64-24.10.7 + +# Install required packages +RUN opkg update && opkg install \ + luci \ + luci-base \ + ucode \ + conntrack \ + ip-bridge \ + iptables \ + kmod-ipt-conntrack \ + kmod-ipt-connmark \ + kmod-ipt-mark \ + kmod-ipt-extra \ + && rm -rf /var/opkg-lists/* + +# Enable uhttpd +RUN /etc/init.d/uhttpd enable +RUN /etc/init.d/uhttpd start + +# Enable rpcd +RUN /etc/init.d/rpcd enable +RUN /etc/init.d/rpcd start + +# Set root password (required for LuCI login) +RUN echo "root:root" | chpasswd + +EXPOSE 80 + +CMD ["/sbin/init"] +``` + +Build it: + +```sh +docker build -t openwrt-zt-gateway-dev -f Dockerfile.openwrt-dev . +``` + +**Note:** The exact `kmod-*` packages available depend on the OpenWRT version. If `opkg install` fails on kernel modules, skip them — the Docker container's kernel is the host kernel, so the modules are already loaded. What matters is that `iptables -t mangle` and `conntrack` work. Verify with: + +```sh +# Run a quick check after building +docker run --rm --privileged openwrt-zt-gateway-dev sh -c \ + "iptables -t mangle -L && echo 'mangle OK' && conntrack -L 2>/dev/null && echo 'conntrack OK'" +``` + +### Step 2: Create the Docker Compose file + +```yaml +# docker-compose.yml +version: "3.8" + +networks: + # Simulates the WIBLAN client network (10.11.13.0/24) + zt-gateway-lan: + driver: macvlan + driver_opts: + mode: bridge + ipam: + config: + - subnet: "10.11.13.0/24" + gateway: "10.11.13.1" + ip_range: "10.11.13.100/28" + + # Simulates the ZeroTier exit node network (10.11.12.0/24) + zt-exit-net: + driver: macvlan + driver_opts: + mode: bridge + ipam: + config: + - subnet: "10.11.12.0/24" + ip_range: "10.11.12.100/28" + +services: + # The OpenWRT router running luci-app-zt-gateway + openwrt-router: + image: openwrt-zt-gateway-dev + container_name: openwrt-router + privileged: true + networks: + zt-gateway-lan: + ipv4_address: "10.11.13.1" + zt-exit-net: + ipv4_address: "10.11.12.1" + ports: + - "8080:80" + volumes: + # Hot-mount source files for live iteration (key advantage) + - ./luci-app-zt-gateway/htdocs/luci-static/resources/view/zt-gateway/overview.js:/www/luci-static/resources/view/zt-gateway/overview.js + - ./luci-app-zt-gateway/root/usr/share/rpcd/ucode/zt-gateway.uc:/usr/share/rpcd/ucode/zt-gateway.uc + - ./luci-app-zt-gateway/root/usr/sbin/zt-gateway-switch:/usr/sbin/zt-gateway-switch + - ./luci-app-zt-gateway/root/etc/config/zt-gateway:/etc/config/zt-gateway + - ./luci-app-zt-gateway/root/usr/share/rpcd/acl.d/luci-app-zt-gateway.json:/usr/share/rpcd/acl.d/luci-app-zt-gateway.json + - ./luci-app-zt-gateway/root/usr/share/luci/menu.d/luci-app-zt-gateway.json:/usr/share/luci/menu.d/luci-app-zt-gateway.json + cap_add: + - NET_ADMIN + - SYS_ADMIN + sysctls: + - net.ipv4.ip_forward=1 + - net.ipv4.conf.all.send_redirects=0 + command: ["/sbin/init"] + + # A WIBLAN client to test routing through the gateway + wiblan-client: + image: alpine:latest + container_name: wiblan-client + networks: + zt-gateway-lan: + ipv4_address: "10.11.13.10" + cap_add: + - NET_ADMIN + command: ["sleep", "infinity"] + + # Simulates the amsterdam ZeroTier exit gateway (10.11.12.3) + zt-gw-amsterdam: + image: alpine:latest + container_name: zt-gw-amsterdam + networks: + zt-exit-net: + ipv4_address: "10.11.12.3" + cap_add: + - NET_ADMIN + sysctls: + - net.ipv4.ip_forward=1 + command: ["sleep", "infinity"] + + # Simulates the tirunelveli ZeroTier exit gateway (10.11.12.5) + zt-gw-tirunelveli: + image: alpine:latest + container_name: zt-gw-tirunelveli + networks: + zt-exit-net: + ipv4_address: "10.11.12.5" + cap_add: + - NET_ADMIN + sysctls: + - net.ipv4.ip_forward=1 + command: ["sleep", "infinity"] +``` + +### Step 3: Launch and configure the environment + +```sh +# Start all containers +docker compose up -d + +# Wait a few seconds for init, then configure the simulated exit gateways +# These simulate real ZeroTier exit nodes with NAT +``` + +#### Configure the amsterdam exit gateway (10.11.12.3) + +```sh +docker exec -it zt-gw-amsterdam sh -c ' + # Enable forwarding + echo 1 > /proc/sys/net/ipv4/ip_forward + + # Add a default route back to the router so return traffic works + ip route add 10.11.13.0/24 via 10.11.12.1 + + # Add NAT so traffic from WIBLAN clients can reach the internet + # (or just other container networks for testing) + apk add --no-cache iptables + iptables -t nat -A POSTROUTING -s 10.11.13.0/24 -j MASQUERADE + iptables -A FORWARD -s 10.11.13.0/24 -j ACCEPT + iptables -A FORWARD -d 10.11.13.0/24 -m state --state ESTABLISHED,RELATED -j ACCEPT + + # Add a fake "internet" endpoint (just the Docker host gateway) + # In real use, this would be the actual internet + ping -c 1 10.11.12.1 2>/dev/null +' +``` + +#### Configure the tirunelveli exit gateway (10.11.12.5) + +```sh +docker exec -it zt-gw-tirunelveli sh -c ' + echo 1 > /proc/sys/net/ipv4/ip_forward + ip route add 10.11.13.0/24 via 10.11.12.1 + apk add --no-cache iptables + iptables -t nat -A POSTROUTING -s 10.11.13.0/24 -j MASQUERADE + iptables -A FORWARD -s 10.11.13.0/24 -j ACCEPT + iptables -A FORWARD -d 10.11.13.0/24 -m state --state ESTABLISHED,RELATED -j ACCEPT +' +``` + +#### Configure the OpenWRT router + +```sh +docker exec -it openwrt-router sh +``` + +Inside the router container: + +```sh +# 1. Create the br-zt bridge (simulates the ZeroTier bridge) +ip link add name br-zt type bridge +ip link set br-zt up + +# Assign the ZT-exit IP to the bridge so the router can reach 10.11.12.0/24 +ip addr add 10.11.12.1/24 dev br-zt + +# 2. Set up policy routing (same as the real router) + +# Host route to the current gateway (ARP fix) +ip route replace 10.11.12.3 dev br-zt + +# Default route via the current gateway in table 100 +ip route replace default via 10.11.12.3 dev br-zt table 100 + +# Route WIBLAN subnet via br-zt in mwan3 table (for return traffic) +ip route replace 10.11.13.0/24 dev br-zt table 1 + +# Source-based routing rule: WIBLAN traffic → table 100 +ip rule add from 10.11.13.0/24 table 100 priority 100 + +# 3. Verify routing +ip route show table 100 +# Expected: default via 10.11.12.3 dev br-zt + +ip rule show +# Expected: 100: from 10.11.13.0/24 lookup 100 + +# 4. Test from the router itself +ping -c 2 10.11.12.3 # Should reach amsterdam gateway +``` + +#### Configure the WIBLAN client + +```sh +docker exec -it wiblan-client sh -c ' + # Add a route to the router for the ZT subnet + ip route add 10.11.12.0/24 via 10.11.13.1 + + # Set the OpenWRT router as default gateway + ip route add default via 10.11.13.1 + + # Test connectivity + ping -c 2 10.11.12.3 +' +``` + +### Step 4: Install and test the luci-app-zt-gateway package + +#### Option A: The package is already volume-mounted + +If you used the `volumes:` section in `docker-compose.yml`, the files are already in place. Just restart services: + +```sh +docker exec -it openwrt-router sh -c ' + chmod +x /usr/sbin/zt-gateway-switch + /etc/init.d/rpcd restart + rm -rf /tmp/luci-indexcache /tmp/luci-modulecache +' +``` + +#### Option B: Install the .ipk + +If you built an `.ipk`, copy it into the container: + +```sh +docker cp luci-app-zt-gateway_1.0.0-1_all.ipk openwrt-router:/tmp/ +docker exec -it openwrt-router opkg install /tmp/luci-app-zt-gateway_1.0.0-1_all.ipk +``` + +#### Verify the rpcd backend + +```sh +docker exec -it openwrt-router ubus list zt-gateway.* +# Expected: +# zt-gateway.status +# zt-gateway.switch +# zt-gateway.health +# zt-gateway.drain_status +# zt-gateway.cancel_drain + +docker exec -it openwrt-router ubus call zt-gateway status +``` + +#### Access the LuCI UI + +Open http://localhost:8080 in your browser. Log in with `root` / `root`. Navigate to **Services → ZeroTier Exit Gateway**. + +### Step 5: Run the test scenarios + +#### Force switch test + +```sh +# 1. Verify current gateway is amsterdam (10.11.12.3) +docker exec -it openwrt-router ip route show table 100 +# Expected: default via 10.11.12.3 dev br-zt + +# 2. Trigger a force switch to tirunelveli via the CLI script +docker exec -it openwrt-router zt-gateway-switch 10.11.12.5 force + +# 3. Verify the route changed +docker exec -it openwrt-router ip route show table 100 +# Expected: default via 10.11.12.5 dev br-zt + +# 4. Verify conntrack was flushed +docker exec -it openwrt-router conntrack -L -s 10.11.13.0/24 2>/dev/null +# Expected: empty (all entries flushed) + +# 5. Verify from the client side +docker exec -it wiblan-client ping -c 2 10.11.12.5 +``` + +#### Graceful drain test + +```sh +# 1. Start a long-running connection from the client +docker exec -d wiblan-client ping 10.11.12.3 # background ping + +# 2. Trigger a graceful switch (this will be done from the LuCI UI +# or via ubus call) +docker exec -it openwrt-router zt-gateway-switch 10.11.12.5 graceful 300 + +# 3. Verify both routing tables exist during drain +docker exec -it openwrt-router ip route show table 100 +# Expected: default via 10.11.12.5 dev br-zt (new gateway) + +docker exec -it openwrt-router ip route show table 101 +# Expected: default via 10.11.12.3 dev br-zt (old gateway, drain) + +# 4. Verify the fwmark rule exists +docker exec -it openwrt-router ip rule show +# Expected: 99: from all fwmark 0x100 lookup 101 + +# 5. Verify the mangle rules +docker exec -it openwrt-router iptables -t mangle -L -v + +# 6. Watch drain progress +docker exec -it openwrt-router conntrack -L -m 0x100 2>/dev/null | wc -l + +# 7. Wait for drain to complete (or kill the background ping to speed it up) +docker exec -it wiblan-client pkill ping + +# 8. After drain completes, verify cleanup +docker exec -it openwrt-router ip route show table 101 +# Expected: empty (drain table removed) +docker exec -it openwrt-router ip rule show +# Expected: no fwmark 0x100 rule +``` + +#### Switch to unreachable gateway test + +```sh +# 1. Stop the tirunelveli gateway to simulate it being down +docker stop zt-gw-tirunelveli + +# 2. Try to switch to it (should fail pre-flight) +docker exec -it openwrt-router zt-gateway-switch 10.11.12.5 force +# Expected: "Gateway unreachable" error, no route changes + +# 3. Verify routes unchanged +docker exec -it openwrt-router ip route show table 100 + +# 4. Restart for subsequent tests +docker start zt-gw-tirunelveli +``` + +#### Drain timeout test + +```sh +# 1. Start a persistent connection +docker exec -d wiblan-client ping 10.11.12.3 + +# 2. Graceful switch with short timeout (30s) +docker exec -it openwrt-router zt-gateway-switch 10.11.12.5 graceful 30 + +# 3. Watch the drain monitor — it will hit the 30s timeout +# because the ping keeps conntrack entries alive +docker exec -it openwrt-router conntrack -L -m 0x100 2>/dev/null | wc -l + +# 4. After 30s, should auto-fallback to force switch +docker exec -it openwrt-router ip route show table 100 +# Expected: default via 10.11.12.5 dev br-zt (force applied) +docker exec -it openwrt-router ip route show table 101 +# Expected: empty (drain cleaned up by force) + +# 5. Clean up +docker exec -it wiblan-client pkill ping +``` + +### Step 6: Iterate on fixes + +The key advantage of the Docker setup is the **volume mounts** in the compose file. When you edit a source file on the host, it's immediately reflected inside the container. + +#### Live-editing workflow + +```sh +# 1. Edit a source file on the host +vim luci-app-zt-gateway/root/usr/sbin/zt-gateway-switch + +# 2. Re-test inside the container (file is already updated via volume mount) +docker exec -it openwrt-router zt-gateway-switch 10.11.12.5 force + +# 3. Check results +docker exec -it openwrt-router ip route show table 100 +``` + +For LuCI JS changes, you need to clear the cache: + +```sh +# Edit JS on host +vim luci-app-zt-gateway/htdocs/luci-static/resources/view/zt-gateway/overview.js + +# Clear cache inside container +docker exec -it openwrt-router rm -rf /tmp/luci-indexcache /tmp/luci-modulecache + +# Refresh browser +``` + +For ucode backend changes: + +```sh +# Edit ucode on host +vim luci-app-zt-gateway/root/usr/share/rpcd/ucode/zt-gateway.uc + +# Restart rpcd to reload the backend +docker exec -it openwrt-router /etc/init.d/rpcd restart +``` + +#### Quick-snapshot the exact container state + +If you want to save a known-good state before making a risky change: + +```sh +# Save container state as a Docker image +docker commit openwrt-router openwrt-zt-gateway-snapshot:good + +# ... make changes, break things ... + +# Restore from snapshot +docker compose down +docker run -d --name openwrt-router \ + --privileged \ + -p 8080:80 \ + openwrt-zt-gateway-snapshot:good +``` + +#### Watch logs in real-time + +```sh +# OpenWRT system log +docker exec -it openwrt-router logread -f + +# rpcd errors specifically +docker exec -it openwrt-router logread -f -e rpcd + +# conntrack events in real-time +docker exec -it openwrt-router conntrack -E +``` + +#### Run the full test suite in one shot + +```sh +#!/bin/bash +# test-zt-gateway.sh — Run all test scenarios in Docker +set -e + +ROUTER="openwrt-router" +CLIENT="wiblan-client" +GW_AMS="10.11.12.3" +GW_TIR="10.11.12.5" + +echo "=== Test 1: Force switch amsterdam → tirunelveli ===" +docker exec $ROUTER zt-gateway-switch $GW_TIR force +ROUTE=$(docker exec $ROUTER ip route show table 100 | grep default) +echo " Route after force: $ROUTE" +echo "$ROUTE" | grep -q "$GW_TIR" && echo " PASS" || echo " FAIL" + +echo "=== Test 2: Force switch tirunelveli → amsterdam ===" +docker exec $ROUTER zt-gateway-switch $GW_AMS force +ROUTE=$(docker exec $ROUTER ip route show table 100 | grep default) +echo " Route after force: $ROUTE" +echo "$ROUTE" | grep -q "$GW_AMS" && echo " PASS" || echo " FAIL" + +echo "=== Test 3: Switch to unreachable gateway ===" +docker stop zt-gw-tirunelveli +if docker exec $ROUTER zt-gateway-switch $GW_TIR force 2>&1 | grep -qi "unreachable\|fail\|error"; then + echo " PASS (blocked as expected)" +else + echo " FAIL (should have been blocked)" +fi +docker start zt-gw-tirunelveli +sleep 3 + +echo "=== Test 4: Verify client can reach gateway ===" +docker exec $CLIENT ping -c 2 $GW_AMS && echo " PASS" || echo " FAIL" + +echo "=== Test 5: Verify ubus backend ===" +docker exec $ROUTER ubus call zt-gateway status | grep -q "amsterdam" && echo " PASS" || echo " FAIL" + +echo "=== All tests complete ===" +``` + +### Important limitations of the Docker test environment + +| Limitation | Impact | Mitigation | +|---|---|---| +| Docker shares the host kernel | `kmod-*` packages from opkg won't load (wrong kernel version) | Use host kernel modules; verify with `lsmod`/`iptables -t mangle -L` | +| `br-zt` is simulated, not a real ZeroTier bridge | No actual ZT membership, no `ztk4jpk77j` interface | Manually create `br-zt` bridge and assign IPs; skip hotplug scripts | +| `conntrack -D -s` may behave differently | Docker's conntrack table includes host traffic too | Use `-s` and `-d` filters; or run `conntrack -F` for a clean slate | +| No real `rc.local` / hotplug execution on boot | Persistence scripts can't be tested end-to-end | Test the switch script logic; verify `sed` replacements on the files separately | +| `iptables -t mangle` requires `NET_ADMIN` | Container must run `--privileged` or `--cap-add=NET_ADMIN` | Already configured in compose file | +| Macvlan host isolation | Host can't ping macvlan containers by default | Use a macvlan shim interface (see below) or just `docker exec` | +| No real ZeroTier Central API | Can't test the future ZT API integration | Mock the API calls or skip this test | + +### Fixing the macvlan host-isolation issue + +By default, the Docker host **cannot** communicate directly with containers on a macvlan network. If you need to ping/curl the router or client from your host machine (not just via `docker exec`), create a macvlan shim interface: + +```sh +# Create a macvlan interface on the host that bridges to the same parent +# Replace eth0 with your host's primary interface +sudo ip link add link eth0 name mv-shim type macvlan mode bridge +sudo ip addr add 10.11.13.254/32 dev mv-shim +sudo ip link set mv-shim up + +# Add a route to the WIBLAN subnet via the shim +sudo ip route add 10.11.13.0/24 dev mv-shim + +# Now you can reach the containers from host +ping 10.11.13.1 # the router +curl http://10.11.13.1 # LuCI directly (skip the port mapping) + +# For the ZT exit subnet +sudo ip link add link eth0 name mv-exit-shim type macvlan mode bridge +sudo ip addr add 10.11.12.254/32 dev mv-exit-shim +sudo ip link set mv-exit-shim up +sudo ip route add 10.11.12.0/24 dev mv-exit-shim +``` + +Clean up when done: + +```sh +sudo ip link del mv-shim +sudo ip link del mv-exit-shim +``` + +### Teardown + +```sh +# Stop and remove all containers +docker compose down + +# Remove the shim interfaces (if created) +sudo ip link del mv-shim 2>/dev/null +sudo ip link del mv-exit-shim 2>/dev/null + +# Remove the custom images (optional) +docker rmi openwrt-zt-gateway-dev openwrt-zt-gateway-snapshot:good +``` \ No newline at end of file diff --git a/README.md b/README.md new file mode 100644 index 0000000..5ce3144 --- /dev/null +++ b/README.md @@ -0,0 +1,38 @@ +# luci-app-zt-gateway + +A LuCI application for OpenWRT that lets users switch which remote ZeroTier node +acts as the internet exit gateway for WIBLAN clients (`10.11.13.0/24`). + +Users select a region/gateway, choose a switch mode, and the app reconfigures +kernel routing, conntrack, hotplug scripts, and UCI network routes with minimal +disruption. + +## Switch modes + +- **Force** — instant cutover (~1-2s disruption); existing connections break and + clients auto-reconnect through the new gateway. Flushes conntrack for the + WIBLAN subnet. +- **Graceful drain** — zero disruption; existing connections keep using the old + gateway via fwmark + a drain routing table, while new connections go through + the new gateway immediately. Auto-falls-back to a force switch after a + configurable timeout if drain does not complete. + +## Status + +> **Planning / pre-implementation.** + +The full design — file structure, UCI config schema, rpcd backend (ucode), +`zt-gateway-switch` shell script, LuCI frontend, switch logic for both modes, +persistence model, health checks, rollback/safety, Docker macvlan test harness, +and build/install instructions — is the source of truth in: + +[`docs/.omp/plans/zerotier-gateway-switching.md`](.omp/plans/zerotier-gateway-switching.md) + +(also tracked at `.omp/plans/zerotier-gateway-switching.md`) + +Implementation (Makefile, ucode backend, switch script, `overview.js`, UCI +config, ACL, menu entry) lands in subsequent commits against this repository. + +## License + +Apache-2.0