infra/luci-app-zt-gateway
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
f6bf6e7cd4259daa2e805a696cd7a60039aa8477
luci-app-zt-gateway/.omp/plans/zerotier-gateway-switching.md
Malar Invention f6bf6e7cd4 docs: initial plan for ZeroTier exit gateway switcher 
Design: .omp/plans/zerotier-gateway-switching.md — LuCI app + rpcd/ucode backend + zt-gateway-switch shell script for force / graceful-drain gateway switching on OpenWRT.
2026-06-19 00:03:09 +05:30

47 KiB
Raw Blame History

Plan: luci-app-zt-gateway — ZeroTier Exit Gateway Switcher

Goal

A LuCI app for OpenWRT that lets users switch which remote ZeroTier node acts as the internet exit gateway for WIBLAN clients (10.11.13.0/24). Users select a region/gateway, choose a switch mode, and the app reconfigures routing with zero or minimal disruption.

Existing Architecture (constraints)

  • Router: OpenWRT, 192.168.13.1
  • ZT network: e3918db48378cb6a, interface ztk4jpk77j, bridge br-zt
  • Current gateway: 10.11.12.3 (ocirosea641, amsterdam)
  • WIBLAN clients: 10.11.13.0/24 routed via ip rule priority 100 → table 100
  • Table 100: default via <gateway> dev br-zt
  • Host route: <gateway> dev br-zt (ARP fix — gateway IP not directly reachable via bridge)
  • mwan3: uses table 1; requires 10.11.13.0/24 dev br-zt table 1 for return traffic
  • Hotplug: /etc/hotplug.d/net/99-zerotier-bridge adds ztk4jpk77j to br-zt, removes conflicting routes, adds table 1 route
  • Conntrack: conntrack package available; conntrack -D -s 10.11.13.0/24 flushes WIBLAN entries

Two Switch Modes

1. Force Switch (instant, ~1-2s disruption)

All existing connections break immediately. Clients auto-reconnect through the new gateway.

Steps:

  1. Pre-flight: ping new gateway over ZT, abort if unreachable
  2. Atomically replace host route + table 100 default route
  3. Flush conntrack for 10.11.13.0/24
  4. Update hotplug script + rc.local with new gateway IP
  5. Update UCI network routes for persistence
  6. Verify: ip route get 8.8.8.8 from 10.11.13.1 table 100

User impact: Active downloads/video calls drop and reconnect. Web pages refresh. SSH sessions die.

2. Graceful Drain (zero disruption, complex)

Existing connections continue on the old gateway until they naturally close. New connections go through the new gateway immediately. Uses fwmark + dual routing table.

Steps:

  1. Pre-flight: ping new gateway over ZT, abort if unreachable
  2. Create drain table (101) with default via <OLD_GATEWAY> dev br-zt
  3. Add iptables mangle rules to mark all ESTABLISHED connections from WIBLAN with fwmark 0x100
  4. Add ip rule add fwmark 0x100 table 101 priority 99 (before the existing priority-100 src rule)
  5. Update table 100 default to new gateway
  6. Update host route to new gateway
  7. Start drain monitor: count conntrack entries with mark; when zero → clean up mangle rules, drain table, fwmark rule
  8. Update hotplug script + rc.local + UCI network routes for persistence
  9. If drain doesn't complete within configurable timeout (default 10 min), auto-fallback to force switch

Key insight: fwmark 0x100 at priority 99 matches BEFORE the src 10.11.13.0/24 → table 100 rule. So marked (existing) connections go to table 101 (old gateway), while unmarked (new) connections go to table 100 (new gateway).

User impact: Zero disruption. Active connections keep working. New connections get new gateway latency/exit point.

Conntrack marking detail:

# Restore mark from conntrack for returning packets
iptables -t mangle -A PREROUTING -i br-zt -s 10.11.13.0/24 -m conntrack --ctstate ESTABLISHED,RELATED -j CONNMARK --restore-mark

# Mark ESTABLISHED packets that don't have a mark yet
iptables -t mangle -A PREROUTING -i br-zt -s 10.11.13.0/24 -m conntrack --ctstate ESTABLISHED,RELATED -m mark --mark 0 -j MARK --set-mark 0x100

# Save packet mark back to conntrack
iptables -t mangle -A PREROUTING -i br-zt -s 10.11.13.0/24 -j CONNMARK --save-mark

Drain monitor: Periodic check via conntrack -L -m 0x100 2>/dev/null | wc -l. When count is 0, clean up.

File Structure

luci-app-zt-gateway/
├── Makefile
├── htdocs/
│   └── luci-static/
│       └── resources/
│           └── view/
│               └── zt-gateway/
│                   └── overview.js      # Main UI view
├── root/
│   ├── etc/
│   │   └── config/
│   │       └── zt-gateway              # UCI config: gateway registry + active state
│   ├── usr/
│   │   ├── share/
│   │   │   ├── acl.d/
│   │   │   │   └── luci-app-zt-gateway.json  # ACL permissions
│   │   │   ├── luci/
│   │   │   │   └── menu.d/
│   │   │   │       └── luci-app-zt-gateway.json  # Menu entry
│   │   │   └── rpcd/
│   │   │       └── ucode/
│   │   │           └── zt-gateway.uc   # rpcd backend: switch, status, health, drain
│   │   └── sbin/
│   │       └── zt-gateway-switch       # Shell script: actual switch logic (called by rpcd)

UCI Config Schema (/etc/config/zt-gateway)

config global 'global'
    option active_gateway ''         # Region key of active gateway
    option switch_mode 'force'       # 'force' or 'graceful'
    option drain_timeout '600'       # Graceful drain timeout in seconds (default 10 min)
    option health_interval '60'      # Health check interval in seconds
    option auto_failback '0'         # Auto-failback to default on gateway down

config gateway
    option region 'amsterdam'
    option label 'Amsterdam (ocirosea641)'
    option ip '10.11.12.3'
    option default '1'               # Default gateway on boot
    option health_check 'ping'        # 'ping' | 'tcp' | 'none'

config gateway
    option region 'tirunelveli'
    option label 'Tirunelveli (rpi1000)'
    option ip '10.11.12.5'
    option default '0'
    option health_check 'ping'

config gateway
    option region 'bangalore'
    option label 'Bangalore (sensecap-m4)'
    option ip '10.11.12.4'
    option default '0'
    option health_check 'ping'        # Not yet joined to ZT network; assign IP after joining

rpcd Backend (/usr/share/rpcd/ucode/zt-gateway.uc)

Exposes these ubus methods:

Method Args Returns Description
status — { active_gateway, gateways: [...], drain_status? } Current state + all configured gateways
switch { region, mode } { success, message } Execute gateway switch
health { region } { reachable, latency_ms } Ping a gateway via ZT
drain_status — { active, remaining_connections } Graceful drain progress
cancel_drain — { success } Cancel graceful drain, force switch to new gateway

Shell Script (/usr/sbin/zt-gateway-switch)

The rpcd backend calls this script for the actual system changes. Separating it from ucode makes it testable from SSH.

Usage: zt-gateway-switch <new_ip> <mode> [drain_timeout]

Modes:
  force      - Instant switch + conntrack flush
  graceful   - fwmark drain with timeout fallback to force

Force switch logic:

  1. Validate new gateway IP reachable: ping -c 2 -W 3 <new_ip>
  2. ip route replace <new_ip> dev br-zt (host route)
  3. ip route replace default via <new_ip> dev br-zt table 100
  4. conntrack -D -s 10.11.13.0/24 2>/dev/null
  5. Update /etc/hotplug.d/net/99-zerotier-bridge with new gateway IP
  6. Update /etc/rc.local with new gateway IP
  7. Update UCI network routes: zt_gateway_host, zt_gateway_default
  8. uci commit network
  9. Verify new route

Graceful switch logic:

  1. Validate new gateway IP reachable
  2. Read current gateway from UCI
  3. Create table 101: ip route add default via <OLD_IP> dev br-zt table 101
  4. Add mangle rules (fwmark 0x100 for ESTABLISHED)
  5. Add ip rule add fwmark 0x100 table 101 priority 99
  6. Update table 100 default + host route to new gateway
  7. Start drain monitor loop:
    • Every 5s: conntrack -L -m 0x100 2>/dev/null | wc -l
    • If count == 0: clean up mangle rules, fwmark rule, table 101 → done
    • If timeout exceeded: force flush + clean up → done
  8. Update hotplug, rc.local, UCI

Cleanup function (called after drain completes or on cancel):

  1. iptables -t mangle -D <each rule>
  2. ip rule del fwmark 0x100 table 101 priority 99
  3. ip route del default dev br-zt table 101

LuCI Frontend (overview.js)

Layout

┌─────────────────────────────────────────────────┐
│ ZeroTier Exit Gateway                            │
├─────────────────────────────────────────────────┤
│                                                  │
│  Active Gateway:  amsterdam (10.11.12.3)  ● UP  │
│  Switch Mode:     [Force v]                     │
│  Drain Timeout:   600s  (shown if graceful)     │
│                                                  │
│  ┌─────────────────────────────────────────────┐│
│  │ Region      │ Label            │ IP          ││
│  ├─────────────┼──────────────────┼─────────────┤│
│  │ ● amsterdam │ Amsterdam        │ 10.11.12.3  ││
│  │ ○ tirunelveli│ Tirunelveli     │ 10.11.12.5  ││
│  │ ○ bangalore │ Bangalore        │ 10.11.12.4  ││
│  └─────────────┴──────────────────┴─────────────┘│
│                                                  │
│  [Switch to Selected]                            │
│                                                  │
│  ── Status ──                                    │
│  Drain progress: 14 connections remaining        │
│  [Cancel Drain → Force Switch]                  │
│                                                  │
└─────────────────────────────────────────────────┘

JavaScript architecture

  • Form.Map tied to zt-gateway UCI config for settings section (switch mode, drain timeout)
  • Radio-button list for gateway selection (reads config gateway sections)
  • Custom button handler for "Switch to Selected" — calls zt-gateway.switch RPC
  • Polling for drain status when graceful switch is in progress
  • Health status indicator (green/red dot) per gateway, refreshed every health_interval

Key imports

'use strict';
import { view } from 'luci.view';
import { form } from 'luci.form';
import { rpc } from 'luci.rpc';
import { ui } from 'luci.ui';
import { dom } from 'luci.dom';
import { Poll } from 'luci.poll';

Persistence Model

The gateway IP appears in five places on the router. The switch script updates them all atomically:

Location What How updated
Kernel: host route <gw_ip> dev br-zt ip route replace
Kernel: table 100 default default via <gw_ip> dev br-zt table 100 ip route replace
UCI: /etc/config/network zt_gateway_host + zt_gateway_default routes uci set + uci commit
Hotplug: 99-zerotier-bridge host route + table 100 route + table 1 return sed replace
Boot: /etc/rc.local all three routes sed replace

The hotplug and rc.local currently hardcode 10.11.12.3. The script replaces these with the new gateway IP. A cleaner future approach would be to have the hotplug/rc.local read from UCI, but that's a refactoring step.

Health Check

Default method: ping -c 2 -W 3 <gateway_ip> over the ZT network.

  • Runs every health_interval seconds via cron or the LuCI polling mechanism
  • If active gateway becomes unreachable AND auto_failback='1': switch to the default='1' gateway
  • Health status displayed in UI with latency

Rollback / Safety

  • Pre-flight: refuse to switch if new gateway is unreachable
  • Graceful drain timeout: if drain doesn't complete, auto-fallback to force
  • If switch script fails partway: the host route + table 100 are updated atomically; worst case the old conntrack entries persist and time out naturally
  • Manual recovery: uci set zt-gateway.global.active_gateway='<region>' + run /usr/sbin/zt-gateway-switch <ip> force

Files to Create

File Purpose
luci-app-zt-gateway/Makefile OpenWRT build system
luci-app-zt-gateway/root/etc/config/zt-gateway UCI config skeleton
luci-app-zt-gateway/root/usr/share/acl.d/luci-app-zt-gateway.json RPC + UCI ACL
luci-app-zt-gateway/root/usr/share/luci/menu.d/luci-app-zt-gateway.json Menu entry
luci-app-zt-gateway/root/usr/share/rpcd/ucode/zt-gateway.uc rpcd backend
luci-app-zt-gateway/root/usr/sbin/zt-gateway-switch Switch script
luci-app-zt-gateway/htdocs/luci-static/resources/view/zt-gateway/overview.js Frontend UI

Verification

  1. Install package on router
  2. Configure 2+ gateways in /etc/config/zt-gateway
  3. Force switch test: select new region → Force → verify ip route show table 100 shows new gateway, curl ifconfig.me from WIBLAN client shows new exit IP
  4. Graceful switch test: select new region → Graceful → verify existing SSH on WIBLAN client stays alive, new connections go through new gateway, drain counter decrements
  5. Fail test: switch to unreachable gateway → verify pre-flight blocks the switch with error message
  6. Drain timeout test: graceful switch with 30s timeout → verify it falls back to force after timeout
  7. Reboot persistence: switch gateway → reboot router → verify table 100 still has the new gateway

Future: ZeroTier Central API Integration

Skipping for now. When added, the switch script should:

  1. Call DELETE /api/network/{networkId}/route/{routeId} for the old 0.0.0.0/0 managed route
  2. Call POST /api/network/{networkId}/route with { "target": "0.0.0.0/0", "via": "<new_gateway_ip>" }
  3. Toggle "Allow Default Route Override" on new gateway node, disable on old

Requires storing a ZeroTier Central API token in UCI:

config global 'global'
    option zt_central_token ''
    option zt_network_id 'e3918db48378cb6a'

This is only needed so other ZeroTier peers also get the correct default route — WIBLAN clients are already handled by the local ip rule.

Future: Gateway Auto-Provisioning

Currently, each gateway node must be manually set up with:

  • ZeroTier membership + static IP
  • IP forwarding + NAT
  • iptables FORWARD rules

A future enhancement could SSH into candidate nodes from the controller and set up NAT automatically, or use an agent on each gateway that phones home. Out of scope for v1.

Building and Installing luci-app-zt-gateway

This section covers how to build the luci-app-zt-gateway package from source into an .ipk using the OpenWRT SDK, then install it on the target router.

Prerequisites

  • A Linux development machine (Ubuntu 22.04+ / Debian 12+ recommended)
  • ~2 GB free disk space for the SDK
  • Basic build tools: gcc, make, perl, python3, zstd
  • SSH/SCP access to the target router (192.168.13.1)
  • The target router's OpenWRT version and architecture (determine these before downloading the SDK)

Determine router architecture

SSH into the router and run:

cat /etc/openwrt_release   # shows OPENWRT_RELEASE, OPENWRT_ARCH, OPENWRT_BOARD
opkg print-architecture    # lists available architectures

The OPENWRT_ARCH line tells you the target (e.g. mipsel_24kc, aarch64_cortex-a53, x86_64).

Step 1: Download the OpenWRT SDK

Download the SDK matching your router's exact OpenWRT version and architecture from the OpenWRT downloads server:

Navigate into the appropriate target subdirectory (e.g. x86/64/, ramips/mt7621/, mediatek/filogic/) and download the openwrt-sdk-*.tar.zst (or .tar.xz) file.

# Example for x86_64 / OpenWRT 24.10
wget https://downloads.openwrt.org/releases/24.10.7/targets/x86/64/openwrt-sdk-24.10.7-x86-64_gcc-13.3.0_musl.Linux-x86_64.tar.zst

# Extract
tar --zstd -xf openwrt-sdk-24.10.7-*.tar.zst
cd openwrt-sdk-24.10.7-*

Step 2: Prepare the SDK

# Update and install standard feed definitions
./scripts/feeds update -a
./scripts/feeds install -a

Step 3: Add the package source

Place the luci-app-zt-gateway directory into the SDK's package tree. There are two approaches:

Option A: Direct placement (simplest)

# Copy or symlink the package directory into the SDK
# If your source is at ~/projects/luci-app-zt-gateway/:
ln -s ~/projects/luci-app-zt-gateway package/luci-app-zt-gateway

Option B: Custom feed (recommended for multiple packages)

# 1. Create a feed directory
mkdir -p ~/my-openwrt-feed
ln -s ~/projects/luci-app-zt-gateway ~/my-openwrt-feed/luci-app-zt-gateway

# 2. Register the feed in the SDK
echo 'src-link customfeed /home/YOURUSER/my-openwrt-feed' >> feeds.conf.default

# 3. Update and install the custom feed
./scripts/feeds update customfeed
./scripts/feeds install -p customfeed luci-app-zt-gateway

Step 4: The Makefile

The luci-app-zt-gateway/Makefile uses the LuCI build system (luci.mk) which automates directory mapping, installation, and packaging. The Makefile must be in the package root alongside the htdocs/ and root/ directories.

# luci-app-zt-gateway/Makefile
include $(TOPDIR)/rules.mk

PKG_NAME:=luci-app-zt-gateway
LUCI_TITLE:=LuCI support for ZeroTier Exit Gateway Switching
LUCI_DEPENDS:=+luci-base +ucode +conntrack
LUCI_PKGARCH:=all
PKG_VERSION:=1.0.0
PKG_RELEASE:=1
PKG_LICENSE:=Apache-2.0
PKG_MAINTAINER:=Your Name <you@example.com>

include $(TOPDIR)/feeds/luci/luci.mk

# call BuildPackage - OpenWrt buildroot signature

Key points:

  • luci.mk (not package.mk) is used for LuCI applications. It scans htdocs/ and root/ and automatically creates install rules that map files to their target paths:
    • htdocs/luci-static/resources/view/ → /www/luci-static/resources/view/
    • root/ → mirrors directly onto the target filesystem root (/)
  • LUCI_PKGARCH:=all because LuCI apps are architecture-independent (JS, ucode, shell only).
  • include $(TOPDIR)/feeds/luci/luci.mk uses the absolute path variant. If building inside the main openwrt/luci feed, use include ../../luci.mk instead.
  • The trailing comment # call BuildPackage - OpenWrt buildroot signature is required by the OpenWRT feed scanner to discover this package.
  • Dependencies: +luci-base is mandatory. +ucode is needed for the rpcd backend. +conntrack is needed because the switch script uses conntrack CLI.

Directory-to-path mapping (luci.mk automatics)

When luci.mk processes this package, it automatically installs:

Source path in package Target path on router
htdocs/luci-static/resources/view/zt-gateway/overview.js /www/luci-static/resources/view/zt-gateway/overview.js
root/etc/config/zt-gateway /etc/config/zt-gateway
root/usr/share/rpcd/ucode/zt-gateway.uc /usr/share/rpcd/ucode/zt-gateway.uc
root/usr/share/acl.d/luci-app-zt-gateway.json /usr/share/rpcd/acl.d/luci-app-zt-gateway.json
root/usr/share/luci/menu.d/luci-app-zt-gateway.json /usr/share/luci/menu.d/luci-app-zt-gateway.json
root/usr/sbin/zt-gateway-switch /usr/sbin/zt-gateway-switch

Step 5: Configure and build

# Select the package in menuconfig
make menuconfig
# Navigate: LuCI → 3. Applications → luci-app-zt-gateway
# Press 'M' to build as module (do NOT press 'Y' — we want an .ipk, not a firmware image)
# Save and exit

# Build the package with verbose output
make package/luci-app-zt-gateway/compile V=s

If make menuconfig doesn't show the package, ensure the feeds are updated:

./scripts/feeds update -a
./scripts/feeds install luci-app-zt-gateway

Step 6: Locate the built .ipk

find bin/ -name 'luci-app-zt-gateway*.ipk' -type f

Typical output path: bin/packages/<architecture>/customfeed/luci-app-zt-gateway_1.0.0-1_all.ipk

Because LUCI_PKGARCH:=all, the .ipk filename ends with _all.ipk — it's architecture-independent and works on any OpenWRT target.

Step 7: Install on the router

Transfer the package

scp bin/packages/*/customfeed/luci-app-zt-gateway_1.0.0-1_all.ipk root@192.168.13.1:/tmp/

Install via opkg (OpenWRT ≤ 24.10)

ssh root@192.168.13.1
opkg update
opkg install /tmp/luci-app-zt-gateway_1.0.0-1_all.ipk

If you get dependency errors (e.g. missing luci-base), install them first:

opkg install luci-base ucode conntrack
opkg install /tmp/luci-app-zt-gateway_1.0.0-1_all.ipk

Install via apk (OpenWRT 25.12+)

Starting with OpenWRT 25.12, the package manager is apk instead of opkg:

ssh root@192.168.13.1
apk add --allow-untrusted /tmp/luci-app-zt-gateway_1.0.0-1_all.ipk

Post-install: refresh LuCI cache

rm -rf /tmp/luci-indexcache /tmp/luci-modulecache

Then navigate to the LuCI web UI — the "ZeroTier Exit Gateway" page should appear under Services.

Step 8: Configure gateways

After installation, edit the UCI config to add your gateway nodes:

vi /etc/config/zt-gateway

Or use uci commands:

uci set zt-gateway.global.active_gateway='amsterdam'
uci set zt-gateway.global.switch_mode='force'

uci add zt-gateway gateway
uci set zt-gateway.@gateway[-1].region='amsterdam'
uci set zt-gateway.@gateway[-1].label='Amsterdam (ocirosea641)'
uci set zt-gateway.@gateway[-1].ip='10.11.12.3'
uci set zt-gateway.@gateway[-1].default='1'
uci set zt-gateway.@gateway[-1].health_check='ping'

uci add zt-gateway gateway
uci set zt-gateway.@gateway[-1].region='tirunelveli'
uci set zt-gateway.@gateway[-1].label='Tirunelveli (rpi1000)'
uci set zt-gateway.@gateway[-1].ip='10.11.12.5'
uci set zt-gateway.@gateway[-1].default='0'
uci set zt-gateway.@gateway[-1].health_check='ping'

uci add zt-gateway gateway
uci set zt-gateway.@gateway[-1].region='bangalore'
uci set zt-gateway.@gateway[-1].label='Bangalore (sensecap-m4)'
uci set zt-gateway.@gateway[-1].ip='10.11.12.4'
uci set zt-gateway.@gateway[-1].default='0'
uci set zt-gateway.@gateway[-1].health_check='ping'

uci commit zt-gateway

Step 9: Verify the rpcd backend is registered

After install, verify the ubus backend is available:

# Restart rpcd to pick up the new backend
/etc/init.d/rpcd restart

# List available methods
ubus list zt-gateway.*
# Expected output:
#   zt-gateway.status
#   zt-gateway.switch
#   zt-gateway.health
#   zt-gateway.drain_status
#   zt-gateway.cancel_drain

# Test the status method
ubus call zt-gateway status

If ubus list doesn't show zt-gateway.*, check:

  1. The ucode script is executable and at the correct path: ls -la /usr/share/rpcd/ucode/zt-gateway.uc
  2. The ACL file exists: cat /usr/share/rpcd/acl.d/luci-app-zt-gateway.json
  3. rpcd log output: logread -e rpcd

Quick-build cheat sheet (no SDK, manual .ipk)

If you don't want to bother with the full SDK and just want to produce an .ipk manually, you can use the opkg-utils tools on any Linux machine:

# Install opkg-utils
sudo apt install opkg-utils   # Debian/Ubuntu

# Create the package staging directory
mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/CONTROL
mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/www/luci-static/resources/view/zt-gateway
mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/etc/config
mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/rpcd/ucode
mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/rpcd/acl.d
mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/luci/menu.d
mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/sbin

# Copy source files into staging (adjust source paths as needed)
cp htdocs/luci-static/resources/view/zt-gateway/overview.js \
   /tmp/luci-app-zt-gateway_1.0.0-1_all/www/luci-static/resources/view/zt-gateway/
cp root/etc/config/zt-gateway /tmp/luci-app-zt-gateway_1.0.0-1_all/etc/config/
cp root/usr/share/rpcd/ucode/zt-gateway.uc /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/rpcd/ucode/
cp root/usr/share/rpcd/acl.d/luci-app-zt-gateway.json /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/rpcd/acl.d/
cp root/usr/share/luci/menu.d/luci-app-zt-gateway.json /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/luci/menu.d/
cp root/usr/sbin/zt-gateway-switch /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/sbin/
chmod +x /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/sbin/zt-gateway-switch

# Write the control file
cat > /tmp/luci-app-zt-gateway_1.0.0-1_all/CONTROL/control << 'EOF'
Package: luci-app-zt-gateway
Version: 1.0.0-1
Depends: libc, luci-base, ucode, conntrack
Source: luci-app-zt-gateway
Section: luci
Architecture: all
Installed-Size: 20480
Description: LuCI support for ZeroTier Exit Gateway Switching
EOF

# Build the .ipk
cd /tmp
opkg-build -O -o root -g root luci-app-zt-gateway_1.0.0-1_all

# Result: /tmp/luci-app-zt-gateway_1.0.0-1_all.ipk

This manual approach skips the SDK entirely. It's suitable for quick iteration during development. The tradeoff: no automatic dependency resolution, no translation compilation, no JS minification.

Development workflow: iterate fast without rebuilding

During development, you can skip the build step entirely by editing files directly on the router:

# Edit the JS view directly
scp htdocs/luci-static/resources/view/zt-gateway/overview.js \
    root@192.168.13.1:/www/luci-static/resources/view/zt-gateway/overview.js

# Edit the ucode backend
scp root/usr/share/rpcd/ucode/zt-gateway.uc \
    root@192.168.13.1:/usr/share/rpcd/ucode/zt-gateway.uc
/etc/init.d/rpcd restart

# Edit the switch script
scp root/usr/sbin/zt-gateway-switch \
    root@192.168.13.1:/usr/sbin/zt-gateway-switch

# Edit the UCI config
scp root/etc/config/zt-gateway \
    root@192.168.13.1:/etc/config/zt-gateway

# Clear LuCI browser cache after JS changes
ssh root@192.168.13.1 'rm -rf /tmp/luci-indexcache /tmp/luci-modulecache'

Build the .ipk only when you need a clean install or are sharing the package.

Troubleshooting

Problem Cause Fix
make menuconfig doesn't show the package Feed not registered Run ./scripts/feeds update -a && ./scripts/feeds install -a
opkg install fails with "cannot install package" Missing dependencies Run opkg update then install dependencies first
LuCI page shows 404 JS view not found at expected path Verify /www/luci-static/resources/view/zt-gateway/overview.js exists on router
ubus methods missing rpcd didn't pick up the ucode backend /etc/init.d/rpcd restart; check logread -e rpcd for errors
ACL denied when calling ubus from LuCI ACL JSON file missing or wrong format Verify /usr/share/rpcd/acl.d/luci-app-zt-gateway.json and restart rpcd
luci.mk not found during build Building outside the LuCI feed Use include $(TOPDIR)/feeds/luci/luci.mk (absolute) instead of include ../../luci.mk (relative)
JS view loads but is blank Import path errors in JS Open browser dev console; LuCI JS modules use luci.view, luci.form, etc.

Local Testing with Docker (macvlan)

This section covers how to test luci-app-zt-gateway locally using the official openwrt/rootfs Docker image with macvlan networking. This gives you a full OpenWRT environment with real ip rule, iptables, conntrack, and rpcd — no physical router required.

Test architecture

┌──────────────────────────────────────────────────────────────────┐
│  Docker host (your dev machine)                                  │
│                                                                  │
│  ┌──────────────────────┐    ┌──────────────────────┐           │
│  │  openwrt-router      │    │  wiblan-client        │           │
│  │  (macvlan 10.11.13.1)│    │  (macvlan 10.11.13.10)│           │
│  │                      │    │                       │           │
│  │  - LuCI on :80       │    │  - curl / ping        │           │
│  │  - rpcd + ucode      │    │  - traffic exits via  │           │
│  │  - zt-gateway-switch │    │    router's table 100 │           │
│  │  - br-zt + ip rules  │    │                       │           │
│  └──────────┬───────────┘    └───────────┬───────────┘           │
│             │                            │                       │
│             └──────────┬─────────────────┘                       │
│                        │                                         │
│               zt-gateway-macvlan                                 │
│             10.11.13.0/24 subnet                                 │
│             (macvlan bridge mode)                                 │
│                                                                  │
│  ┌──────────────────────┐                                        │
│  │  zt-gw-amsterdam     │  ← Simulates a remote ZeroTier exit   │
│  │  (macvlan 10.11.12.3) │    node with NAT + ip_forward         │
│  │                      │                                        │
│  │  - ip_forward=1      │                                        │
│  │  - iptables MASQ     │                                        │
│  └──────────────────────┘                                        │
│                                                                  │
│               zt-exit-macvlan                                    │
│             10.11.12.0/24 subnet                                 │
└──────────────────────────────────────────────────────────────────┘

The key insight: we use two macvlan networks to simulate the two network segments the real router has:

  1. zt-gateway-macvlan (10.11.13.0/24) — represents the WIBLAN subnet. The router sits at .1, client at .10.
  2. zt-exit-macvlan (10.11.12.0/24) — represents the ZeroTier network where exit gateway nodes live.

Step 1: Create the Dockerfile

Create a Dockerfile that builds a pre-configured OpenWRT image with all dependencies and the zt-gateway package:

# Dockerfile.openwrt-dev
FROM openwrt/rootfs:x86-64-24.10.7

# Install required packages
RUN opkg update && opkg install \
    luci \
    luci-base \
    ucode \
    conntrack \
    ip-bridge \
    iptables \
    kmod-ipt-conntrack \
    kmod-ipt-connmark \
    kmod-ipt-mark \
    kmod-ipt-extra \
    && rm -rf /var/opkg-lists/*

# Enable uhttpd
RUN /etc/init.d/uhttpd enable
RUN /etc/init.d/uhttpd start

# Enable rpcd
RUN /etc/init.d/rpcd enable
RUN /etc/init.d/rpcd start

# Set root password (required for LuCI login)
RUN echo "root:root" | chpasswd

EXPOSE 80

CMD ["/sbin/init"]

Build it:

docker build -t openwrt-zt-gateway-dev -f Dockerfile.openwrt-dev .

Note: The exact kmod-* packages available depend on the OpenWRT version. If opkg install fails on kernel modules, skip them — the Docker container's kernel is the host kernel, so the modules are already loaded. What matters is that iptables -t mangle and conntrack work. Verify with:

# Run a quick check after building
docker run --rm --privileged openwrt-zt-gateway-dev sh -c \
  "iptables -t mangle -L && echo 'mangle OK' && conntrack -L 2>/dev/null && echo 'conntrack OK'"

Step 2: Create the Docker Compose file

# docker-compose.yml
version: "3.8"

networks:
  # Simulates the WIBLAN client network (10.11.13.0/24)
  zt-gateway-lan:
    driver: macvlan
    driver_opts:
      mode: bridge
    ipam:
      config:
        - subnet: "10.11.13.0/24"
          gateway: "10.11.13.1"
          ip_range: "10.11.13.100/28"

  # Simulates the ZeroTier exit node network (10.11.12.0/24)
  zt-exit-net:
    driver: macvlan
    driver_opts:
      mode: bridge
    ipam:
      config:
        - subnet: "10.11.12.0/24"
          ip_range: "10.11.12.100/28"

services:
  # The OpenWRT router running luci-app-zt-gateway
  openwrt-router:
    image: openwrt-zt-gateway-dev
    container_name: openwrt-router
    privileged: true
    networks:
      zt-gateway-lan:
        ipv4_address: "10.11.13.1"
      zt-exit-net:
        ipv4_address: "10.11.12.1"
    ports:
      - "8080:80"
    volumes:
      # Hot-mount source files for live iteration (key advantage)
      - ./luci-app-zt-gateway/htdocs/luci-static/resources/view/zt-gateway/overview.js:/www/luci-static/resources/view/zt-gateway/overview.js
      - ./luci-app-zt-gateway/root/usr/share/rpcd/ucode/zt-gateway.uc:/usr/share/rpcd/ucode/zt-gateway.uc
      - ./luci-app-zt-gateway/root/usr/sbin/zt-gateway-switch:/usr/sbin/zt-gateway-switch
      - ./luci-app-zt-gateway/root/etc/config/zt-gateway:/etc/config/zt-gateway
      - ./luci-app-zt-gateway/root/usr/share/rpcd/acl.d/luci-app-zt-gateway.json:/usr/share/rpcd/acl.d/luci-app-zt-gateway.json
      - ./luci-app-zt-gateway/root/usr/share/luci/menu.d/luci-app-zt-gateway.json:/usr/share/luci/menu.d/luci-app-zt-gateway.json
    cap_add:
      - NET_ADMIN
      - SYS_ADMIN
    sysctls:
      - net.ipv4.ip_forward=1
      - net.ipv4.conf.all.send_redirects=0
    command: ["/sbin/init"]

  # A WIBLAN client to test routing through the gateway
  wiblan-client:
    image: alpine:latest
    container_name: wiblan-client
    networks:
      zt-gateway-lan:
        ipv4_address: "10.11.13.10"
    cap_add:
      - NET_ADMIN
    command: ["sleep", "infinity"]

  # Simulates the amsterdam ZeroTier exit gateway (10.11.12.3)
  zt-gw-amsterdam:
    image: alpine:latest
    container_name: zt-gw-amsterdam
    networks:
      zt-exit-net:
        ipv4_address: "10.11.12.3"
    cap_add:
      - NET_ADMIN
    sysctls:
      - net.ipv4.ip_forward=1
    command: ["sleep", "infinity"]

  # Simulates the tirunelveli ZeroTier exit gateway (10.11.12.5)
  zt-gw-tirunelveli:
    image: alpine:latest
    container_name: zt-gw-tirunelveli
    networks:
      zt-exit-net:
        ipv4_address: "10.11.12.5"
    cap_add:
      - NET_ADMIN
    sysctls:
      - net.ipv4.ip_forward=1
    command: ["sleep", "infinity"]

Step 3: Launch and configure the environment

# Start all containers
docker compose up -d

# Wait a few seconds for init, then configure the simulated exit gateways
# These simulate real ZeroTier exit nodes with NAT

Configure the amsterdam exit gateway (10.11.12.3)

docker exec -it zt-gw-amsterdam sh -c '
  # Enable forwarding
  echo 1 > /proc/sys/net/ipv4/ip_forward

  # Add a default route back to the router so return traffic works
  ip route add 10.11.13.0/24 via 10.11.12.1

  # Add NAT so traffic from WIBLAN clients can reach the internet
  # (or just other container networks for testing)
  apk add --no-cache iptables
  iptables -t nat -A POSTROUTING -s 10.11.13.0/24 -j MASQUERADE
  iptables -A FORWARD -s 10.11.13.0/24 -j ACCEPT
  iptables -A FORWARD -d 10.11.13.0/24 -m state --state ESTABLISHED,RELATED -j ACCEPT

  # Add a fake "internet" endpoint (just the Docker host gateway)
  # In real use, this would be the actual internet
  ping -c 1 10.11.12.1 2>/dev/null
'

Configure the tirunelveli exit gateway (10.11.12.5)

docker exec -it zt-gw-tirunelveli sh -c '
  echo 1 > /proc/sys/net/ipv4/ip_forward
  ip route add 10.11.13.0/24 via 10.11.12.1
  apk add --no-cache iptables
  iptables -t nat -A POSTROUTING -s 10.11.13.0/24 -j MASQUERADE
  iptables -A FORWARD -s 10.11.13.0/24 -j ACCEPT
  iptables -A FORWARD -d 10.11.13.0/24 -m state --state ESTABLISHED,RELATED -j ACCEPT
'

Configure the OpenWRT router

docker exec -it openwrt-router sh

Inside the router container:

# 1. Create the br-zt bridge (simulates the ZeroTier bridge)
ip link add name br-zt type bridge
ip link set br-zt up

# Assign the ZT-exit IP to the bridge so the router can reach 10.11.12.0/24
ip addr add 10.11.12.1/24 dev br-zt

# 2. Set up policy routing (same as the real router)

# Host route to the current gateway (ARP fix)
ip route replace 10.11.12.3 dev br-zt

# Default route via the current gateway in table 100
ip route replace default via 10.11.12.3 dev br-zt table 100

# Route WIBLAN subnet via br-zt in mwan3 table (for return traffic)
ip route replace 10.11.13.0/24 dev br-zt table 1

# Source-based routing rule: WIBLAN traffic → table 100
ip rule add from 10.11.13.0/24 table 100 priority 100

# 3. Verify routing
ip route show table 100
# Expected: default via 10.11.12.3 dev br-zt

ip rule show
# Expected: 100: from 10.11.13.0/24 lookup 100

# 4. Test from the router itself
ping -c 2 10.11.12.3   # Should reach amsterdam gateway

Configure the WIBLAN client

docker exec -it wiblan-client sh -c '
  # Add a route to the router for the ZT subnet
  ip route add 10.11.12.0/24 via 10.11.13.1

  # Set the OpenWRT router as default gateway
  ip route add default via 10.11.13.1

  # Test connectivity
  ping -c 2 10.11.12.3
'

Step 4: Install and test the luci-app-zt-gateway package

Option A: The package is already volume-mounted

If you used the volumes: section in docker-compose.yml, the files are already in place. Just restart services:

docker exec -it openwrt-router sh -c '
  chmod +x /usr/sbin/zt-gateway-switch
  /etc/init.d/rpcd restart
  rm -rf /tmp/luci-indexcache /tmp/luci-modulecache
'

Option B: Install the .ipk

If you built an .ipk, copy it into the container:

docker cp luci-app-zt-gateway_1.0.0-1_all.ipk openwrt-router:/tmp/
docker exec -it openwrt-router opkg install /tmp/luci-app-zt-gateway_1.0.0-1_all.ipk

Verify the rpcd backend

docker exec -it openwrt-router ubus list zt-gateway.*
# Expected:
#   zt-gateway.status
#   zt-gateway.switch
#   zt-gateway.health
#   zt-gateway.drain_status
#   zt-gateway.cancel_drain

docker exec -it openwrt-router ubus call zt-gateway status

Access the LuCI UI

Open http://localhost:8080 in your browser. Log in with root / root. Navigate to Services → ZeroTier Exit Gateway.

Step 5: Run the test scenarios

Force switch test

# 1. Verify current gateway is amsterdam (10.11.12.3)
docker exec -it openwrt-router ip route show table 100
# Expected: default via 10.11.12.3 dev br-zt

# 2. Trigger a force switch to tirunelveli via the CLI script
docker exec -it openwrt-router zt-gateway-switch 10.11.12.5 force

# 3. Verify the route changed
docker exec -it openwrt-router ip route show table 100
# Expected: default via 10.11.12.5 dev br-zt

# 4. Verify conntrack was flushed
docker exec -it openwrt-router conntrack -L -s 10.11.13.0/24 2>/dev/null
# Expected: empty (all entries flushed)

# 5. Verify from the client side
docker exec -it wiblan-client ping -c 2 10.11.12.5

Graceful drain test

# 1. Start a long-running connection from the client
docker exec -d wiblan-client ping 10.11.12.3   # background ping

# 2. Trigger a graceful switch (this will be done from the LuCI UI
#    or via ubus call)
docker exec -it openwrt-router zt-gateway-switch 10.11.12.5 graceful 300

# 3. Verify both routing tables exist during drain
docker exec -it openwrt-router ip route show table 100
# Expected: default via 10.11.12.5 dev br-zt  (new gateway)

docker exec -it openwrt-router ip route show table 101
# Expected: default via 10.11.12.3 dev br-zt  (old gateway, drain)

# 4. Verify the fwmark rule exists
docker exec -it openwrt-router ip rule show
# Expected: 99: from all fwmark 0x100 lookup 101

# 5. Verify the mangle rules
docker exec -it openwrt-router iptables -t mangle -L -v

# 6. Watch drain progress
docker exec -it openwrt-router conntrack -L -m 0x100 2>/dev/null | wc -l

# 7. Wait for drain to complete (or kill the background ping to speed it up)
docker exec -it wiblan-client pkill ping

# 8. After drain completes, verify cleanup
docker exec -it openwrt-router ip route show table 101
# Expected: empty (drain table removed)
docker exec -it openwrt-router ip rule show
# Expected: no fwmark 0x100 rule

Switch to unreachable gateway test

# 1. Stop the tirunelveli gateway to simulate it being down
docker stop zt-gw-tirunelveli

# 2. Try to switch to it (should fail pre-flight)
docker exec -it openwrt-router zt-gateway-switch 10.11.12.5 force
# Expected: "Gateway unreachable" error, no route changes

# 3. Verify routes unchanged
docker exec -it openwrt-router ip route show table 100

# 4. Restart for subsequent tests
docker start zt-gw-tirunelveli

Drain timeout test

# 1. Start a persistent connection
docker exec -d wiblan-client ping 10.11.12.3

# 2. Graceful switch with short timeout (30s)
docker exec -it openwrt-router zt-gateway-switch 10.11.12.5 graceful 30

# 3. Watch the drain monitor — it will hit the 30s timeout
#    because the ping keeps conntrack entries alive
docker exec -it openwrt-router conntrack -L -m 0x100 2>/dev/null | wc -l

# 4. After 30s, should auto-fallback to force switch
docker exec -it openwrt-router ip route show table 100
# Expected: default via 10.11.12.5 dev br-zt (force applied)
docker exec -it openwrt-router ip route show table 101
# Expected: empty (drain cleaned up by force)

# 5. Clean up
docker exec -it wiblan-client pkill ping

Step 6: Iterate on fixes

The key advantage of the Docker setup is the volume mounts in the compose file. When you edit a source file on the host, it's immediately reflected inside the container.

Live-editing workflow

# 1. Edit a source file on the host
vim luci-app-zt-gateway/root/usr/sbin/zt-gateway-switch

# 2. Re-test inside the container (file is already updated via volume mount)
docker exec -it openwrt-router zt-gateway-switch 10.11.12.5 force

# 3. Check results
docker exec -it openwrt-router ip route show table 100

For LuCI JS changes, you need to clear the cache:

# Edit JS on host
vim luci-app-zt-gateway/htdocs/luci-static/resources/view/zt-gateway/overview.js

# Clear cache inside container
docker exec -it openwrt-router rm -rf /tmp/luci-indexcache /tmp/luci-modulecache

# Refresh browser

For ucode backend changes:

# Edit ucode on host
vim luci-app-zt-gateway/root/usr/share/rpcd/ucode/zt-gateway.uc

# Restart rpcd to reload the backend
docker exec -it openwrt-router /etc/init.d/rpcd restart

Quick-snapshot the exact container state

If you want to save a known-good state before making a risky change:

# Save container state as a Docker image
docker commit openwrt-router openwrt-zt-gateway-snapshot:good

# ... make changes, break things ...

# Restore from snapshot
docker compose down
docker run -d --name openwrt-router \
  --privileged \
  -p 8080:80 \
  openwrt-zt-gateway-snapshot:good

Watch logs in real-time

# OpenWRT system log
docker exec -it openwrt-router logread -f

# rpcd errors specifically
docker exec -it openwrt-router logread -f -e rpcd

# conntrack events in real-time
docker exec -it openwrt-router conntrack -E

Run the full test suite in one shot

#!/bin/bash
# test-zt-gateway.sh — Run all test scenarios in Docker
set -e

ROUTER="openwrt-router"
CLIENT="wiblan-client"
GW_AMS="10.11.12.3"
GW_TIR="10.11.12.5"

echo "=== Test 1: Force switch amsterdam → tirunelveli ==="
docker exec $ROUTER zt-gateway-switch $GW_TIR force
ROUTE=$(docker exec $ROUTER ip route show table 100 | grep default)
echo "  Route after force: $ROUTE"
echo "$ROUTE" | grep -q "$GW_TIR" && echo "  PASS" || echo "  FAIL"

echo "=== Test 2: Force switch tirunelveli → amsterdam ==="
docker exec $ROUTER zt-gateway-switch $GW_AMS force
ROUTE=$(docker exec $ROUTER ip route show table 100 | grep default)
echo "  Route after force: $ROUTE"
echo "$ROUTE" | grep -q "$GW_AMS" && echo "  PASS" || echo "  FAIL"

echo "=== Test 3: Switch to unreachable gateway ==="
docker stop zt-gw-tirunelveli
if docker exec $ROUTER zt-gateway-switch $GW_TIR force 2>&1 | grep -qi "unreachable\|fail\|error"; then
  echo "  PASS (blocked as expected)"
else
  echo "  FAIL (should have been blocked)"
fi
docker start zt-gw-tirunelveli
sleep 3

echo "=== Test 4: Verify client can reach gateway ==="
docker exec $CLIENT ping -c 2 $GW_AMS && echo "  PASS" || echo "  FAIL"

echo "=== Test 5: Verify ubus backend ==="
docker exec $ROUTER ubus call zt-gateway status | grep -q "amsterdam" && echo "  PASS" || echo "  FAIL"

echo "=== All tests complete ==="

Important limitations of the Docker test environment

Limitation Impact Mitigation
Docker shares the host kernel kmod-* packages from opkg won't load (wrong kernel version) Use host kernel modules; verify with lsmod/iptables -t mangle -L
br-zt is simulated, not a real ZeroTier bridge No actual ZT membership, no ztk4jpk77j interface Manually create br-zt bridge and assign IPs; skip hotplug scripts
conntrack -D -s may behave differently Docker's conntrack table includes host traffic too Use -s and -d filters; or run conntrack -F for a clean slate
No real rc.local / hotplug execution on boot Persistence scripts can't be tested end-to-end Test the switch script logic; verify sed replacements on the files separately
iptables -t mangle requires NET_ADMIN Container must run --privileged or --cap-add=NET_ADMIN Already configured in compose file
Macvlan host isolation Host can't ping macvlan containers by default Use a macvlan shim interface (see below) or just docker exec
No real ZeroTier Central API Can't test the future ZT API integration Mock the API calls or skip this test

Fixing the macvlan host-isolation issue

By default, the Docker host cannot communicate directly with containers on a macvlan network. If you need to ping/curl the router or client from your host machine (not just via docker exec), create a macvlan shim interface:

# Create a macvlan interface on the host that bridges to the same parent
# Replace eth0 with your host's primary interface
sudo ip link add link eth0 name mv-shim type macvlan mode bridge
sudo ip addr add 10.11.13.254/32 dev mv-shim
sudo ip link set mv-shim up

# Add a route to the WIBLAN subnet via the shim
sudo ip route add 10.11.13.0/24 dev mv-shim

# Now you can reach the containers from host
ping 10.11.13.1        # the router
curl http://10.11.13.1  # LuCI directly (skip the port mapping)

# For the ZT exit subnet
sudo ip link add link eth0 name mv-exit-shim type macvlan mode bridge
sudo ip addr add 10.11.12.254/32 dev mv-exit-shim
sudo ip link set mv-exit-shim up
sudo ip route add 10.11.12.0/24 dev mv-exit-shim

Clean up when done:

sudo ip link del mv-shim
sudo ip link del mv-exit-shim

Teardown

# Stop and remove all containers
docker compose down

# Remove the shim interfaces (if created)
sudo ip link del mv-shim 2>/dev/null
sudo ip link del mv-exit-shim 2>/dev/null

# Remove the custom images (optional)
docker rmi openwrt-zt-gateway-dev openwrt-zt-gateway-snapshot:good
Reference in New Issue View Git Blame Copy Permalink